Video call method, device, medium and product

Abstract

The invention relates to the technical field of terminals, and discloses a video call method and device, a medium and a product. The video call method can be applied to a one-to-one video call scene. In the method, the electronic device can obtain a call video and determine at least one frame of to-be-detected image from the call video, and each to-be-detected image comprises a first user call window and a second user call window. The far-end user call window displays the face of the call object by default, the home-end user call window displays the face of the user by default, and the area of the far-end user call window is larger than that of the home-end user call window. Therefore, the electronic equipment can determine the area of the first user call window and the area of the second user call window respectively, and anti-counterfeiting detection is carried out on the face of the user in the user call window with the large area. Therefore, the face of the user in the call window of the user at the home terminal can be prevented from being detected, and the computing resources of the electronic equipment can be saved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a video call method, device, medium and product. Background Technology

[0002] Electronic devices such as mobile phones, tablets, and computers typically offer video calling capabilities. During a video call, both parties can see each other's faces to verify their identity. However, some individuals with ulterior motives use artificial intelligence (AI) face-swapping technology to replace the face of the other person in a video call with that of a relative or friend, then impersonate that relative or friend to request money transfers or payments. This can easily lead to financial losses for the user. Summary of the Invention

[0003] To address the issue of potential financial losses for users, embodiments of this application provide a video call method, device, medium, and product, including:

[0004] In a first aspect, embodiments of this application provide a video call method applied to an electronic device, comprising: acquiring a first image to be detected during a video call, the first image to be detected including a first user call window and a second user call window; detecting that the area of ​​the first user call window is larger than the area of ​​the second user call window; and performing anti-spoofing detection on the user's face in the first user call window.

[0005] Based on the above solution, during video calls, it is possible to detect whether the caller is acting with ulterior motives, thereby reducing potential financial losses for users. Furthermore, it avoids the need for anti-spoofing detection on the user's face within the relatively small area of ​​the second user's call window, conserving the computing resources of electronic devices.

[0006] It is understandable that the first user call window can be called the remote user call window, and the second user call window can be called the local user call window.

[0007] In some alternative implementations, the first image to be detected can be any video frame from the call video. This first image to be detected may include four color channels: red (R), green (G), blue (B), and alpha (A).

[0008] In some optional implementations of the first aspect, anti-spoofing detection of the user's face in the first user call window includes: downsampling the first image to be detected to obtain a second image to be detected, wherein the resolution of the second image to be detected is less than the resolution of the first image to be detected, and the second image to be detected includes facial key points; determining the matching region corresponding to the user's face in the first user window from the second image to be detected based on the facial key points; determining the first face region corresponding to the matching region from the first image to be detected based on the resolution relationship between the first image to be detected and the second image to be detected; and performing anti-spoofing detection on the image of the first face region in the first image to be detected to obtain an anti-spoofing detection result.

[0009] In some alternative implementations, the electronic device can input a second image to be detected into an AI model, in which facial keypoint features related to facial keypoints can be extracted from the second image to be detected. Based on the facial keypoint features, the coordinates of the facial keypoints in the image coordinate system corresponding to the second image to be detected are predicted, and based on the coordinates of the facial keypoints in the image coordinate system corresponding to the second image to be detected, the matching region corresponding to the user's face in the first user window is determined.

[0010] In some alternative implementations, the electronic device can directly perform anti-counterfeiting detection on the first face region in the first image to be detected, or it can extract the image of the first face region from the first image to be detected and perform anti-counterfeiting detection on the image of the first face region. This application embodiment does not make specific limitations.

[0011] In this embodiment, by downsampling the first image to be detected to identify the matching region corresponding to the user's face in the first user window, the computing resources of the electronic device can be further saved. Furthermore, by determining the first face region corresponding to the matching region from the first image to be detected based on the resolution relationship between the first and second images to be detected, and then performing anti-spoofing detection on the image of the first face region, the accuracy of face anti-spoofing detection can be improved by utilizing the detailed features in the image of the first face region.

[0012] In some optional implementations of the first aspect, the method further includes: determining a second face region based on a first face region, wherein the area of ​​the second face region is larger than the area of ​​the first face region; performing anti-counterfeiting detection on the image of the second face region in the first image to be detected, and obtaining an anti-counterfeiting detection result.

[0013] In some alternative implementations, the electronic device can directly perform anti-counterfeiting detection on the second face region in the first image to be detected, or it can extract the image of the second face region from the first image to be detected and perform anti-counterfeiting detection on the image of the second face region. This application embodiment does not make specific limitations.

[0014] In this embodiment, a second face region is obtained by enlarging the first face region for anti-counterfeiting detection. This can save the computing resources of electronic devices while increasing the facial features of the user's face in the first user call window, thereby further improving the accuracy of face anti-counterfeiting detection.

[0015] In some optional implementations of the first aspect, anti-spoofing detection of the user's face in the first user call window includes: determining the coordinates of each pixel corresponding to the facial contour of the user's face in the image coordinate system corresponding to the image of the second face region, and the image data of adjacent pixels in the image of the second face region, from the image of the second face region; determining the forgery confidence level of the user's face in the image of the second face region as a forged face based on at least one of the coordinates of each pixel corresponding to the facial contour of the user's face in the image coordinate system corresponding to the image of the second face region and the image data of adjacent pixels; and determining that the user's face in the image of the second face region is a forged face if the forgery confidence level of the user's face in the image of the second face region is greater than the forgery confidence level threshold.

[0016] It is understandable that in practical applications, electronic devices can perform the above processing on multiple video frames (i.e., the first image to be detected) in a video call, and based on the forgery confidence of the forged face corresponding to multiple video frames, comprehensively judge whether the caller in the video call is a caller with an improper purpose. This avoids the randomness of a single video frame and can further improve the accuracy of anti-counterfeiting detection.

[0017] In addition, electronic devices can determine whether the caller is making a call for an illegitimate purpose based on the continuity of adjacent video frames in multiple video frames. This avoids the randomness of a single video frame and can further improve the accuracy of anti-counterfeiting detection.

[0018] In some alternative implementations of the first aspect, the image data includes at least one of the following: color, texture, and depth.

[0019] In some optional implementations of the first aspect, corresponding to a resolution of M*N for the first image to be detected, downsampling the first image to be detected to obtain a second image to be detected includes: determining a first ratio of M to a preset length and a second ratio of N to a preset width; corresponding to a first ratio being greater than the second ratio, determining the length of the second image to be detected based on the ratio of M to the first ratio and determining the width of the second image to be detected based on the ratio of N to the first ratio.

[0020] In some optional implementations of the first aspect, corresponding to the first user window including multiple user faces, the anti-spoofing detection of the user faces in the first user call window includes: determining the first face region corresponding to each user face among the multiple user faces from the first image to be detected; performing anti-spoofing detection on the first face region with the largest area among the first face regions corresponding to each user face among the multiple user faces, and obtaining the anti-spoofing detection result.

[0021] In this embodiment, for multiple user faces within the first user window, anti-counterfeiting detection is performed only on the user face with the largest area, which can further save the computing resources of electronic devices.

[0022] In a second aspect, this application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the video call method mentioned in the first aspect or any one of the first aspects of this application.

[0023] Thirdly, this application provides a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the video call method mentioned in the first aspect or any of the claims of the first aspect.

[0024] Fourthly, embodiments of this application provide a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device executes the computer program code of the video call method mentioned in the first aspect or any one of the first aspects of this application. Attached Figure Description

[0025] Figure 1 A schematic diagram of a video call is shown;

[0026] Figure 2 According to some embodiments of this application, a flowchart of a video call method is shown;

[0027] Figure 3 According to some embodiments of this application, a schematic diagram of a process for anti-counterfeiting detection of a user's face in a first user call window is shown.

[0028] Figure 4 According to some embodiments of this application, a flowchart of another video call method is shown;

[0029] Figure 5 According to some embodiments of this application, a schematic diagram of the hardware structure of an electronic device is shown;

[0030] Figure 6According to some embodiments of this application, a schematic diagram of the software structure of an electronic device is shown. Detailed Implementation

[0031] The embodiments of this application include, but are not limited to, a video call method, device, medium, and product.

[0032] It is understood that the video call method mentioned in the embodiments of this application can be applied to video call scenarios. For example, an electronic device may have an application (APP) that supports video calls installed, and the electronic device can conduct video calls through these APPs. The video call scenarios can include scenarios where the user actively initiates a video call and scenarios where the user passively receives a video call.

[0033] It is understood that the aforementioned electronic device can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. In some specific implementations, the electronic device can be a mobile phone, tablet, laptop, or other device capable of providing video calling functionality, such as a device with a video calling app installed. The following uses a mobile phone as an example to illustrate the solution mentioned in the embodiments of this application.

[0034] As mentioned earlier, some callers with ulterior motives use AI face-swapping technology to replace the face of the caller with that of the user's relative or friend during video calls. They then impersonate the user's relative or friend and ask the user for money transfers or payment assistance, which can easily lead to financial losses for the user.

[0035] Therefore, in some specific implementations, electronic devices can perform anti-spoofing detection on the facial image of the person making the call displayed on the electronic device. Currently, since some AI models have facial recognition capabilities, electronic devices can perform facial recognition on each video frame in the call video based on the AI ​​model, and then perform anti-spoofing detection on the identified facial regions.

[0036] Specifically, in the facial recognition process, the AI ​​model can extract facial key point features (such as eyes, nose, and mouth) from each video frame of a call video. Based on these features, the model predicts the coordinates of the facial key points in the corresponding image coordinate system for each video frame. For example, the image coordinate system could have its origin at the bottom left corner of the video frame, with the horizontal axis pointing to the right and the vertical axis pointing upwards. The facial region is then determined based on the coordinates of the facial key points in the corresponding image coordinate system for each video frame.

[0037] For example, based on facial key point features, the coordinates of the outer corner of the left eye are predicted to be (50, 80), the inner corner of the left eye to be (80, 80), the inner corner of the right eye to be (120, 80), the outer corner of the right eye to be (150, 80), the tip of the nose to be (100, 120), the left corner of the mouth to be (70, 150), and the right corner of the mouth to be (130, 150). Furthermore, the electronic device can determine a rectangle with the top left corner (30, 45) and the bottom right corner (170, 180) from the video frame based on the distance of the facial key points from the facial contours. For example, the outer corner of the left eye is 20 units away from the left side of the face contour, and the mouth is 30 units away from the bottom of the face contour.

[0038] In the process of facial spoofing detection, electronic devices can use edge detection algorithms to extract the facial contour from the facial region and extract features such as color, texture, and depth of adjacent pixels within the facial region. Since replacing the face of a caller in a video call with the face of a relative or friend requires compositing the face with the background or compositing facial landmarks with a facial model, the facial contour of the caller's face may not be smooth enough, and the differences in color, texture, and depth between adjacent pixels may be significant. Therefore, when the electronic device determines that the smoothness of the facial contour is less than a smoothness threshold, or that the differences in color, texture, and depth between adjacent pixels within the facial region exceed a difference threshold, the electronic device can identify the face in the video call as a forged face.

[0039] However, in one-on-one video call scenarios, the call video can include multiple faces, for example, such as Figure 1 As shown, the electronic device can display a remote user call window 110 and a local user call window 120. The remote user call window 110 displays the face of the person on the call by default, while the local user call window 120 displays the face of the user of the electronic device (i.e., the user) by default. Furthermore, the area of ​​the remote user call window is larger than that of the local user call window. Thus, the electronic device performs face spoofing detection on both the remote user call window 110 and the local user call window 120, resulting in a waste of the electronic device's computing resources.

[0040] Therefore, to solve the above problems, this application provides a video call method. In this method, an electronic device can acquire a call video and determine at least one frame of an image to be detected from the call video. Each image to be detected includes a first user call window and a second user call window. As mentioned above, the remote user call window displays the face of the call subject by default, and the local user call window displays the user's face by default. Furthermore, the area of ​​the remote user call window is larger than the area of ​​the local user call window. Therefore, the electronic device can determine the areas of the first user call window and the second user call window respectively, and perform anti-spoofing detection on the user's face within the larger user call window. This avoids detecting the user's face in the smaller local user call window, saving the computing resources of the electronic device.

[0041] In some specific implementations, electronic devices can perform anti-spoofing detection on user faces within a large user call window using the following method: The image to be detected (i.e., the first image to be detected mentioned above) is downsampled to obtain a downsampled image (i.e., the second image to be detected mentioned above), where the resolution of the downsampled image is lower than the resolution of the corresponding image to be detected, and the downsampled image includes facial key points. An AI model is used to identify the matching region corresponding to the user's face within the large user call window from the downsampled image. Based on the resolution correspondence between the downsampled image and the image to be detected, the facial region corresponding to the matching region (i.e., the first facial region mentioned above) is determined from the image to be detected. Then, the electronic device can perform anti-spoofing detection on the user's face within the facial region of the image to be detected, obtaining the anti-spoofing detection result.

[0042] Since a smaller image resolution means fewer pixels and less detail, downsampling the image to be detected to identify the matching region can further conserve the computing resources of electronic devices. Furthermore, by determining the corresponding facial region in the image to be detected based on the resolution relationship between the downsampled image and the image to be detected, anti-spoofing detection can be performed on the user's face within the largest facial region. Because the high-resolution image to be detected contains more image details, this improves the accuracy of face anti-spoofing detection.

[0043] In some optional implementations, the electronic device may also filter the face size within the remote user's call window. For example, when there are multiple face regions within the remote user's call window, face regions larger than a preset area can be selected for subsequent face spoofing detection, or the largest face region can be selected for subsequent face spoofing detection. This reduces the amount of data processing required for subsequent face detection while preserving the details contained in the face region, thereby further improving the accuracy of face spoofing detection.

[0044] The video call method mentioned in the embodiments of this application will be described in detail below. For example... Figure 2 The diagram shows a flowchart of a video call method that can be executed by an electronic device.

[0045] Specifically, such as Figure 2 As shown, the video call method may include:

[0046] 201: Acquire the first image to be detected during the video call, wherein the first image to be detected includes the first user call window and the second user call window.

[0047] It is understood that the first image to be detected can be any video frame in the call video. In some optional implementations, the first image to be detected may include four color channels: red (R), green (G), blue (B), and alpha (A).

[0048] 202: Detect the area of ​​the first user's call window and the area of ​​the second user's call window.

[0049] In some alternative implementations, the first user call window can be called the remote user call window, and the second user call window can be called the local user call window, wherein the area of ​​the remote user call window is larger than the area of ​​the local user call window.

[0050] 203: This corresponds to the area of ​​the first user's call window being larger than the area of ​​the second user's call window.

[0051] 204: Perform anti-counterfeiting detection on the user's face in the first user's call window.

[0052] It is understood that the electronic device can downsample the first image to be detected to obtain a second image to be detected, wherein the resolution of the second image to be detected is lower than that of the first image to be detected, and the second image to be detected includes facial key points. Then, the electronic device can determine the matching region corresponding to the user's face in the first user window from the second image to be detected based on the facial key points. Furthermore, the electronic device can determine the first face region corresponding to the matching region from the first image to be detected based on the resolution relationship between the first and second images to be detected; and perform anti-spoofing detection on the image of the first face region in the first image to be detected to obtain the anti-spoofing detection result. Thus, by downsampling the first image to be detected to identify the matching region corresponding to the user's face in the first user window, the computing resources of the electronic device can be further saved. Moreover, by determining the first face region corresponding to the matching region from the first image to be detected based on the resolution relationship between the first and second images to be detected, and then performing anti-spoofing detection on the image of the first face region, the accuracy of face anti-spoofing detection can be improved by utilizing the detailed features in the image of the first face region.

[0053] In some alternative implementations, the electronic device can input a second image to be detected into an AI model, in which facial keypoint features related to facial keypoints can be extracted from the second image to be detected. Based on the facial keypoint features, the coordinates of the facial keypoints in the image coordinate system corresponding to the second image to be detected are predicted, and based on the coordinates of the facial keypoints in the image coordinate system corresponding to the second image to be detected, the matching region corresponding to the user's face in the first user window is determined.

[0054] In some alternative implementations, the electronic device can directly perform anti-counterfeiting detection on the first face region in the first image to be detected, or it can extract the image of the first face region from the first image to be detected and perform anti-counterfeiting detection on the image of the first face region. This application embodiment does not make specific limitations.

[0055] It is understandable that the electronic device can determine a second face region based on the first face region, where the area of ​​the second face region is larger than that of the first face region. Anti-spoofing detection is then performed on the image of the second face region in the first image to be detected, yielding the anti-spoofing detection result. Thus, by enlarging the first face region to obtain the second face region for anti-spoofing detection, the computational resources of the electronic device can be saved while increasing the facial features of the user's face in the first user's call window, further improving the accuracy of face anti-spoofing detection.

[0056] In some alternative implementations, the electronic device can directly perform anti-counterfeiting detection on the second face region in the first image to be detected, or it can extract the image of the second face region from the first image to be detected and perform anti-counterfeiting detection on the image of the second face region. This application embodiment does not make specific limitations.

[0057] It is understood that the electronic device can extract the coordinates of each pixel corresponding to the facial contour of the user's face in the image coordinate system corresponding to the second face region image from the image of the second face region. Furthermore, the electronic device can extract the image data of adjacent pixels from the image of the second face region. Then, the electronic device can determine the forgery confidence level of the user's face in the image of the second face region as a forged face based on at least one of the coordinates of each pixel corresponding to the facial contour of the user's face in the image of the second face region in the image coordinate system corresponding to the second face region image and the image data of adjacent pixels; if the forgery confidence level of the user's face in the image of the second face region is greater than the forgery confidence threshold, then the user's face in the image of the second face region is determined to be a forged face.

[0058] Understandably, in practical applications, electronic devices can perform the aforementioned processing on multiple video frames (i.e., the first image to be detected) in a video call, and based on the forgery confidence of the forged face corresponding to multiple video frames, comprehensively determine whether the caller in the video call is a caller with an illegitimate purpose. This avoids the randomness of a single video frame and can further improve the accuracy of anti-spoofing detection. Furthermore, electronic devices can also determine whether the caller is a caller with an illegitimate purpose based on the continuity of adjacent video frames among multiple video frames, thus also avoiding the randomness of a single video frame and further improving the accuracy of anti-spoofing detection.

[0059] In some alternative implementations of the first aspect, the image data includes at least one of the following: color, texture, and depth.

[0060] In some optional implementations of the first aspect, corresponding to a resolution of M*N for the first image to be detected, downsampling the first image to be detected to obtain a second image to be detected includes: determining a first ratio of M to a preset length and a second ratio of N to a preset width; corresponding to a first ratio being greater than the second ratio, determining the length of the second image to be detected based on the ratio of M to the first ratio and determining the width of the second image to be detected based on the ratio of N to the first ratio.

[0061] It is understandable that if the first user window includes multiple user faces, the electronic device can determine the first face region corresponding to each user face from the first image to be detected; and perform anti-spoofing detection on the largest first face region among the multiple user faces to obtain the anti-spoofing detection result. Thus, for multiple user faces within the first user window, anti-spoofing detection is only performed on the user face with the largest face region, which can further save the computing resources of the electronic device.

[0062] The following is about Figure 2 Step 204 will be described in detail, such as Figure 3 The diagram illustrates a process for anti-spoofing detection of a user's face in the first user's call window. Figure 3 As shown:

[0063] First, the electronic device can perform size normalization processing on the first image to be detected (Step 1: Size Normalization).

[0064] In some specific implementations, the smaller the image resolution, the fewer the number of pixels and the less detail the image contains. Therefore, in order to reduce the amount of data processing required for anti-counterfeiting detection while ensuring the detail contained in the image, the electronic device can downsample the first image to be detected. For example, the first detection image with a resolution of 1106*2450 can be downsampled to a second image to be detected with a resolution of 640*480, thereby achieving size normalization processing of the first image to be detected.

[0065] Next, the electronic device can perform face detection on the second image to be detected, and determine the matching region from the second image to be detected (Step 2: Face Detection (Visual Platform)). The second image to be detected includes facial key points, such as eyes, nose, mouth and other facial key points.

[0066] In some specific implementations, the electronic device may include a visual platform, which may include an AI model for face recognition. Therefore, the AI ​​model can extract facial key point features related to facial key points from a second image to be detected. Based on the facial key point features, the coordinates of the facial key points in the image coordinate system corresponding to the second image to be detected are predicted. And based on the coordinates of the facial key points in the image coordinate system corresponding to the second image to be detected, the region to be matched (e.g., Figure 3 The area indicated by the thin dashed line is also called the face frame.

[0067] Then, the electronic device can perform face bounding box position coordinate mapping to determine the face region corresponding to the region to be matched from the first image to be detected (Step 3: face bounding box position coordinate mapping).

[0068] As mentioned earlier, due to the lack of smoothness in the facial contours of counterfeit faces and the significant differences in color, texture, and depth between adjacent pixels, it is necessary to map the matching region determined from the second image to the first image to improve the accuracy of face anti-spoofing detection. This is based on the resolution correspondence between the first and second images to be detected, i.e., the transformation relationship between the image coordinate systems corresponding to the first and second images. This allows for the determination of the face region (i.e., the first face region mentioned above) from the first image. Subsequently, face anti-spoofing detection can be performed based on the fine-grained features corresponding to the face region in the first image.

[0069] Furthermore, electronic devices can expand the background area based on the face area (Step 4: Background Area Expansion).

[0070] It is understandable that, in order to enhance facial features in the face region, the face region can be expanded by extending the background region in the first image to be detected, resulting in an expanded face region (i.e., the second face region mentioned above). For example, ... Figure 3 The area indicated by the thin dashed box is expanded to the area indicated by the thick dashed line. This avoids the situation where the user's face in the first user call window is not standard, resulting in the determination of the face area lacking some facial features. This can further improve the accuracy of subsequent face anti-spoofing detection.

[0071] Furthermore, electronic devices can perform cutout and normalization processing on the expanded face area (Step 5: Cutout and Normalization Processing).

[0072] Understandably, to reduce the data processing load of electronic devices during face anti-spoofing detection, the electronic device can perform image cutout of the extended face region. This involves extracting the extended face region from the first image to be detected and normalizing the extracted extended face region (i.e., the image of the second face region mentioned above) to a preset resolution. Specifically, when the resolution of the extracted extended face region is lower than the preset resolution, the electronic device can upsample the extracted extended face region, padding the insufficient areas with 0 pixels. When the resolution of the extracted extended face region is higher than the preset resolution, the electronic device can downsample the extracted extended face region. In this way, by unifying the resolution of the extracted extended face region, the impact of resolution differences on the accuracy of face anti-spoofing detection can be reduced, further improving the accuracy of subsequent face anti-spoofing detection.

[0073] Next, the electronic device can perform forgery detection model inference on the normalized expanded face region to determine the forgery confidence of the normalized expanded face region (Step 6: Forgery Detection Model Inference (Visual Platform)).

[0074] In some specific implementations, the aforementioned visual platform may also include a forgery detection model for face forgery detection, such as an edge detection model. This forgery detection model can then be used to predict forgery in the normalized expanded face region. For example, it can determine the scores corresponding to the smoothness of the facial contour, the color, texture, and depth of adjacent pixels in the normalized expanded face region, and predict the forgery confidence of the normalized expanded face region based on the weights corresponding to the smoothness of the facial contour and the color, texture, and depth of adjacent pixels.

[0075] It is understandable that the first image to be detected is any video frame in the call video. For other video frames in the call video, the electronic device can also perform the above processing to obtain multiple normalized forgery confidence scores for the expanded face regions.

[0076] Finally, the electronic device can perform confidence fusion and heuristic calculation on the forgery confidence of multiple normalized expanded face regions (Step 7: Confidence Fusion and Heuristic Calculation (System Manager)).

[0077] In some optional implementations, the electronic device may include a system manager that provides risk protection services. After predicting the forgery confidence scores of multiple normalized augmented face regions, these scores can be fused together. For example, the average or sum of the forgery confidence scores of multiple normalized augmented face regions can be calculated to determine the forgery confidence score corresponding to the user's face in the first user call window. When the forgery confidence score corresponding to the user's face in the first user call window is greater than a forgery confidence score threshold, the electronic device can determine that the user's face in the first user call window is a forged face, thus enabling the electronic device to perform risk protection. For example, the system manager of the electronic device can display a prompt text box to alert the user that the person in the video call is at risk of face swapping.

[0078] The following is about Figure 2 The specific procedures for the mentioned video call methods will be described in further detail. For example... Figure 4 The diagram illustrates a flowchart of another video call method that can be performed by an electronic device.

[0079] Specifically, the video call method may include:

[0080] 401: Video frame capture.

[0081] In some optional implementations, during a video call, the electronic device can acquire all or part of the call video and store it in the device's buffer (referred to as the raw video frame buffer). Then, by detecting the status of at least one of the sensing module, camera, and microphone, it can determine whether the video call is active. When the video call is active, the electronic device can capture video frames from the raw video frame buffer.

[0082] It is understandable that in some optional implementations, each video frame in a video call may include four color channels: red (R), green (G), blue (B), and alpha (A). In other optional implementations, each video frame in a video call may include three color channels: red (R), green (G), and blue (B).

[0083] For ease of explanation, the following description uses RGBA format for each video frame in a video call as an example. When the video call is connected, the electronic device can capture video frames from the video call based on a preset time interval, such as capturing 10 frames per second, or 8 frames per second.

[0084] 402: Face detection.

[0085] It is understandable that after capturing video frames, electronic devices can first perform face detection preprocessing on each video frame, and then perform face detection processing on each video frame after face detection preprocessing.

[0086] During the face detection preprocessing of each video frame, since the transparency of the video frame has little effect on subsequent face detection, the electronic device can perform format conversion on each video frame during the face detection preprocessing. For example, the electronic device can filter the transparency of the video frame frame by frame and convert the RGBA format video frame to the RGB format video frame to reduce the amount of data processing for subsequent face detection and improve processing efficiency.

[0087] Furthermore, the smaller the resolution of a video frame, the fewer pixels it contains, and the less detail it contains. Therefore, to reduce the amount of data processing required for subsequent face detection while preserving the detail in the video frames, electronic devices can normalize each video frame after format conversion. For example, the electronic device can pre-set the resolution for normalization, and for video frames whose resolution does not meet the resolution threshold, the electronic device can downsample or upsample the video frame to achieve normalization of each video frame after format conversion.

[0088] In some specific implementations, the resolution threshold is assumed to be 640*480. If the resolution of the format-converted video frame is M*N, and M≠640, N≠480, where M and N are positive numbers, the electronic device can calculate M / 640 and N / 480, and take the maximum of the two as the sampling parameter Q. This sampling parameter Q is used to normalize the format-converted video frame. For example, the width (M) and height (N) of the video frame are both divided by the sampling parameter Q to normalize the format-converted video frame.

[0089] Understandably, electronic devices can also be pre-set to not perform subsequent face detection, such as 320*240. If the resolution of the video frame after format conversion is M*N, and M<320, or N>240, then the electronic device can filter the video frame, that is, not perform subsequent face detection on the video frame.

[0090] During the face detection processing of each video frame after preprocessing, the electronic device can input the normalized video frames into the face detection module to determine the coordinates of facial key points in the corresponding image coordinate system of each normalized video frame. In some specific implementations, the face detection module may include AI models such as a multi-task cascaded convolutional neural network (MTCNN) detector or a single-shot scale-invariant face detector (S3FD). Furthermore, the coordinates of facial key points in the corresponding image coordinate system of each normalized video frame can be stored in the form of a list.

[0091] In some specific implementations, the face detection module can extract facial key point features (such as eyes, nose, and mouth) related to facial key points from each normalized video frame. Furthermore, it can predict the coordinates of these facial key points in the corresponding image coordinate system of each normalized video frame based on these features. Then, based on the coordinates of the facial key points in the corresponding image coordinate system of each normalized video frame, it can determine the region to be matched from the normalized video frames. Finally, the electronic device can determine the facial region corresponding to the region to be matched from the format-converted video frames based on the sampling parameter Q.

[0092] 403: The result returned is 1 or 2 faces.

[0093] It is understandable that if the electronic device detects 0 or 3 or more faces in a frame of the call video, it can be determined that it does not meet the preset application scenario, and thus it can return to step 401 to resample the video frame.

[0094] If the electronic device detects two faces in a frame of the call video, it can proceed to step 404 to determine the area of ​​the user call window where each face region is located in order to filter the face location.

[0095] If the electronic device detects only one face in a frame of the video call, for example, if the camera of the electronic device is damaged and can only display the face of the person being called, then the process can proceed to step 404 or step 405.

[0096] 404: Face location filtering.

[0097] The following describes the face location filtering process, assuming a result of two faces. It can be understood that when an electronic device detects two faces in a frame of a video call, it can determine whether the face region belongs to the local window (i.e., the local user's call window displaying the user's face). If the result is yes, the face region is filtered. If the result is no, subsequent face anti-spoofing detection is performed on the face region.

[0098] It is understood that the local user's call window is displayed by default in the upper right corner of the remote user's window. Therefore, in some specific implementations, the electronic device can pre-set a height ratio threshold, such as 1 / 5. Furthermore, the electronic device can determine the coordinates of the center point of the face region in the image coordinate system corresponding to the format-converted video frame based on the coordinates of the four corners of the face region in the image coordinate system. If the electronic device determines that the ordinate of the center point of the face region in the image coordinate system corresponding to the format-converted video frame is less than the height ratio threshold, it returns to step 401 to resample the video frame. If the electronic device determines that the abscissa of the center point of the face region in the image coordinate system corresponding to the format-converted video frame is greater than the height ratio threshold, it proceeds to step 405 to perform face size filtering.

[0099] 405: Face size filtering.

[0100] As mentioned earlier, it's understandable that the smaller the resolution of a video frame, the fewer pixels it contains, and therefore the less detail it includes. Thus, to reduce the amount of data processing required for subsequent face detection while preserving the detail within the video frame, electronic devices can perform size filtering on the facial regions in the converted video frames. Facial regions with a resolution less than 128*128 are not subject to subsequent face spoofing detection.

[0101] For example, when multiple facial regions are detected from the remote user's call window, face anti-spoofing detection can be performed only on facial regions with a resolution greater than 128*128.

[0102] 406: Face ROI normalization.

[0103] Understandably, to reduce the data processing load of electronic devices during face spoofing detection, the electronic device can crop the face region, also known as the region of interest (ROI), that is, extract the face ROI from the converted video frames. Furthermore, to standardize the resolution of the extracted face ROI and reduce the impact of resolution differences on the accuracy of face spoofing detection, the extracted face ROI can be normalized, that is, the resolution of the extracted extended face region can be adjusted to a preset resolution. For example, the preset resolution could be 256*256.

[0104] 407: Face anti-spoofing detection model inference.

[0105] It is understandable that electronic devices can use a forgery detection model to infer the forgery confidence of the normalized face ROI.

[0106] For example, the electronic device determines the scores corresponding to the smoothness of the facial contour, the color, texture, and depth of adjacent pixels in the normalized face ROI, and predicts the forgery confidence of the normalized face ROI based on the weights corresponding to the smoothness of the facial contour, the color, texture, and depth of adjacent pixels.

[0107] 408: Has a sufficient number of valid frames been detected?

[0108] It is understandable that, due to the dynamic nature of the face of the person in a video call, a single frame cannot reflect the true state of the facial ROI. For example, network lag can cause misalignment of the facial ROI, leading to errors in the anti-spoofing detection results. Therefore, electronic devices can process multiple video frames in the video call using steps 401 to 407 to detect a sufficient number of valid video frames, thereby further improving the accuracy of facial anti-spoofing detection.

[0109] 409: Multi-frame detection confidence fusion.

[0110] In some alternative implementations, after predicting the forgery confidence scores of multiple normalized face ROIs, these scores can be fused. For example, the average or sum of the forgery confidence scores of multiple normalized face ROIs can be calculated to determine the forgery confidence score corresponding to the user's face in the selected remote user call window. For instance, when the forgery confidence score corresponding to the user's face in the selected remote user call window is greater than 0.5, the electronic device can determine that the user's face in the remote user call window is a forged face; when the forgery confidence score corresponding to the user's face in the selected remote user call window is less than or equal to 0.5, the electronic device can determine that the user's face in the remote user call window is a non-forged face, i.e., a real face.

[0111] It is understood that the gesture matching method provided in this application embodiment can be applied to electronic devices. The hardware structure of the electronic device to which the gesture matching provided in this application embodiment is applicable will be described exemplarily below.

[0112] In some specific implementations, the electronic device can be a device that provides video calling functionality, such as a mobile phone, tablet, or laptop computer, as mentioned above, or a device with a video calling app installed.

[0113] like Figure 5 As shown, the electronic device 500 may include a processor 510, an external memory interface 520, an internal memory 521, a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna, a wireless communication module 550, an audio module 560, a speaker 560A, a receiver 560B, a microphone 560C, a camera 570, etc.

[0114] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0115] Processor 510 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0116] In some alternative implementations, the processor 510 may execute the video call method mentioned in the embodiments of this application.

[0117] Specifically, the processor 510 can acquire the call video and determine at least one frame of image to be detected from the call video. Each frame to be detected includes a first user call window and a second user call window. As mentioned earlier, the remote user call window displays the face of the call participant by default, while the local user call window displays the user's face by default. Therefore, the electronic device can determine the areas of the first and second user call windows respectively, and perform anti-spoofing detection on the user's face within the larger user call window. This avoids detecting the user's face in the local user call window, saving the electronic device's computing resources.

[0118] Electronic devices can achieve shooting functions through ISP, camera 570, video codec, GPU and application processor.

[0119] The ISP (Image Signal Processor) is used to process data fed back from the camera 570. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 570.

[0120] Camera 570 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 570, where N is a positive integer greater than 1.

[0121] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0122] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0123] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as gesture recognition, image recognition, facial recognition, speech recognition, and text understanding.

[0124] The above describes the possible hardware structures of electronic devices. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0125] The system architecture of the electronic device to which the video call method provided in the embodiments of this application is applicable will be described exemplarily below.

[0126] Figure 6 This is a system architecture block diagram of an electronic device according to an embodiment of this application.

[0127] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system architecture of an electronic device is divided into three layers, from top to bottom: the cloud layer, the application layer, and the framework layer.

[0128] Continue as Figure 6 As shown, the cloud side can include a cloud platform, also known as a configuration service management console. The cloud platform includes a detection parameter configuration module and a video call APP whitelist module.

[0129] The detection parameter configuration module can be used to store pre-set parameters such as the resolution for normalization, the resolution for not performing subsequent face detection, and the height ratio threshold.

[0130] The video call whitelist module can be used to store the installation packages of video call apps that support face anti-spoofing detection.

[0131] Continue as Figure 6 As shown, the application layer can include third-party applications (APP), perception modules, system managers, and intelligent vision platforms.

[0132] Third-party applications (APPs) can include applications such as adding friends and video calls.

[0133] The perception module can include an activity perception module.

[0134] The behavior awareness module can be used to detect whether a video call is connected.

[0135] In some specific implementations, when the perception module detects that the call recipient is a newly added friend whose duration is less than or equal to t (Step 0: Add friend (duration t)), and a video call is detected within the time period from t to t+α, such as a video call incoming, a video call being answered, or a video call being hung up (Step 1: Video call in the time period [t, t+α]), the perception module can send third-party video call activity behavior perception information to the system manager (Step 2: Third-party video call activity behavior perception).

[0136] The system manager can include an AI face-swapping detection module, a fraud risk identification and handling module, and a full-chain identification module.

[0137] Specifically, the AI ​​face-swapping detection module can include a detection scheduling submodule and a interception submodule. Furthermore, the detection invocation submodule can be used to send a third-party call activity event subscription request to the perception module, and respond to the third-party video call activity behavior perception information sent by the perception module when the perception module detects a third-party call activity event.

[0138] The interception submodule obtains video call layer screenshots by calling the CaptureLayer interface of the SurfaceControl class in the system framework. This method filters out interference from irrelevant layers such as pop-ups, input methods, notifications, status bars, and floating windows that could affect the detection algorithm.

[0139] The fraud risk identification and handling module can be used to provide risk protection services. When the confidence level of a user's face in the remote user's call window is greater than the confidence level threshold for forgery, the user's face in the first user's call window can be determined to be a forged face.

[0140] In some specific implementations, the fraud risk identification and handling module can receive confidence result feedback sent by the smart module (Step 7: Confidence Result Feedback). When the confidence result feedback indicates that the user's face in the remote user's call window is a fake face, a prompt instruction can be sent to the window manager to alert the user that there is a risk of face swapping in the call during this video call.

[0141] The end-to-end identification module combines the risk detection results of AI face swapping during video calls with subsequent user risky behaviors, such as installing unknown payment applications, screen sharing, and payment, to identify and determine whether the current user is at risk and issue a warning.

[0142] The intelligent vision platform (i.e., the vision platform mentioned above) can include a forgery detection model inference module and a face ROI detection / outlining module.

[0143] The forgery detection model inference module can include a forgery detection model for face forgery detection, such as an edge detection model. This forgery detection model can be used to predict forgery on normalized face ROIs. For example, it can determine the scores corresponding to facial contour smoothness, color, texture, and depth of adjacent pixels in the normalized face ROI, and predict the forgery confidence of the normalized face ROI based on the weights corresponding to facial contour smoothness, color, texture, and depth of adjacent pixels.

[0144] The face ROI detection / masking module can include an AI model for face recognition. This AI model can extract facial keypoint features (e.g., eyes, nose, mouth) from normalized video frames, and predict the coordinates of these features in the corresponding image coordinate system of each normalized video frame. Then, based on these coordinates, the region to be matched can be determined from the normalized video frames. Finally, the face region corresponding to the region to be matched can be determined from the format-converted video frames.

[0145] The face ROI detection / outlining module can also be used to extract face ROIs from video frames after format conversion.

[0146] The framework layer provides the application programming interface (API) and programming framework for the application layer applications. The framework layer includes some predefined functions. Continuing... Figure 6 As shown, the framework layer may include a window manager (WindowManager) and an image compositor (SurfaceFinger).

[0147] The window manager is used to manage windowed applications. It can obtain the screen size, determine the presence of a status bar, lock the screen, and capture screenshots. Furthermore, the window manager can provide screen recording interfaces for application-level applications, such as SurfaceControlEx.

[0148] In some specific implementations, the window manager can send a command to the image compositor to capture video layer data (Step 4: Capture Video Layer Data) to obtain the layers corresponding to the remote user's call window and the local user's call window, thereby performing subsequent face anti-spoofing detection on the remote user's call window.

[0149] An image synthesizer can be used to determine the positional relationship between different user call windows. For example, the local user's call window is displayed by default in the upper right corner of the large user window.

[0150] In some cases, the embodiments disclosed in this application may be implemented in hardware, firmware, software, or any combination thereof.

[0151] The embodiments disclosed in this application can also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions can be distributed via a network or via other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, magnetic disks, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0152] Embodiments of this application can be implemented as computer programs or program code that execute on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0153] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0154] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0155] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0156] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0157] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A video call method applied to an electronic device, comprising: obtaining a first to-be-detected image in a video call process, the first to-be-detected image comprising a first user call window and a second user call window; detecting that an area of the first user call window is greater than an area of the second user call window; performing anti-spoofing detection on a user face in the first user call window.

2. The method of claim 1, wherein, The anti-spoofing detection on the user face in the first user call window comprises: down-sampling the first to-be-detected image to obtain a second to-be-detected image, wherein a resolution of the second to-be-detected image is less than a resolution of the first to-be-detected image, and the second to-be-detected image comprises face key points; determining a to-be-matched region corresponding to the user face in the first user window from the second to-be-detected image according to the face key points; determining a first face region corresponding to the to-be-matched region from the first to-be-detected image according to a resolution relationship between the first to-be-detected image and the second to-be-detected image; and performing anti-spoofing detection on an image of the first face region in the first to-be-detected image to obtain an anti-spoofing detection result.

3. The method of claim 2, wherein, The method further comprises: determining a second face region from the first face region, wherein an area of the second face region is greater than an area of the first face region; and performing anti-spoofing detection on an image of the second face region in the first to-be-detected image to obtain an anti-spoofing detection result.

4. The method of claim 3, wherein, The anti-spoofing detection on the user face in the first user call window comprises: determining coordinates of each pixel point corresponding to a face contour of the user face in an image coordinate system corresponding to the image of the second face region, and image data of adjacent pixel points in the image of the second face region from the image of the second face region; determining a spoofing confidence that the user face in the image of the second face region is a spoofed face according to at least one of the coordinates of each pixel point corresponding to the face contour of the user face in the image coordinate system corresponding to the image of the second face region and the image data of the adjacent pixel points; and determining that the user face in the image of the second face region is a spoofed face corresponding to a spoofing confidence of the user face in the image of the second face region being greater than a spoofing confidence threshold.

5. The method of claim 4, wherein, The image data comprises at least one of the following: color, texture, depth.

6. The method of claim 2, wherein, A resolution of the first to-be-detected image is M*N, The down-sampling of the first to-be-detected image to obtain the second to-be-detected image comprises: determining a first ratio of the M to a preset length and a second ratio of the N to a preset width; determining a length of the second to-be-detected image based on a ratio of the M to the first ratio and a width of the second to-be-detected image based on a ratio of the N to the first ratio corresponding to the first ratio being greater than the second ratio.

7. The method according to any one of claims 1 to 6, characterized in that, Corresponding to the first user window comprising a plurality of user faces, The anti-spoofing detection on the user face in the first user call window comprises: From the first image to be detected, determine the first face area corresponding to each user face in the plurality of user faces; Perform anti-counterfeiting detection on the first face area with the largest area in the first face area corresponding to each user face in the plurality of user faces, to obtain an anti-counterfeiting detection result.

8. An electronic device, comprising: Comprise: A memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the one or more processors of the electronic device, for executing the method of any one of claims 1-7.

9. A readable storage medium, characterized by, The readable storage medium has instructions stored thereon, which, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-7.

10. A computer program product, characterised in that, The computer program product includes computer instructions, which, when executed by an electronic device, cause the electronic device to execute the computer program code of the method of any one of claims 1-7.

Images