Interface rotation control method and terminal device

Abstract

The embodiment of the present application provides an interface rotation control method and a terminal device. Among them, an interface rotation control method includes: in response to a terminal posture change event, starting the front camera of the terminal to collect data information; processing the data information to determine the target face angle; wherein, if the target face is missing, the data information related to the target face can be supplemented during processing to determine the target face angle based on the supplemented target face related information; according to the target face angle, determine whether to control the rotation of the interface displayed by the terminal. By adopting this solution, the interface rotation control in the scenario where the entire user face cannot be collected can be effectively optimized, thereby improving the user terminal usage experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an interface rotation control method and a terminal device. Background Art

[0002] With the development of technology, the screens of terminals such as smartphones and tablets all have an automatic rotation function. When the automatic rotation function is turned on, the front camera of the terminal can be used to capture the face image of the terminal user to identify and analyze the face direction, and then determine the rotation direction of the interface displayed on the terminal screen based on the face direction. However, due to the limitation of the acquisition range of the front camera, it is easy to capture incomplete face images in certain usage scenarios, and the existing solutions have poor recognition effects for incomplete face images, which can easily cause interface rotation errors and bring a bad experience to users. Summary of the invention

[0003] In view of this, multiple aspects of the present application provide an interface rotation control method and a terminal device to optimize the interface rotation control in scenarios where it is not possible to capture all of the user's faces or in scenarios where multiple faces are captured.

[0004] In one embodiment of the present application, a method for controlling interface rotation is provided. The method comprises:

[0005] In response to a terminal posture change event, starting a front camera of the terminal to collect data information;

[0006] Processing the data information to determine the target face angle; wherein, if the target face is missing, the information related to the target face in the data information can be supplemented during processing, so as to determine the target face angle based on the supplemented target face related information;

[0007] According to the target face angle, it is determined whether to control the interface displayed by the terminal to rotate.

[0008] In another embodiment of the present application, a method for controlling interface rotation is also provided. The method comprises:

[0009] In response to a terminal posture change event, starting a front camera of the terminal to collect data information;

[0010] When it is determined based on the data information that there are multiple faces, selecting a target face from the multiple faces according to the distance information included in the data information;

[0011] According to the target face, it is determined whether to control the interface displayed by the terminal to rotate.

[0012] In another embodiment of the present application, a method for controlling interface rotation is provided. The method comprises:

[0013] In response to a terminal posture change event, a grayscale image and a corresponding depth image are acquired through a front camera of the terminal; wherein the grayscale image includes at least one human face, and the depth image includes the distance between each human face and the terminal;

[0014] Processing the grayscale image and the depth image to determine the angle of the target face;

[0015] Determining whether to trigger rotation of an interface displayed on the terminal according to the target face angle;

[0016] Among them, the target face is one of the at least one face; if any of the at least one face is missing, the information related to the missing face in the grayscale image and the depth image can be supplemented during processing to determine the target face angle based on the supplemented face-related information.

[0017] In another embodiment of the present application, a terminal device is provided. The terminal device includes:

[0018] Screen, used to display the interface;

[0019] A front camera is arranged on the side of the terminal device where the screen is located;

[0020] The control device is communicatively connected to the front camera and is used to implement the steps in the interface rotation control method provided in each embodiment of the present application.

[0021] In a technical solution provided by an embodiment of the present application, in response to a terminal posture change event, the front camera of the terminal is activated to collect data information, and the target face angle can be determined by processing the data information; wherein, if the target face is missing, the information related to the target face in the data information can be supplemented during processing to determine the target face angle based on the supplemented target face related information; thereafter, whether to control the rotation of the interface displayed by the terminal is determined based on the target face angle. With this solution, even in scenarios where the entire user's face cannot be collected, the high accuracy of face recognition detection and the calculation of the angle between the face and the terminal can be guaranteed through data supplementation, thereby effectively optimizing the interface rotation control and improving the user's terminal usage experience.

[0022] In another technical solution provided by the embodiment of the present application, in response to the terminal posture change event, the front camera of the terminal is activated to collect data information, and when it is determined that there are multiple faces based on the data information, a target face can be selected from the multiple faces according to the distance information contained in the data information, and then whether to control the rotation of the interface displayed by the terminal is determined according to the target face. It can be seen that this solution can solve the problem of interface rotation control in multi-user face scenarios, which helps to improve the user experience of the terminal.

[0023] In another technical solution provided by the embodiment of the present application, in response to the terminal posture change event, a grayscale image containing at least one face and a corresponding depth map can be obtained through the front camera of the terminal, and the depth map contains the distance between each face and the terminal; then, by processing the grayscale image and the depth map, the angle between the target face and the terminal can be obtained, so as to determine whether to trigger the rotation of the interface displayed by the control terminal according to the angle between the target face and the terminal; wherein the above-mentioned target face is one of the at least one face; if any face of the at least one face is missing, when processing the grayscale image and the depth map, the information related to the missing face in the grayscale image and the depth map can be supplemented with data. With this solution, even in scenarios where all the user's faces cannot be collected, the high accuracy of face recognition detection and the angle calculation between the face and the terminal can be guaranteed through data supplementation, so that the interface rotation control can be effectively optimized. In addition, the interface rotation control problem in multi-user face scenarios can also be solved. In summary, it can be seen that the application scheme of the present application has a wider range of applicable scenarios and fewer limitations, which helps to improve the user terminal usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1a A principle flow chart of screen interface rotation provided for an exemplary embodiment of the present application;

[0026] Figure 1b A schematic diagram of a face image provided for an exemplary embodiment of the present application;

[0027] Figure 2 A flowchart of an interface rotation control method provided by an exemplary embodiment of the present application;

[0028] Figure 3 A network architecture diagram of a processing model (convolutional network model) provided for an exemplary embodiment of the present application;

[0029] Figure 4A schematic diagram of a face heat map provided for an exemplary embodiment of the present application;

[0030] Figure 5 and Figure 6 A flowchart of an interface rotation control method provided for other exemplary embodiments of the present application;

[0031] Figures 7 to 9 A schematic diagram of an application scenario of interface rotation control provided in an embodiment of the present application;

[0032] Figures 10 to 12 A schematic diagram of the structure of the interface rotation control device provided in an embodiment of the present application;

[0033] Fig.13 A schematic diagram of the structure of a terminal device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] Currently on the market, smart phones, tablet computers and other terminals have automatic rotation functions turned on, as shown in Figure 1a , when the sensor detects that the posture of the terminal device has changed, the front camera on it will be activated to capture the image in front of the terminal. The image contains the face of the user corresponding to the terminal. The face direction is obtained by performing face recognition analysis on the image, and then the rotation direction of the interface displayed on the terminal screen is determined according to the face direction. However, since the images captured in scenes such as dim light (darkness) and backlighting are often unclear, this also leads to the problems of low recognition accuracy and long time consumption in the subsequent image recognition analysis scheme to identify the face direction to determine the rotation direction of the interface. In response to this problem, an existing solution is to use a ToF (Time of Flight) camera as the front camera for the terminal device. Although the ToF camera is suitable for all scenes and the image effect captured in scenes such as dim light and backlighting is much better than that of the RGB camera, the field of view FoV (Field of View, specifically the range that the lens can cover) of the ToF camera is generally 78°, which is smaller than the 90°~100° of the RGB camera. Therefore, it is easier to capture images containing only part of the face in certain scenes without usage restrictions (such as Figure 1b As shown in the figure, traditional face recognition algorithms have poor recognition effects on images containing only part of the face and are prone to misrecognition, making it difficult to ensure the accuracy of face direction determination.

[0035] In order to solve the above problems, the present application provides a technical solution for interface rotation control. The basic idea of ​​this solution is to obtain the face image and corresponding depth map of the user in front of the terminal through the front camera of the terminal, and input the face image and depth map into the trained processing model. The processing model is obtained by improving the model built based on the Center-net algorithm. Finally, according to the output results of the processing model (such as face heat map, face angle information relative to the terminal, etc.), the interface rotation control is realized. Through the above, it can be ensured that even if the image containing the entire face of the user cannot be collected in certain scenarios and / or the image clarity is low, the accuracy of the calculation of the face direction (or the angle of the face relative to the terminal) can be effectively ensured, thereby ensuring the accuracy of the interface rotation control; in addition, the interface rotation control problem in multiple face scenes can also be solved. For detailed description of the face image and depth map, please refer to the relevant content below.

[0036] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0037] In the embodiments of the present application, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar phases with basically the same functions and effects. For example, the first convolution result and the second convolution result are only used to distinguish different convolution results, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0038] It should be noted that the words "exemplary" or "for example" in this application are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way. In addition, "at least one" in this application refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, B exists alone, wherein A and B can be singular or plural, etc. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b and c can represent: a, b, c, a, b and c, a and b, a and c, b and c.

[0039] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0040] Figure 2 This is a flowchart of an interface rotation control method provided in one embodiment of the present application. The execution subject of the method is a terminal, more specifically, the terminal is a portable (such as handheld) movable electronic device, such as a smart phone, tablet computer, POS machine, netbook, etc. For a detailed description of the terminal, please refer to the relevant content of the terminal device embodiment provided in the following text application. Figure 2 As shown, the interface rotation control method includes the following steps:

[0041] 101. In response to a terminal posture change event, start a front camera of the terminal to collect data information;

[0042] 102. Process the data information to determine the target face angle; if the target face is missing, supplement the information related to the target face in the data information during processing, so as to determine the target face angle based on the supplemented target face related information;

[0043] 103. Determine whether to control the interface displayed on the terminal to rotate according to the target face angle.

[0044] In the embodiment of the present application, the automatic screen rotation function of the terminal is turned on. And a posture sensor may be provided on the terminal, wherein the posture sensor may include a motion sensor such as a three-axis gyroscope and a three-axis accelerometer. Accordingly, in the above 101, the posture sensor provided on the terminal may be used to detect whether the posture changes.

[0045] For example, see Figure 7 When the user lies prone (or flat or sitting) to use the terminal, the terminal can be considered to be in a vertical state relative to the ground. When the user lies sideways to use the terminal, the terminal's posture relative to the ground changes from a vertical state to an approximately horizontal state. At this time, the front camera of the terminal can be activated to collect data information in front of the terminal.

[0046] In specific implementation, the front camera of the terminal is a depth camera with a ranging function, such as but not limited to a ToF camera, a structured light camera, or a binocular camera. Preferably, in the present application, the front camera is a ToF camera, because the ToF camera is more suitable for dynamic scenes than other cameras, and has a longer measurement distance, is less affected by light environments such as strong light and dark light, and has a simpler data output process. The front camera of the terminal can collect data information in front of the terminal, which may include face information and distance information.

[0047] For example, the front camera of the terminal is a ToF camera. During the shooting process, the ToF camera emits light (such as infrared light) outward, and the light is reflected back after encountering the shooting object such as the user and is collected by the ToF camera, or the light emitted by the user and other objects themselves is collected by the ToF camera. After the ToF camera images the collected light, a face image containing face information (such as Figure 1b ), the face image is a color image (specifically, it can be an infrared image). In addition, the ToF camera can also use the time difference or phase difference from the light emission to the reflection back to the ToF camera to calculate the distance information between the user and the camera. The distance information is also regarded as the distance information from the subject to the terminal, thereby generating an image containing the distance information. The image including the distance information is the depth map (also called the depth map). In the above, the ToF camera starts shooting only when it detects a change in the posture of the terminal (such as a change from a vertical state to a horizontal state relative to the ground), which can avoid unnecessary invalid shooting.

[0048] Based on the above example content, in one possible implementation, the above 102 “processing the data information to determine the target face angle” may specifically include:

[0049] 1021. Based on the data information, obtain a face map and a corresponding depth map; the face map includes at least one face, the target face is one of the at least one face, and the depth map includes the distance between each face and the terminal;

[0050] 1022. Process the face image and the depth image to obtain a face heat map and face angle information;

[0051] 1023. Determine the target face angle based on the face heat map and the face angle information.

[0052] In the above 1021, the face image is a grayscale image (IR image) or a color image. Generally, the original face image of the user directly obtained by the front camera is a color image. However, in order to reduce the subsequent calculation amount and the use of terminal memory, this embodiment will convert the color image into a grayscale image for subsequent use. That is, preferably, the above face image is a grayscale image.

[0053] In order to extract comprehensive features with strong expressive power from the face image and the depth image, so as to accurately identify the face and obtain highly accurate heat map and angle information for the face, the technical solution provided in this embodiment can process the face image and the depth image based on artificial intelligence learning algorithm. Therefore, the above 1022 "processing the face image and the depth image to obtain the face heat map and face angle information" can specifically include:

[0054] 10221. Get the processing model;

[0055] 10222. Use the processing model to process the face image and the depth image to obtain the face heat map and the face angle information.

[0056] In the above 10221, the processing model may be a convolutional network model, which may be but is not limited to being obtained by improving and training the original Center-net model, and the Center-net model is a model built based on the Center-net algorithm. Specifically, when training the improved Center-net model, a data enhancement method can be used to obtain a training sample set. Through the data enhancement method, a small amount of existing training samples can be used to generate more training samples for training, thereby improving the model detection accuracy. Commonly used data enhancement methods can be divided into: data enhancement based on geometric transformation, such as translation, shearing, adding noise, scaling, rotation, etc.; data enhancement based on color space transformation, such as adjusting brightness, contrast, saturation, grayscale conversion, etc. In addition, some other data enhancement methods are also included, such as Mosaic data enhancement.

[0057] For example, each training sample can be translated by translation to reveal the face contained in each training sample to different degrees, such as revealing 20% ​​to 50% of the face, thereby generating a training sample set containing only part of the face for training, so as to obtain the processing model described in this embodiment, that is, the convolutional network model.

[0058] The above-mentioned convolutional network model includes at least two convolutional layers with convolution operations and edge expansion operations, wherein the operation parameters corresponding to the edge expansion operations of the two convolutional layers are different. The above-mentioned edge expansion operation (padding) is also called edge zero padding operation, which refers to adding a number of pixels with circle values ​​of "0" to the outer edge of the original image. The operation parameter corresponding to the edge expansion operation is the number of supplementary circles (number of filling circles). If the target face is missing, when the processing model is used to process the face image and the depth map, these two convolutional layers with edge expansion operations are used to realize data supplementation of the information related to the target face in the face image and the depth map.

[0059] Figure 3 The network architecture diagram of the convolutional network model described above is shown in FIG. Figure 3 , the convolutional network model includes a first feature extraction network, a second feature extraction network and an output prediction network. Among them,

[0060] The first feature extraction network is mainly used to perform preliminary pre-feature extraction processing on the input image, which can be but not limited to a Backblone network.

[0061] The above-mentioned second feature extraction network is an extension and expansion after the first feature extraction network, and is used to further process the feature map data output by the first feature extraction network to obtain a more effective feature map that is easy to recognize by the output prediction network, which can be but not limited to ResNet (residual neural network). In the second feature extraction network, the second convolutional layer with a stride of Stride=4 and the third convolutional layer with a stride of Stride=8 downstream thereof are two convolutional layers with convolution operations and edge expansion operations, and the operation parameters corresponding to the edge expansion operation of the second convolutional layer are greater than the operation parameters corresponding to the edge expansion operation of the third convolutional layer.

[0062] The output prediction network includes at least a first network branch and a second network branch. The first network branch is used to detect faces to generate a corresponding face heat map for the detected faces. The white bright area in the face heat map indicates the center point of the face. The white bright area is often a Gaussian circle. Specifically, in the face heat map, the center point of the face is used as the center of the Gaussian circle, and a Gaussian circle with a radius of R is rendered to represent the corresponding face. For example, see Figure 4 The face heat map Heat2 shown in the figure, the white bright area in the face heat map Heat2 indicates Figure 1b The center point of the user's face is shown in FIG. 2 . The face heat map Heat2 is obtained by Figure 3 The convolutional network model shown is Figure 1bThe face heat map Heat2 is obtained by processing the face image shown in the figure. In the face heat map Heat2, the center point of the face falls within the heat map (specifically falls within the expanded area), which enables the subsequent detection of the face and the face direction (i.e., the angle between the face and the terminal) based on the heat map. Among them, the area surrounded by the square box S in the middle of the face heat map Heat2 is equivalent to Figure 4 The face heat map Heat1 is shown in FIG. Figure 4 The face heat map Heat1 shown in the figure is a face heat map of the original Center-net model. Figure 1b The heat map of the face obtained by processing the face image shown in , in which the center point of the face in the face heat map Heat1 is outside the heat map, which makes it impossible to perform face detection based on the heat map in the future. The second network branch is used to predict the roll angle (roll, which is the angle of the side rotation of the face) of the face, and the roll angle of the face can be used as the angle between the face and the terminal. In addition to the above, the output prediction network may also include other network branches, such as a third network branch for regressing and predicting facial feature points (Landmark) of the face, and a fourth network branch for regressing and predicting the size of the face.

[0063] Based on the above content, in a specific implementable solution, the above 10222 "using the processing model to process the face image and the depth image to obtain the face heat map and the face angle information" can be implemented by the following steps:

[0064] S01, performing pre-feature extraction processing on the face image and the depth image respectively to obtain corresponding feature image data;

[0065] S02, performing a convolution operation on the feature map data and then performing an edge expansion operation to obtain a first convolution result;

[0066] S03, performing a convolution operation on the first convolution result and then performing an edge expansion operation to obtain a second convolution result;

[0067] S04, performing a convolution operation on the second convolution result to obtain a third convolution result;

[0068] S05. Based on the third convolution result, obtain the face heat map and the face angle information.

[0069] When implementing it, if combined with Figure 3, the above step S01 can be completed by the first feature extraction network in the convolutional network model. After the first feature extraction network performs pre-feature extraction processing on the face image and the depth image respectively, the output feature image data (including the feature image corresponding to the face image and the feature image corresponding to the depth image) will be used as the input of the second extraction network model, triggering the execution of the second extraction network model to complete the above steps S02 to S04. Specifically, when the second extraction network model is executed, after the feature map data reaches the second convolution layer therein, it will be convolved with the second convolution kernel corresponding to the second convolution layer with a step size of Stride=4, and then an edge expansion operation will be performed according to the padding number of 20, so as to obtain the first convolution result C1; further, the first convolution result C1 is used as the input of the third convolution layer, and when the third convolution layer is executed, the first convolution result with a step size of Stride=8 will be convolved with the third convolution kernel corresponding to the third convolution layer, and then an edge expansion operation will be performed according to the padding number of 10, so as to obtain the second convolution result C2; further, the second convolution result C2 is used as the input of the fourth convolution layer, and when the fourth convolution layer is executed, the second convolution result will be convolved with the fourth convolution kernel corresponding to the fourth convolution layer, so as to obtain the third convolution result C3.

[0070] It should be noted here that in the above-mentioned second extraction network, if the upstream of the second convolutional layer with stride = 4 is connected to other convolutional layers, such as the first convolutional layer with stride = 2, then: the feature map data output by the first extraction network can first enter the first convolutional layer with stride = 2 after being input into the second extraction network, and the convolution result C0 obtained after convolution with the first convolution kernel corresponding to the first convolutional layer is used as the input of the second convolutional layer.

[0071] The third convolution result C3 obtained above will be used as the output prediction network input to trigger the execution of the output prediction network to complete the above step S05. Specifically, the third convolution result C3 will be sent to each network branch in the output prediction network for processing to obtain at least the face heat map and face angle information (i.e., the angle information between the face and the terminal).

[0072] The above-mentioned face heat map can reflect the position of the face, and is obtained by processing the third convolution result C3 by the first network branch.

[0073] The angle in the above-mentioned face angle information may be the roll angle of the face, and accordingly, it may be obtained by processing the third convolution result C3 through the second network branch. Considering that if the second network branch is designed in a traditional way to use angle regression to calculate the roll angle of the face, there will often be a jump between 0 degrees and 360 degrees, thereby affecting the angle accuracy. Therefore, in order to effectively optimize the angle calculation and improve the angle determination accuracy, in the embodiment of the present application, the second network branch will not use regression to calculate the roll angle of the face, but will use angle classification to determine the roll angle of the face. Specifically, in the second network branch, multiple angle classifications are pre-divided. For example, if each interval of 10° is divided into an angle classification, 36 angle classifications are pre-divided, that is, 0-10°, 11°-20°, 21°-30°, and so on are respectively one angle classification; the second network branch processes the third convolution result C3, and can determine the corresponding target angle classification from multiple angle classifications for each face, so as to determine the roll angle of the face (i.e., the angle between the face and the terminal) according to the target angle classification corresponding to each face. Of course, in other embodiments, other types of angles can also be used as the angle between the face and the terminal for the face. For example, the angle between the line connecting two specific facial points of the face and the interface display direction displayed on the terminal screen can be used as the angle between the face and the terminal. The facial feature points of the face may include but are not limited to feature points representing the nose, eyes, corners of the eyes, lips, face clips, etc., which can be obtained by processing the third convolution result C3 by the third network branch in the output prediction network. Based on the above content, a specific implementation scheme of "obtaining the face angle information based on the third convolution result" in the above S05 can be implemented by using the steps included in any one of the following items:

[0074] ① Based on the third convolution result, determine the target angle classification corresponding to each face from multiple angle classifications; determine the angle between each face and the terminal according to the target angle classification corresponding to each face; or

[0075] ② Based on the third convolution result, determine the facial feature points of each face; and determine the angle between each face and the terminal according to the facial specific points of each face.

[0076] In the above ①, several angle classifications with the highest scores among the multiple angle classifications can be determined as target angle classifications, and then the angle between the face and the terminal is calculated based on the target angle classifications and their corresponding scores. Therefore, the above "based on the third convolution result, determine the target angle classification corresponding to each face from the multiple angle classifications, and determine the angle between each face and the terminal based on the target angle classification corresponding to each face" can be achieved through the following specific steps:

[0077] S11. Based on the third convolution result, determine, for each face, a score corresponding to each of the multiple angle classifications;

[0078] S12. Determine the angle between each face and the terminal according to a set number of angle categories with the highest scores and their corresponding scores.

[0079] In specific implementation, multiple angle classifications can be sorted from high to low according to the scores of each angle classification. Then, the angle classifications with a set number of sortings are selected as the target angle classifications. The set number can be set flexibly, such as two or three, which is not limited here. Finally, according to the target angle classification and its corresponding score, the angle between the face and the terminal can be determined by weighted summation with the score as the weight.

[0080] For example, the target angles determined for a face are classified into 0-10° and 11-20°, the score corresponding to 0-10° is 0.8, and the score corresponding to 11-20° is 0.75. The angle between the face and the terminal can be: 0.8*0-10°+0.75*11-20°=8.25-23°.

[0081] For the implementation description of ② above, please refer to the relevant context, which will not be repeated here.

[0082] Furthermore, after obtaining the face heat map and face and angle information, in order to accurately determine which face should be used to rotate the interface displayed on the terminal screen when multiple faces appear, so as to avoid the interface direction jumping back and forth, the embodiment of the present application will perform post-processing analysis on the face heat map and face angle information to determine the target face closest to the terminal and the angle corresponding to the target face, and implement interface rotation control based on the angle corresponding to the target face. Based on this, in an implementable solution, the above 1023 "determine the target face angle based on the face heat map and the face angle information" can specifically include:

[0083] 10231. When it is determined based on the face heat map that there are multiple faces, determine a target face with the shortest distance from the terminal from the multiple faces according to the distances between the multiple faces included in the face heat map and the terminal;

[0084] 10232. Obtain an angle corresponding to the target face from the face angle information;

[0085] In specific implementation, the number of faces can be determined based on the distribution information of faces in the face heat map. Figure 4From the related content described, it can be learned that a white bright area in the face heat map represents the center point position of a face, that is, a white bright area in the heat map can represent a face, so the above distribution information can refer to the distribution information of the center point position of each face obtained from the face heat map. And correspondingly, the distance between each face contained in the face heat map and the terminal can refer to the distance from the center point of each face to the terminal screen (more specifically, to the front camera of the terminal). The distance information between the face and the terminal is determined based on the depth map obtained in the above step 1021.

[0086] Based on the distribution information, a face can be selected as the target face to determine whether the interface displayed on the terminal screen needs to be rotated. Specifically, when the number of faces represented by the above distribution information does not exceed one (that is, one, the face can be directly selected as the target face; and accordingly, the face angle information also only contains one angle, that is, the angle between the face represented by the distribution information and the terminal, and the angle can be directly determined whether to trigger the interface rotation and the corresponding rotation direction. Otherwise, when the number of faces represented by the above distribution information is multiple, accordingly: the above face heat map contains the distances of multiple faces from the terminal; and the face angle information will contain multiple angles, one angle corresponding to a face. In this case, the face with the shortest distance to the terminal can be determined as the target face according to the distances of multiple faces from the terminal contained in the face heat map; then, the angle corresponding to the target face can be further determined from the face angle information, so as to determine whether to trigger the interface rotation and the corresponding rotation direction according to the angle corresponding to the target face.

[0087] The interface mentioned above may refer to the system interface of the terminal or may refer to the application interface of various applications such as video applications, browsers, social applications, etc. installed on the terminal, and this embodiment does not limit this.

[0088] Regarding the content related to step 102 described above, it should be additionally explained here that: if it is determined based on the data information that there is only one face, it is also possible to obtain only a face image from the data information, and then use the processing model described above to process the face image to obtain a unique face angle (that is, the target face), and then determine whether to control the rotation of the interface based on the face angle.

[0089] In the above 103, it can be but not limited to that when the target face angle is greater than or equal to the preset angle, it is determined that the interface needs to be controlled to rotate, and then the face deflection direction is determined in combination with the target face angle, so as to determine the rotation direction of the interface. For example descriptions, please refer to the relevant content described below in combination with the application scenario.

[0090] In summary, the technical solution provided by this embodiment, in response to the terminal posture change event, will start the front camera of the terminal to collect data information, and by processing the data information, the target face angle can be determined; wherein, if the target face is missing, the information related to the target face in the data information can be supplemented during processing, so as to determine the target face angle based on the supplemented target face related information; then, determine whether to control the rotation of the interface displayed by the terminal according to the target face angle. With this solution, even in scenarios where the entire face of the user cannot be collected, the high accuracy of face recognition detection and the calculation of the angle between the face and the terminal can be guaranteed through data supplementation, thereby effectively optimizing the interface rotation control, and this also makes the solution more applicable to a wider range of scenarios with fewer limitations, which helps to improve the user terminal usage experience.

[0091] Based on the above content, the present application also provides two interface rotation control method embodiments from other perspectives. The execution subject of the other method embodiments is also a terminal. For details about the terminal, please refer to the context-related content of the present application. Specifically,

[0092] Figure 5 FIG. 2 is a flow chart of an interface rotation control method provided by another embodiment of the present application, which mainly describes the present solution from the perspective of a multi-face scene. Figure 5 As shown, the interface rotation control method includes the following steps:

[0093] 201. In response to a terminal posture change event, start a front camera of the terminal to collect data information;

[0094] 202. When it is determined based on the data information that there are multiple faces, select a target face from the multiple faces according to distance information included in the data information;

[0095] 203. Determine whether to control the interface displayed on the terminal to rotate according to the target face.

[0096] For detailed description of the specific implementation of 201 above, please refer to the relevant content in other embodiments above.

[0097] In the above 202-203, a face map and a corresponding depth map containing distance information can be determined based on the data information. By performing image recognition analysis on the face map, it can be determined whether there are multiple faces and the missing status of each face. In the case that multiple faces are not missing, the face closest to the terminal can be directly selected from multiple faces as the target face based on the distance information contained in the depth map, and the target face angle (such as the roll angle of the target face, the angle between the line connecting the two facial feature points (such as two eye feature points) of the target face and the terminal) can be determined using the existing scheme or the method provided in this application to determine whether to control the rotation of the interface displayed by the terminal according to the target face angle. However, if one or two or more faces are missing among multiple faces, considering the use of existing solutions for face recognition detection, face angle determination, etc., problems such as face misrecognition and low angle calculation accuracy may occur, resulting in inaccurate subsequent interface rotation control. Therefore, in order to better cope with the face missing scenario, this solution can supplement the data of the face image and the depth image with information related to the missing faces, and analyze the number of faces, target faces, target face angles, etc. based on the supplemented face-related information, so as to determine whether to control the interface rotation and the corresponding rotation direction according to the target face.

[0098] Based on the above content, in an implementable solution, the step of “selecting a target face from the multiple faces based on the distance information included in the data information” in 202 above may specifically include:

[0099] 2021. According to the distance information contained in the data information, select a face among the multiple faces that is closest to the terminal as the target face.

[0100] Furthermore, the above step 1021 can be implemented by the following steps:

[0101] 20211. Based on the data information, obtain a facial image including multiple faces and a depth map including the distance information;

[0102] 20212. Process the face image and the depth image to obtain a face heat map;

[0103] 20213. Determine, according to the distances between the multiple faces included in the face heat map and the terminal, a face closest to the terminal as the target face;

[0104] Among them, if any face among the multiple faces is missing, when processing the face image and the depth map, data supplementation can be performed on the information related to the missing face in the face image and the depth map to determine the face heat map based on the supplemented face-related information.

[0105] In specific implementation, this embodiment uses a preset processing model to process the face image and the depth map. Through this processing model, the information related to the missing face in the face image and the depth map can be supplemented, and the face heat map can be determined based on the supplemented face-related information. In addition, the face angle information, the face feature point information, etc. can also be determined. For a detailed description of the processing model and the use of the processing model to process the face image and the depth map, please refer to the relevant content in other embodiments of this application above, and no further details will be given here.

[0106] In the above 203, it can be determined whether the interface needs to be controlled to rotate according to the target face angle (i.e., the angle between the target face and the terminal), and if it is determined to be necessary, the corresponding rotation direction, rotation angle, etc. can be further determined, so that the interface can be controlled to rotate according to the determined rotation direction and rotation angle, etc. For a detailed description of the determination of the target face angle, please refer to the relevant content in other embodiments above.

[0107] The solution provided in this embodiment starts the front camera of the terminal to collect data information in response to the terminal posture change event, and when it is determined that there are multiple faces based on the data information, a target face can be selected from the multiple faces according to the distance information contained in the data information, and then whether to control the rotation of the interface displayed by the terminal is determined according to the target face. It can be seen that this solution can solve the problem of interface rotation control in multi-user face scenarios, which helps to improve the user experience of the terminal.

[0108] For the specific implementation description of each step described above in this embodiment, please refer to the relevant content in other embodiments above. In addition to the above steps, the embodiment of this application may also include other steps. For the other steps that may be included and their specific implementation description, please refer to the relevant content in other embodiments above.

[0109] Figure 6 FIG. 2 is a flow chart of an interface rotation control method provided by another embodiment of the present application. Figure 6 As shown, the interface rotation control method includes the following steps:

[0110] 301. In response to a terminal posture change event, obtain a grayscale image and a corresponding depth map through a front camera of the terminal, wherein the grayscale image includes at least one face, and the depth map includes a distance from each face to the terminal;

[0111] 302. Process the grayscale image and the depth image to determine the angle of the target face;

[0112] 303. Determine whether to trigger rotation of an interface displayed on the terminal according to the target face angle;

[0113] Among them, the target face is one of the at least one face; if any of the at least one face is missing, the information related to the missing face in the grayscale image and the depth image can be supplemented during processing to determine the target face angle based on the supplemented face-related information.

[0114] In the above, a preset processing model can be used to process the grayscale image and the depth image to determine the face heat map, face angle information, etc., and then a target face can be determined from at least one face based on the face heat map, and the angle corresponding to the target face can be obtained from the face angle information. The angle corresponding to the target face is the target face angle.

[0115] For the specific implementation description of the above steps 301 to 303 and the above processing model, please refer to the relevant content in other embodiments above. In addition to the above steps, the embodiments of the present application may also include other steps. For the other steps that may be included and their specific implementation description, please refer to the relevant content in other embodiments above.

[0116] In the technical solution provided by this embodiment, in response to the terminal posture change event, a grayscale image containing at least one face and a corresponding depth map can be obtained through the front camera of the terminal, and the depth map contains the distance between each face and the terminal; then, by processing the grayscale image and the depth map, the angle between the target face and the terminal can be obtained, so as to determine whether to trigger the rotation of the interface displayed by the control terminal according to the angle between the target face and the terminal; wherein the above-mentioned target face is one of the at least one face; if any face of the at least one face is missing, when processing the grayscale image and the depth map, the information related to the missing face in the grayscale image and the depth map can be supplemented with data, so as to determine the target face angle based on the supplemented face-related information. By adopting this solution, even in the scenario where all the user's faces cannot be collected, the high accuracy of face recognition detection and the angle calculation between the face and the terminal can be guaranteed through data supplementation, so as to effectively optimize the interface rotation control, and in addition, the interface rotation control problem in the scenario of multiple faces can also be solved. In summary, it can be seen that the application scheme of this application has a wider range of applicable scenarios and fewer limitations, which helps to improve the user terminal usage experience.

[0117] To facilitate understanding of this application, Figure 7 to Figure 9 Taking the terminal 6 as a smart phone (hereinafter referred to as mobile phone) as an example, the technical solution of the present application is described in combination with several application scenarios.

[0118] Application scenario 1: Users change from lying prone to looking at their phone to lying sideways to looking at their phone

[0119] See Figure 7, by default, the automatic rotation function of the mobile phone is turned off. When the mobile phone receives an instruction input by the user clicking the automatic rotation control 60 on the control center interface, the automatic rotation control 60 is selected, and the screen interface (i.e., the interface displayed on the screen) rotation function is turned on. Assume that the screen interface rotation function of the mobile phone is turned on, and the user initially looks at the phone in a prone position (such as watching a video in a prone position) and the phone is vertical relative to the ground, and then changes to lying sideways to look at the phone (watching a video in a sideways position) and the phone is at a certain inclination angle relative to the ground (roughly horizontal). When the user changes to lying sideways to look at the phone, the phone detects that its posture has changed, starts the front camera, and uses the front camera to collect data information from the front of the phone (i.e., the direction in which the phone screen is facing). Based on the data information, the user's face image and the corresponding depth map can be obtained. The face image is Figure 1b As shown in the figure, it is a grayscale image, which contains a small part of a user's face. Figure 3 The input of the convolutional network model shown in triggers the execution of the convolutional network model, and the following is obtained: Figure 4 The face heat map Heat2 and the face roll angle are shown on the right side. The heat map Heat2 reflects a face (that is, there is a user in front of the phone). At this time, it can be directly determined whether it is necessary to trigger the rotation of the interface F currently displayed on the phone screen and the corresponding rotation direction based on the roll angle of the face. For example, the roll angle of the face is 278.25°~293°, which means that when the user lies on his side, the face rotates to the left by 67°~81.75°. At this time, the angle of rotation of the face to the left is greater than the preset angle (such as 45°). It is determined that the rotation of the currently displayed interface F needs to be triggered and the rotation direction is clockwise. Based on the above determination result, the currently displayed interface F can be controlled to rotate 90° clockwise. Figure 7 The interface F' shown in FIG. 1 is the interface F rotated 90° clockwise.

[0120] Alternatively, the face image and depth image are used as Figure 3 The input of the convolutional network model shown in triggers the execution of the convolutional network model, and the following is obtained: Figure 4 The face heat map Heat2 and the facial feature points of the face are shown on the right side of the figure. Assuming that the facial feature points include the left eye specific point c1 and the right eye specific point c2 (determined by data supplementation), then: Figure 8In a coordinate system established with one corner endpoint of the mobile phone as the origin, a line parallel to the horizontal side of the mobile phone screen as the X-axis, and a line parallel to the vertical side of the mobile phone screen as the Y-axis, there is an angle θ between the line connecting the specific points c1 of the left eye and c2 of the right eye on the user's face and the positive direction of the Y-axis. This angle θ is the angle between the face and the terminal, so based on this angle θ, it can be determined whether it is necessary to trigger the rotation of the interface F currently displayed on the mobile phone screen and the corresponding rotation direction. For example, when the angle θ is in the range of [-45°, -90°], the currently displayed interface F is controlled to rotate 90° clockwise.

[0121] Application scenario 2: Multiple people watching mobile phones together

[0122] See Fig. 9 , the automatic rotation function of the mobile phone is turned on. User u1 and user u2 initially sit together to watch the mobile phone (such as watching a video). After a while, the two users change to lying sideways to watch the mobile phone. At this time, the mobile phone detects that its posture has changed and starts the front camera to use the front camera to obtain the face image and corresponding depth map of the user in front of the mobile phone. The face image contains at least part of the face of user u1 and user u2. The face image and depth map are used as Figure 3 The input of the convolutional network model shown in triggers the execution of the convolutional network model, and the heat map of the corresponding face and the angle information of the face and the mobile phone are obtained. There are two white bright areas in this heat map to represent the center point position of the face of user u1 and the center point position of the face of user u2, and the angle information includes the roll angle of the face of user u1 and the roll angle of the face of user u2. Based on the heat map, it is determined that the face of user u1 is closest to the mobile phone screen. For this purpose, the face of user u1 is used as the target face, and the roll angle of the face of user u1 is obtained from the angle information to determine whether it is necessary to control the interface currently displayed on the mobile phone screen to rotate and the corresponding rotation direction.

[0123] It should be noted here that the solutions provided in the embodiments of the present application can be applied not only to the interface rotation control scenarios described above, but also to any other scenarios requiring face detection, such as face quantity detection, facial feature point detection, etc.

[0124] Fig.10 FIG. 1 is a schematic diagram showing the structure of an interface rotation control device provided by an embodiment of the present application. Fig. 9 The interface rotation control device includes: a starting module 31, a processing module 32 and a determining module 33; wherein,

[0125] The startup module 31 is used to start the front camera of the terminal to collect data information in response to the terminal posture change event;

[0126] The processing module 32 processes the data information to determine the target face angle; if the target face is missing, the information related to the target face in the data information can be supplemented during the processing to determine the target face angle based on the supplemented target face related information;

[0127] The determination module 33 is used to determine whether to control the interface displayed on the terminal to rotate according to the target face angle.

[0128] Furthermore, the above-mentioned processing module 32, when used to process the data information to determine the target face angle, is specifically used to: obtain a face map and a corresponding depth map based on the data information; the face map contains at least one face, the target face is one of the at least one face, and the depth map contains the distance between each face and the terminal; process the face map and the depth map to obtain a face heat map and face angle information; determine the target face angle based on the face heat map and the face angle information.

[0129] Furthermore, the above-mentioned processing module 32, when used to process the face image and the depth map to obtain the face heat map and the face angle information, is specifically used to: obtain a processing model; use the processing model to process the face image and the depth map to obtain the face heat map and the face angle information.

[0130] Furthermore, the above-mentioned processing model is a convolutional network model, which includes at least two convolutional layers with edge expansion operations, and the edge expansion operations of the two convolutional layers have different operation parameters corresponding to them; and, if the target face is missing, when the processing model is used to process the face image and the depth map, the two convolutional layers are used to supplement the information related to the target face in the face image and the depth map.

[0131] Furthermore, the above-mentioned processing module 32, when used to process the face image and the depth map using the processing model to obtain the face heat map and the face angle information, is specifically used to: perform pre-feature extraction processing on the face image and the depth map respectively to obtain corresponding feature map data; perform a convolution operation on the feature map data and then perform an edge expansion operation to obtain a first convolution result; perform a convolution operation on the first convolution result and then perform an edge expansion operation to obtain a second convolution result; perform a convolution operation on the second convolution result to obtain a third convolution result; and obtain the face heat map and the face angle information based on the third convolution result.

[0132] Furthermore, the above-mentioned processing module 32, when used to obtain the facial angle information based on the third convolution result, is specifically used to: determine the target angle classification corresponding to each face from multiple angle classifications based on the third convolution result, and determine the angle between each face and the terminal according to the target angle classification corresponding to each face; or, determine the facial feature points of each face based on the third convolution result, and determine the angle between each face and the terminal according to the facial feature points of each face.

[0133] Furthermore, the above-mentioned processing module 32, when used to determine the target angle category corresponding to each face from multiple angle categories based on the third convolution result, and determine the angle between each face and the terminal according to the target angle category corresponding to each face, is specifically used to: determine the scores corresponding to each of the multiple angle categories for each face based on the third convolution result; determine the angle between each face and the terminal according to a set number of angle categories with the highest scores and the corresponding scores.

[0134] Furthermore, the above-mentioned determination module 33, when used to determine the target face angle based on the face heat map and the face angle information, is specifically used to: when it is determined based on the face heat map that there are multiple faces, determine the target face with the shortest distance to the terminal from the multiple faces according to the distance information contained in the face heat map; and obtain the angle corresponding to the target face from the face angle information.

[0135] Furthermore, the front camera is a depth camera, and the face image is a grayscale image or a color image.

[0136] It should be noted that: the interface rotation control device provided in the above embodiment can realize the above Figure 2 The technical solutions described in the method embodiments shown and the specific implementation principles of the above modules or units can be found in the corresponding contents of the above corresponding method embodiments, which will not be repeated here.

[0137] Another embodiment of the present application further provides an interface rotation control device. The structure of the device can be seen in Fig.11 Specifically, the interface rotation control device includes: a starting module 41, a selection module 42 and a determination module 43; wherein,

[0138] A starting module 41, configured to start a front camera of the terminal to collect data information in response to a terminal posture change event;

[0139] A selection module 42, configured to select a target face from the multiple faces according to distance information included in the data information when it is determined that there are multiple faces based on the data information;

[0140] The determination module 43 is used to determine whether to control the interface displayed by the terminal to rotate according to the target face.

[0141] Furthermore, the above-mentioned selection module 42, when used to select a target face from the multiple faces according to the distance information contained in the data information, is specifically used to: select a face among the multiple faces that is closest to the terminal as the target face according to the distance information contained in the data information.

[0142] Furthermore, when the above 43 is used to select a face among the multiple faces that is closest to the terminal as the target face based on the distance information contained in the data information, it can be specifically used to: based on the data information, obtain a face map containing multiple faces and a depth map containing the distance information; process the face map and the depth map to obtain a face heat map; determine that a face closest to the terminal is the target face based on the distance between each of the multiple faces contained in the face heat map and the terminal; wherein, if any of the multiple faces is missing, when processing the face map and the depth map, data supplementation can be performed on the face map and the information related to the missing face in the depth map to determine the face heat map based on the supplemented face-related information.

[0143] It should be noted that: the interface rotation control device provided in the above embodiment can realize the above Figure 5 The technical solutions described in the method embodiments shown and the specific implementation principles of the above modules or units can be found in the corresponding contents of the above corresponding method embodiments, which will not be repeated here.

[0144] Another embodiment of the present application further provides an interface rotation control device. The structure of the device can be seen in Fig.12 Specifically, the interface rotation control device includes: an acquisition module 51, a processing module 52 and a determination module 53; wherein,

[0145] The acquisition module 51 is used to acquire a grayscale image and a corresponding depth map through the front camera of the terminal in response to the terminal posture change event; wherein the grayscale image contains at least one face, and the depth map contains the distance between each face and the terminal;

[0146] A processing module 52, used for processing the grayscale image and the depth image to determine the angle of the target face;

[0147] A determination module 53, used to determine whether to trigger rotation of the interface displayed on the control terminal according to the target face angle;

[0148] Among them, the target face is one of the at least one face; if any of the at least one face is missing, the information related to the missing face in the grayscale image and the depth image can be supplemented during processing to determine the target face angle based on the supplemented face-related information.

[0149] It should be noted that: the interface rotation control device provided in the above embodiment can realize the above Figure 6 The technical solutions described in the method embodiments shown and the specific implementation principles of the above modules or units can be found in the corresponding contents of the above corresponding method embodiments, which will not be repeated here.

[0150] The above describes the internal functions and structure of the interface rotation control device. In practice, the interface rotation control device can be implemented on a terminal device. The terminal device can be a smart phone, a tablet computer, a PDA (Personal Digital Assistant), a netbook, an e-book, or other portable electronic devices. Based on this, another embodiment of the present application provides a terminal device. Fig.13 FIG. 1 shows a schematic diagram of the structure of the terminal device. Fig.13 Said terminal device comprises:

[0151] Screen 63, used for displaying the interface;

[0152] A front camera 61 is arranged on the side of the terminal device where the screen is located, and is used to collect image data in front of the terminal device and send the image data to the control device so that the control device can obtain a face image and a corresponding depth image of the user in front of the terminal device based on the image data, wherein the face image is a grayscale image or a color image;

[0153] The control device 62 is connected to the front camera for implementing the steps of the interface rotation control method provided in each embodiment of the present application.

[0154] The above-mentioned screen 63 may include an LCD (Liquid Crystal Display) display screen or an OLED (Organic Light-Emitting Diode) display screen for displaying the interface of the terminal system or each application. And the above-mentioned screen 63 may also include a touch panel (TP) and the like. If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0155] The front camera 61 is located on the side where the screen of the terminal device is located, that is, the front of the terminal device (the front side). For detailed description of the front camera 61, please refer to the relevant content in other embodiments above.

[0156] The terminal device described above may also include a memory 64. The memory 64 is used to store computer programs and may be configured to store various other data to support operations on the terminal device. Examples of these data include instructions for any application or method used to operate on the terminal device. The memory 64 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0157] The control device 62 includes a processor. The processor is coupled to the memory 64 and is used to execute the computer program in the memory 64 to implement the steps or functions of the interface rotation control method provided in each embodiment of the present application.

[0158] Further, if Fig.13 As shown, the terminal device also includes: a communication component 65, a power component 66, an audio component 67, a sensor component (not shown in the figure) and other components.

[0159] The above-mentioned communication component 65 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0160] The power supply assembly 66 provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0161] The above-mentioned audio component 67 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0162] The above-mentioned sensor components may include but are not limited to the posture sensors described in other embodiments above.

[0163] Fig.13 The following only schematically shows some components of the terminal device, which does not mean that the terminal device only includes Fig.13 Components shown.

[0164] Accordingly, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, each step that can be executed by a terminal device in the above method embodiment can be implemented.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. Computer-usable storage media include, but are not limited to, disk storage, CD-ROM, optical storage, etc.

[0166] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0167] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An interface rotation control method, characterized in that: include: In response to a terminal posture change event, starting a front camera of the terminal to collect data information; Processing the data information to determine the target face angle; wherein, if the target face is missing, the information related to the target face in the data information can be supplemented during processing, so as to determine the target face angle based on the supplemented target face related information; According to the target face angle, it is determined whether to control the interface displayed by the terminal to rotate.

2. The method according to claim 1, characterized in that: Processing the data information to determine the target face angle includes: Based on the data information, a face map and a corresponding depth map are obtained; the face map includes at least one face, the target face is one of the at least one face, and the depth map contains the distance between each face and the terminal; Processing the face image and the depth image to obtain a face heat map and face angle information; Based on the face heat map and the face angle information, the target face angle is determined.

3. The method according to claim 2, characterized in that Process the face image and the depth image to obtain a face heat map and face angle information, including Get the processing model; The face image and the depth image are processed using the processing model to obtain the face heat map and the face angle information.

4. The method according to claim 3, characterized in that The processing model is a convolutional network model, wherein the convolutional network model includes at least two convolutional layers with edge expansion operations, and the edge expansion operations of the two convolutional layers have different operation parameters corresponding to them; as well as If the target face is missing, when the face image and the depth image are processed by the processing model, the two convolutional layers are used to supplement the information related to the target face in the face image and the depth image.

5. The method according to claim 4, characterized in that Processing the face image and the depth image using the processing model to obtain the face heat map and the face angle information includes: Performing pre-feature extraction processing on the face image and the depth image respectively to obtain corresponding feature image data; Performing a convolution operation on the feature map data and then performing an edge expansion operation to obtain a first convolution result; Performing a convolution operation on the first convolution result and then performing an edge expansion operation to obtain a second convolution result; Performing a convolution operation on the second convolution result to obtain a third convolution result; Based on the third convolution result, the face heat map and the face angle information are obtained.

6. The method according to claim 5, characterized in that Based on the third convolution result, the face angle information is obtained, including: Based on the third convolution result, determine a target angle classification corresponding to each face from multiple angle classifications; determine the angle between each face and the terminal according to the target angle classification corresponding to each face; or Based on the third convolution result, the facial feature points of each face are determined; and according to the facial feature points of each face, the angle between each face and the terminal is determined.

7. The method according to claim 6, characterized in that Based on the third convolution result, determining a target angle classification corresponding to each face from multiple angle classifications, and determining an angle between each face and the terminal according to the target angle classification corresponding to each face, including: Based on the third convolution result, determining, for each face, scores corresponding to each of the multiple angle classifications; The angle between each face and the terminal is determined based on a set number of angle categories with the highest scores and their corresponding scores.

8. The method according to any one of claims 2 to 7, characterized in that Determining the target face angle according to the face heat map and the face angle information includes: When it is determined based on the face heat map that there are multiple faces, determining a target face with the shortest distance from the terminal from the multiple faces according to the distances between the multiple faces included in the face heat map and the terminal; An angle corresponding to the target face is obtained from the face angle information.

9. The method according to any one of claims 2 to 7, characterized in that The front camera is a depth camera, and the face image is a grayscale image or a color image.

10. An interface rotation control method, characterized in that: include: In response to a terminal posture change event, starting a front camera of the terminal to collect data information; When it is determined based on the data information that there are multiple faces, selecting a target face from the multiple faces according to the distance information included in the data information; Determining whether to control the interface displayed on the terminal to rotate according to the target face; Among them, if any of the multiple faces is missing, the data related to the missing face in the face map containing multiple faces and the depth map containing the distance information can be supplemented to determine the target face based on the supplemented face-related information.

11. The method according to claim 10, characterized in that Selecting a target face from the multiple faces according to the distance information included in the data information includes: According to the distance information included in the data information, a face among the multiple faces that is closest to the terminal is selected as the target face.

12. The method according to claim 11, characterized in that According to the distance information included in the data information, a face closest to the terminal among the multiple faces is selected as a target face, including: Based on the data information, obtaining a face map including multiple faces and a depth map including the distance information; Processing the face image and the depth image to obtain a face heat map; According to the distances between the multiple faces included in the face heat map and the terminal, determining a face closest to the terminal as the target face; Among them, if any of the multiple faces is missing, when processing the face image and the depth map, data supplementation can be performed on the information related to the missing face in the face image and the depth map to determine the face heat map based on the supplemented face-related information.

13. An interface rotation control method, characterized in that: include: In response to a terminal posture change event, a grayscale image and a corresponding depth image are acquired through a front camera of the terminal; wherein the grayscale image includes at least one human face, and the depth image includes the distance between each human face and the terminal; Processing the grayscale image and the depth image to determine the angle of the target face; Determining whether to trigger rotation of an interface displayed on the terminal according to the target face angle; Among them, the target face is one of the at least one face; if any of the at least one face is missing, the information related to the missing face in the grayscale image and the depth image can be supplemented during processing to determine the target face angle based on the supplemented face-related information.

14. A terminal device, characterized in that: include: Screen, used to display the interface; A front camera is arranged on the side of the terminal device where the screen is located; A control device, communicatively connected to the front camera, for implementing the steps in the interface rotation control method described in any one of claims 1 to 9, or the steps in the interface rotation control method described in any one of claims 10 to 12, or the steps in the interface rotation control method described in claim 13.

Images