Interface rotation control method and terminal equipment

Abstract

The embodiment of the invention provides an interface rotation control method and terminal equipment. The interface rotation control method comprises the following steps: in response to a terminal posture change event, starting a front camera of a terminal to collect data information; processing the data information to determine a target face angle; wherein if the target face is missing, data supplementation can be performed on information, related to the target face, in the data information during processing, so that the angle of the target face can be determined based on the supplemented information related to the target face; and determining whether to control an interface displayed by the terminal to rotate or not according to the target face angle. By adopting the scheme, the interface rotation control under the scene that all the faces of the user cannot be acquired can be effectively optimized, so that the use experience of the user terminal is improved.

Description

[0001] This divisional application of the invention patent application is a divisional application of a Chinese invention patent application with an application date of September 15, 2023, an application number of 202311199460.7, and a title of "Interface Rotation Control Method and Terminal Device". Technical Field

[0002] This application relates to the field of terminal technologies, and in particular, to an interface rotation control method and a terminal device. Background Art

[0003] With the development of technology, the screens of terminals such as smart phones and tablet computers have an automatic rotation function. When the automatic rotation function is turned on, the front camera of the terminal can be used to capture the face image of the user using the terminal to identify and analyze the face direction, and then determine the rotation direction of the interface displayed on the terminal screen according to the face direction. However, due to the limitation of the acquisition range of the front camera, in some usage scenarios, it is easy to capture an incomplete face image, and the existing solutions have poor recognition effects for incomplete face images, which may easily cause errors in interface rotation and bring a bad experience to users. Summary of the Invention

[0004] In view of this, multiple aspects of this application provide an interface rotation control method and a terminal device to optimize the interface rotation control in scenarios where the entire face of the user cannot be captured or multiple faces are captured.

[0005] In an embodiment of this application, an interface rotation control method is provided. The method includes:

[0006] In response to a terminal posture change event, collect data information;

[0007] Process the data information to determine the target face angle; wherein, if there is a missing part in the target face, when processing, supplement the information related to the target face in the data information, and determine the target face angle based on the supplemented information related to the target face;

[0008] According to the target face angle, determine whether to control the rotation of the interface displayed on the terminal.

[0009] In another embodiment of this application, an interface rotation control method is further provided. The method includes:

[0010] In response to a terminal posture change event, collect data information;

[0011] When it is determined based on the data information that there are multiple faces, select a target face from the multiple faces according to the distance information included in the data information;

[0012] According to the target face, determine whether to control the rotation of the interface displayed on the terminal.

[0013] In another embodiment of the present application, a method for controlling interface rotation is further provided. The method includes:

[0014] In response to a terminal attitude change event, obtain a grayscale image and a corresponding depth image through the front camera of the terminal; wherein, at least one face is included in the grayscale image, and the distance from each face to the terminal is included in the depth image;

[0015] Process the grayscale image and the depth image to determine the target face angle;

[0016] According to the target face angle, determine whether to trigger the rotation of the interface displayed on the terminal;

[0017] Wherein, the target face is one of the at least one face; if any face among the at least one face is missing, during processing, data supplementation can be performed on the information related to the missing face in the grayscale image and the depth image, so as to determine the target face angle based on the supplemented face-related information.

[0018] In another embodiment of the present application, a terminal device is further provided. The terminal device includes:

[0019] A screen for displaying an interface;

[0020] A front camera disposed on the side of the terminal device where the screen is located;

[0021] A control device communicatively connected to the front camera for implementing the steps in the interface rotation control method provided in the embodiments of the present application.

[0022] In a technical solution provided by the embodiments of the present application, in response to a terminal attitude change event, the front camera of the terminal is activated to collect data information, and the target face angle can be determined by processing the data information; wherein, if the target face is missing, during processing, data supplementation can be performed on the information related to the target face in the data information, so as to determine the target face angle based on the supplemented target face-related information; then, according to the target face angle, it is determined whether to control the rotation of the interface displayed on the terminal. By adopting this solution, even in scenarios where it is impossible to collect the entire face of the user, high accuracy of face recognition detection and the calculation of the angle between the face and the terminal can be ensured through data supplementation, thereby effectively optimizing the interface rotation control and being beneficial to improving the user experience of the terminal.

[0023] In another technical solution provided by the embodiments of the present application, in response to a terminal attitude change event, the front camera of the terminal is activated to collect data information. When it is determined that there are multiple faces based on the data information, a target face can be selected from the multiple faces according to the distance information included in the data information, and then whether to control the rotation of the interface displayed on the terminal is determined based on the target face. It can be seen that this solution can solve the problem of interface rotation control in a multi-user face scenario, which helps to improve the user experience of the terminal.

[0024] In yet another technical solution provided by the embodiments of the present application, in response to a terminal attitude change event, a grayscale image including at least one face and a corresponding depth image can be obtained through the front camera of the terminal. The depth image includes the distances from each face to the terminal. Then, by processing the grayscale image and the depth image, the angle between the target face and the terminal can be obtained, and thus whether to trigger the control of the rotation of the interface displayed on the terminal is determined based on the angle between the target face and the terminal. Among them, the above-mentioned target face is one of at least one face. If any face among at least one face is missing, data supplementation can be performed on the information related to the missing face in the grayscale image and the depth image when processing the grayscale image and the depth image. With this solution, even in scenarios where not all faces of the user can be collected, high accuracy of face recognition detection and calculation of the angle between the face and the terminal can be ensured through data supplementation, thereby effectively optimizing the interface rotation control. In addition, the problem of interface rotation control in a multi-user face scenario can also be solved. In summary, the solution of the present application has a wider applicable scenario and less limitation, which helps to improve the user experience of the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0026] Figure 1a It is a schematic flowchart of the principle of screen interface rotation provided by an exemplary embodiment of the present application;

[0027] Figure 1b It is a schematic diagram of a face image provided by an exemplary embodiment of the present application;

[0028] Figure 2 It is a schematic flowchart of a method for controlling interface rotation provided by an exemplary embodiment of the present application;

[0029] Figure 3 It is a network architecture diagram of a processing model (convolutional network model) provided by an exemplary embodiment of the present application;

[0030] Figure 4Schematic diagram of a face heat map provided by an exemplary embodiment of the present application;

[0031] Figure 5 and Figure 6 Schematic flowchart of an interface rotation control method provided by other exemplary embodiments of the present application;

[0032] Figures 7 to 9 Schematic diagram of an interface rotation control application scenario provided by an embodiment of the present application;

[0033] Figures 10 to 12 Schematic diagram of the structure of an interface rotation control device provided by an embodiment of the present application;

[0034] Figure 13 Schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present application. Detailed implementation manners

[0035] Currently in the market, when the auto-rotation function of terminals such as smart phones and tablet computers is turned on, as shown in Figure 1a , when it is detected through a sensor that the posture of the terminal device changes, the front camera on it will be activated to capture an image in front of the terminal. The image contains the face of the corresponding user of the terminal. By performing face recognition analysis on the image to obtain the face direction, and then determining the rotation direction of the interface displayed on the terminal screen according to the face direction. However, since the images taken in low-light (dark), backlight and other scenarios are often unclear, this also leads to problems such as low recognition accuracy and long time consumption in subsequent identifying the face direction through the image recognition analysis scheme to determine the interface rotation direction. In response to this problem, a current solution is to use a ToF (Time of Flight) camera as the front camera arranged for the terminal device. Although the ToF camera is applicable to all scenarios and the image effects in low-light, backlight and other scenarios are much better than those of RGB cameras, the field of view FoV (Field of View, specifically the range that the lens can cover) of the ToF camera is generally 78°, which is smaller than 90° - 100° of the RGB camera. Therefore, in some scenarios without usage restrictions, it is easier to capture images containing only part of the face (as shown in Figure 1b ). Traditional face recognition algorithms have poor recognition effects on images containing only part of the face and are prone to misrecognition, thus it is difficult to ensure the accuracy of face direction determination.

[0036] To solve the above problems, the present application provides a technical solution for interface rotation control. The basic idea of this solution is as follows: obtain the face image and the corresponding depth map of the user in front of the terminal through the front camera of the terminal, and input the face image and the depth map into a trained processing model, which is obtained by improving the model constructed based on the Center-net algorithm. Finally, based on the output results of the processing model (such as face heat map, face angle information of the face relative to the terminal, etc.), interface rotation control is realized. Through the above, it can ensure that even in some scenarios where it is impossible to collect images containing the entire face of the user and / or the image clarity is low, the accuracy of face direction (or the angle of the face relative to the terminal) calculation can be effectively ensured, thereby ensuring the accuracy of interface rotation control; in addition, it can also solve the problem of interface rotation control in the multi-face scenario. For the detailed description of the face image and the depth map, please refer to the relevant content below.

[0037] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0038] In the embodiments of the present application, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and roles. For example, the first convolution result and the second convolution result are only used to distinguish different convolution results, and no limitation is imposed on their order. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily mean different.

[0039] It should be noted that in this application, words such as "exemplary" or "for example" are used to give examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner. In addition, "at least one" in this application means one or more, and "multiple" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural, etc. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or similar expressions below refer to any combination of these items, including any combination of a single item (item) or multiple items (items). For example, at least one (item) of a, b, and c can represent: a, b, c, a, b, and c, a and b, a and c, b and c.

[0040] The following will, with reference to the accompanying drawings, detail the technical solutions provided by the embodiments of this application.

[0041] Figure 2 It is a schematic flowchart of an interface rotation control method provided by an embodiment of this application. The execution subject of this method is a terminal. More specifically, the terminal is a portable (such as a handheld) movable electronic device, such as a smart phone, a tablet computer, a POS machine, a netbook, etc. For a detailed description of the terminal, reference can also be made to the relevant content of the terminal device embodiment provided in the following text of this application. As Figure 2 shown, the interface rotation control method includes the following steps:

[0042] 101. In response to a terminal attitude change event, start the front camera of the terminal to collect data information;

[0043] 102. Process the data information to determine the target face angle; wherein, if there is a missing target face, when processing, data supplementation can be performed on the information related to the target face in the data information to determine the target face angle based on the supplemented information related to the target face.

[0044] 103. According to the target face angle, determine whether to control the rotation of the interface displayed by the terminal.

[0045] In the embodiment of this application, the automatic screen rotation function of the terminal is turned on. And a posture sensor can be set on the terminal, where the posture sensor can include motion sensors such as a three-axis gyroscope and a three-axis accelerometer. Correspondingly, in the above 101, the posture sensor set on the terminal can be used to detect whether a posture change event occurs.

[0046] For example, referring to Figure 7 , when the user uses the terminal in a prone position (or lying flat, sitting and lying), the terminal can be considered to be in a vertical state relative to the ground. When the user uses the terminal in a side-lying position, the attitude of the terminal relative to the ground changes from a vertical state to an approximately horizontal state. At this time, the front camera of the terminal can be activated to collect data information in front of the terminal by using the front camera.

[0047] In specific implementation, the front camera of the terminal is a depth camera with a ranging function, such as but not limited to a ToF camera, a structured light camera, or a binocular camera. Preferably, in the solution of this application, the front camera is a ToF camera because the ToF camera is more suitable for dynamic scenes compared with other cameras, has a longer measurement distance, is less affected by light environments such as strong light and low light, and has a simpler data output process. The data information in front of the terminal can be collected through the front camera of the terminal, and the data information may include face information and distance information.

[0048] For example, the front camera of the terminal is a ToF camera. During the shooting process of the ToF camera, by emitting light (such as infrared light) outward, the light is reflected back after encountering a shooting object such as a user and is collected by the ToF camera, or the light emitted by the user or other objects themselves is collected by the ToF camera. After the ToF camera images the collected light, a face image (such as Figure 1b ) containing face information can be obtained. This face image is a color image (specifically, it can be an infrared image). In addition, the ToF camera can also calculate the distance information reflecting the distance between the shooting object such as the user and the camera by using the time difference or phase difference between the light emission and the reflection back to the ToF camera. This distance information is also regarded as the distance information from the shooting object to the terminal, so as to generate an image containing distance information. This image including distance information is the depth map (also called the depth map). In the above, the start of shooting by the ToF camera is triggered only when it detects that the attitude of the terminal has changed (such as changing from a vertical state to a horizontal state relative to the ground), which can avoid unnecessary invalid shooting.

[0049] Based on the above example content, in a feasible solution, the above 102 "processing the data information to determine the target face angle" may specifically include:

[0050] 1021. Obtaining a face image and a corresponding depth map based on the data information; at least one face is included in the face image, the target face is one of the at least one face, and the distance from each face to the terminal is included in the depth map;

[0051] 1022. Processing the face image and the depth map to obtain a face heat map and face angle information;

[0052] 1023. Determine the target face angle based on the face heat map and the face angle information.

[0053] In the above 1021, the face image is a grayscale image (IR image) or a color image. Generally, the original face image directly obtained by the front camera is a color image. However, in order to reduce the subsequent calculation amount and the use of the terminal memory, this embodiment will convert the color image into a grayscale image for subsequent use. That is, preferably, the above face image is a grayscale image.

[0054] In order to extract comprehensive and strongly expressive features from the face image and the depth image to accurately recognize the face, so as to obtain a highly accurate heat map and angle information for the face, the technical solution provided in this embodiment can use an artificial intelligence learning algorithm to process the face image and the depth image. Thus, the above 1022 "Process the face image and the depth image to obtain a face heat map and face angle information" can specifically include:

[0055] 10221. Obtain a processing model;

[0056] 10222. Use the processing model to process the face image and the depth image to obtain the face heat map and the face angle information.

[0057] In the above 10221, the processing model can be a convolutional network model, which can be but is not limited to being obtained by improving and training the original Center-net model. The Center-net model is a model constructed based on the Center-net algorithm. Specifically, when training the improved Center-net model, data augmentation methods can be used to obtain a training sample set. Through data augmentation, more training samples can be generated from a small amount of existing training samples for training, thereby improving the model detection accuracy. Commonly used data augmentation methods can be divided into: data augmentation based on geometric transformation, such as translation, shearing, adding noise, scaling, rotation, etc.; data augmentation based on color space transformation, such as adjusting brightness, contrast, saturation, grayscale conversion, etc. In addition, there are also some other data augmentation methods, such as Mosaic data augmentation, etc.

[0058] For example, through the translation method, each training sample can be translated to expose different degrees of the face included in each training sample, such as exposing 20% - 50% of the face, so as to generate a training sample set containing only partial faces for training to obtain the processing model described in this embodiment, that is, the convolutional network model.

[0059] The above convolutional network model includes at least two convolutional layers with convolutional operations and edge padding operations. Among them, the operation parameters corresponding to the edge padding operations of the two convolutional layers are different. The above edge padding operation (padding) is also called edge zero-padding operation, which means adding several circles of pixels with a value of "0" to the outer edge of the original image. The operation parameter corresponding to the edge padding operation is the number of added circles (padding circles). If the target face is missing, when using this processing model to process the face image and depth map, these two convolutional layers with edge padding operations are used to supplement the data of the information related to the target face in the face image and depth map.

[0060] Figure 3 shows a schematic diagram of the network architecture of the above-mentioned convolutional network model. As Figure 3 , this convolutional network model includes a first feature extraction network, a second feature extraction network, and an output prediction network. Among them,

[0061] The above first feature extraction network is mainly used to perform preliminary pre-feature extraction processing on the input image, and it can be, but is not limited to, a Backblone network.

[0062] The above second feature extraction network is an extension and expansion after the first feature extraction network, and is used to further process the feature map data output by the first feature extraction network to obtain a more effective feature map convenient for the output prediction network to identify. It can be, but is not limited to, a ResNet (Residual Neural Network). In this second feature extraction network, the second convolutional layer with a stride Stride = 4 and the third convolutional layer with a stride Stride = 8 downstream of it are two convolutional layers with convolutional operations and edge padding operations, and the operation parameter corresponding to the edge padding operation of the second convolutional layer is greater than the operation parameter corresponding to the edge padding operation of the third convolutional layer.

[0063] The above output prediction network includes at least a first network branch and a second network branch. The first network branch is used to detect a face and generate a corresponding face heat map for the detected face. The white bright area in the face heat map represents the center point position of the face. This white bright area is often a Gaussian circle. Specifically, in the face heat map, a Gaussian circle with a radius of R is rendered with the center point of the face as the Gaussian center to represent the corresponding face. For example, see Figure 4 shown in the face heat map Heat2. The white bright area in the face heat map Heat2 represents Figure 1b shown in the center point position of the user's face. Among them, the above face heat map Heat2 is obtained by using the Figure 3 shown convolutional network model for Figure 1bobtained by processing the face image shown in Figure 4 , and the center point of the face in the face heat map Heat2 falls within the heat map (specifically within the extended area), which enables subsequent detection of the face and the face direction (i.e., the angle between the face and the terminal) based on the heat map. Among them, the area enclosed by the positive square frame S in the middle of the face heat map Heat2 is equivalent to Figure 4 the face heat map Heat1 shown in Figure 1b . The face heat map Heat1 shown in

[0064] is the heat map of the face obtained by processing the face image shown in

[0065] using the original Center-net model. In this face heat map Heat1, the center point of the face is outside the heat map, which makes it impossible to implement face detection based on the heat map subsequently. The second network branch is used to predict the roll angle of the face (roll, the angle of the face rotating laterally), and this roll angle of the face can be used as the angle between the face and the terminal. In addition to the above, the output prediction network may also include other network branches, such as the third network branch for regression prediction of the facial feature points (Landmark) of the face and the fourth network branch for regression prediction of the face size.

[0066] Based on the above content, in a specific implementable solution, the above 10222 "processing the face image and the depth image using the processing model to obtain the face heat map and the face angle information" can be implemented by the following steps:

[0067] S01. Respectively perform pre-feature extraction processing on the face image and the depth image to obtain corresponding feature map data;

[0068] S02. Perform a convolution operation on the feature map data and then perform an edge extension operation to obtain a first convolution result;

[0069] S03. Perform a convolution operation on the first convolution result and then perform an edge extension operation to obtain a second convolution result;

[0070] S04. Perform a convolution operation on the second convolution result to obtain a third convolution result; Figure 3, the above step S01 can be completed by the first feature extraction network in the convolutional network model. After the first feature extraction network performs pre-feature extraction processing on the face image and the depth image respectively, the output feature map data (including the feature map corresponding to the face image and the feature map corresponding to the depth image) will be used as the input of the second extraction network model to trigger the execution of the second extraction network model to complete the above steps S02 - S04. Specifically, when the second extraction network model is executed, after the feature map data reaches the second convolutional layer therein, it will be convolved with the second convolutional kernel corresponding to the second convolutional layer according to the stride Stride = 4 and then perform an edge expansion operation according to the padding number 20 to obtain the first convolution result C1; further, the first convolution result C1 is used as the input of the third convolutional layer, and when the third convolutional layer is executed, it will be convolved with the third convolutional kernel corresponding to the third convolutional layer according to the stride Stride = 8 and then perform an edge expansion operation according to the padding number 10 to obtain the second convolution result C2; still further, the second convolution result C2 is used as the input of the fourth convolutional layer, and when the fourth convolutional layer is executed, the second convolution result will be convolved with the fourth convolutional kernel corresponding to the fourth convolutional layer to obtain the third convolution result C3.

[0071] Here, it should be supplemented that in the above second extraction network, if there is also another convolutional layer connected upstream of the second convolutional layer with the stride Stride = 4, such as a first convolutional layer with the stride Stride = 2, then: the feature map data output by the first extraction network, after being input into the second extraction network, can first enter the first convolutional layer with the stride Stride = 2, and the convolution result C0 obtained after convolution with the first convolutional kernel corresponding to the first convolutional layer is used as the input of the second convolutional layer.

[0072] The above-obtained third convolution result C3 will be used as the input of the output prediction network to trigger the execution of the output prediction network to complete the above step S05. Specifically, the third convolution result C3 will be sent to each network branch in the output prediction network for processing to obtain at least a face heat map and face angle information (i.e., the angle information between the face and the terminal).

[0073] The above face heat map can reflect the position of the face and is obtained by processing the third convolution result C3 through the first network branch.

[0074] In the above-mentioned face angle information, the angle can be the roll angle of the face. Correspondingly, it can be obtained by processing the third convolution result C3 through the second network branch. Considering that if the second network branch is designed in the traditional way to use angle regression to calculate the roll angle of the face, there is often a jump between 0 degrees and 360 degrees, which affects the angle accuracy. Therefore, in order to effectively optimize the angle calculation and improve the angle determination accuracy, in the embodiment of the present application, the second network branch will not use regression to calculate the roll angle of the face, but adopt an angle classification method to determine the roll angle of the face. Specifically, in the second network branch, multiple angle classifications are pre-divided. For example, if each interval of 10° is divided into an angle classification, then 36 angle classifications are pre-divided, that is, 0-10°, 11°-20°, 21°-30°, and so on are each an angle classification in turn; by processing the third convolution result C3 through the second network branch, the corresponding target angle classification can be determined for each face from multiple angle classifications, so as to determine the roll angle of the face (that is, the angle between the face and the terminal) according to the target angle classification corresponding to each face. Of course, in other embodiments, other types of angles of the face can also be used as the angle between the face and the terminal. For example, the angle between the line connecting two specific facial points of the face and the display direction of the interface displayed on the terminal screen can be used as the angle between the face and the terminal. The facial feature points of the face can include but are not limited to the feature points representing the nose, eyes, eye corners, lips, cheeks, etc., and can be obtained by processing the third convolution result C3 through the third network branch in the output prediction network. From the above content, a specific implementation solution of "obtaining the face angle information based on the third convolution result" in the above S05 can be implemented by any one of the following steps:

[0075] ① Based on the third convolution result, determine the target angle classification corresponding to each face from multiple angle classifications; according to the target angle classification corresponding to each face, determine the angle between each face and the terminal; or

[0076] ② Based on the third convolution result, determine the facial feature points of each face; according to the specific facial points of each face, determine the angle between each face and the terminal.

[0077] In the above ①, several angle classifications with the highest scores among multiple angle classifications can be determined as the target angle classifications, and then the angle between the face and the terminal can be calculated according to the target angle classification and its corresponding score. Thus, the above "based on the third convolution result, determine the target angle classification corresponding to each face from multiple angle classifications, and determine the angle between each face and the terminal according to the target angle classification corresponding to each face" can be implemented by the following specific steps:

[0078] S11. Based on the third convolution result, determine the scores corresponding to each of the multiple angle classifications for each face;

[0079] S12. Determine the angle between each human face and the terminal according to the set number of angle classifications ranked by scores and the corresponding scores.

[0080] In specific implementation, the multiple angle classifications can be sorted from high to low according to the scores of each angle classification. Then, select the set number of angle classifications ranked at the front as the target angle classifications. The set number can be flexibly set, such as two, three, etc., which is not limited here. Finally, according to the target angle classifications and their corresponding scores, the angle between the human face and the terminal can be determined by weighted summation with the score as the weight.

[0081] For example, for a human face, the determined target angle classifications are 0 - 10°, 11° - 20°, the score corresponding to 0 - 10° is 0.8, and the score corresponding to 11° - 20° is 0.75. Then the angle between this human face and the terminal can be: 0.8 * 0 - 10° + 0.75 * 11° - 20° = 8.25° - 23°.

[0082] For the implementation description in the above ②, reference can be made to the relevant content in the context, and details will not be elaborated here.

[0083] Further, after obtaining the human face heat map and the human face and angle information, in order to accurately determine which human face should be used to control the rotation of the interface displayed on the terminal screen in the case of multiple human faces, so as to avoid the interface direction from jumping back and forth, the solution of the embodiment of the present application will perform post - processing analysis on the human face heat map and the human face angle information to determine the target human face closest to the terminal and the angle corresponding to the target human face, and implement the interface rotation control based on the angle corresponding to this target human face. Based on this, in an implementable solution, the above 1023 "determine the target human face angle based on the human face heat map and the human face angle information" may specifically include:

[0084] 10231. When it is determined based on the human face heat map that there are multiple human faces, determine the target human face with the shortest distance from the multiple human faces according to the distances between each of the multiple human faces included in the human face heat map and the terminal;

[0085] 10232. Obtain the angle corresponding to the target human face from the human face angle information;

[0086] In specific implementation, the number of human faces can be determined based on the distribution information of the human faces in the human face heat map. Based on the above combination Figure 4According to the relevant content described above, a white bright area in the face heat map represents the center point position of a face. That is, a white bright area in the heat map can represent a face. Therefore, the above distribution information can refer to the distribution information of the center point positions of each face obtained from the face heat map. Correspondingly, the distance between each face included in the face heat map and the terminal can refer to the distance from the center point of each face to the terminal screen (more specifically, to the front camera of the terminal). The distance information of the face from the terminal is determined based on the depth map obtained in the above step 1021.

[0087] Based on the distribution information, a face can be selected as the target face to determine whether the interface displayed on the terminal screen needs to be rotated. Specifically, when the number of faces characterized by the above distribution information does not exceed one (when it is one, the one face can be directly selected as the target face); correspondingly, the face angle information only contains one angle, that is, the angle between the one face characterized by the distribution information and the terminal. According to this angle, it can be directly determined whether to trigger the interface rotation and the corresponding rotation direction. Otherwise, when the number of faces characterized by the above distribution information is multiple, correspondingly: the above face heat map contains the distances between multiple faces and the terminal respectively; and the face angle information will contain multiple angles, one angle corresponding to one face. In this case, according to the distances between multiple faces and the terminal respectively included in the face heat map, the face with the shortest distance from the terminal can be determined as the target face; then, further, the angle corresponding to the target face can be determined from the face angle information, and then whether to trigger the interface rotation and the corresponding rotation direction can be determined according to the angle corresponding to the target face.

[0088] The above-mentioned interface can refer to the system interface of the terminal or can also refer to the application interfaces of various applications such as video applications, browsers, and social applications installed on the terminal. This embodiment does not make any limitations in this regard.

[0089] Regarding the content related to step 102 described above, it should be supplemented here that: if it is determined that there is only one face based on the data information, the face map can also be obtained only from the data information, and then the above-mentioned processing model can be used to process the face map to obtain the only face angle (that is, the target face), so as to determine whether to control the interface rotation according to this face angle subsequently.

[0090] In the above 103, it can be determined that it is necessary to control the interface rotation when the target face angle is greater than or equal to the preset angle, and then the face deflection direction can be analyzed in combination with the target face angle to determine the interface rotation direction. For the relevant example description, reference can be made to the relevant content described in the following in combination with the application scenario.

[0091] In summary, for the technical solution provided in this embodiment, in response to a terminal attitude change event, the front camera of the terminal is activated to collect data information, and the target face angle can be determined by processing the data information. Among them, if there is a missing part in the target face, during processing, the information related to the target face in the data information can be supplemented to determine the target face angle based on the supplemented information related to the target face. Then, it is determined whether to control the rotation of the interface displayed by the terminal according to the target face angle. By adopting this solution, even in scenarios where the user's entire face cannot be captured, high accuracy of face recognition detection and face-to-terminal angle calculation can be ensured through data supplementation, thereby effectively optimizing the interface rotation control. Moreover, this also makes the applicable scenarios of this solution more extensive and has fewer limitations, which helps to improve the user experience of the terminal.

[0092] Based on the above content, the present application also provides two embodiments of the interface rotation control method from other perspectives. The execution subject of the other method embodiments is also the terminal. For the detailed description of the terminal, reference can be made to the relevant content in the context of the present application. Specifically,

[0093] Figure 5 FIG. shows a schematic flowchart of an interface rotation control method provided by another embodiment of the present application, which mainly describes this solution from the perspective of a multi-face scenario. As Figure 5 shown, the interface rotation control method includes the following steps:

[0094] 201. In response to a terminal attitude change event, activate the front camera of the terminal to collect data information;

[0095] 202. When it is determined that there are multiple faces based on the data information, select a target face from the multiple faces according to the distance information included in the data information;

[0096] 203. Determine whether to control the rotation of the interface displayed by the terminal according to the target face.

[0097] For the detailed implementation of the above 201, reference can be made to the relevant content in other embodiments above.

[0098] In the above 202-203, a face image and a depth image corresponding to the face and including distance information can be determined based on the data information. By performing image recognition and analysis on the face image, it can be determined whether there are multiple faces and the missing situation of each face. When there is no missing face among multiple faces, the face closest to the terminal can be directly selected from the multiple faces as the target face according to the distance information included in the depth image, and the existing solution or the method provided in this application can be used to determine the target face angle (such as the roll angle of the target face, the angle between the line connecting two facial feature points (such as two eye feature points) of the target face and the terminal), so as to determine whether to control the rotation of the interface displayed by the terminal according to the target face angle. However, if there is one or two or more missing faces among multiple faces, considering that using the existing solution for face recognition detection, face angle determination, etc. may cause problems such as false face recognition and low accuracy of angle calculation, resulting in inaccurate subsequent interface rotation control. Therefore, in order to better handle the face missing scenario, this solution can supplement the data of the information related to the missing face in the face image and the depth image, and then analyze the number of faces, the target face, the target face angle, etc. based on the supplemented face-related information, so as to determine whether to control the interface rotation and the corresponding rotation direction according to the target face.

[0099] Based on the above content, in a feasible solution, in the above 202, "selecting a target face from the multiple faces based on the distance information included in the data information" may specifically include:

[0100] 2021. Select the face closest to the terminal from the multiple faces as the target face according to the distance information included in the data information.

[0101] Further, the above step 1021 can be implemented through the following steps:

[0102] 20211. Based on the data information, obtain a face image including multiple faces and a depth image including the distance information;

[0103] 20212. Process the face image and the depth image to obtain a face heat map;

[0104] 20213. Determine the face closest to the terminal as the target face according to the distances between the multiple faces included in the face heat map and the terminal;

[0105] Among them, if any face among the multiple faces is missing, when processing the face image and the depth image, the information related to the missing face in the face image and the depth image can be supplemented with data, so as to determine the face heat map based on the supplemented face-related information.

[0106] In specific implementation, this embodiment processes the face image and the depth image by using a preset processing model. Through this processing model, data can be supplemented for the information related to the face with missing parts in the face image and the depth image, and a face heat map can be determined based on the supplemented face-related information. In addition, face angle information, facial feature point information of the face, etc. can also be determined. For the detailed description of the processing model and the use of the processing model to process the face image and the depth image, reference can be made to the relevant content in other embodiments above in this application, and no specific elaboration will be made here.

[0107] In the above 203, it is possible to determine whether to control the rotation of the interface according to the target face angle (i.e., the angle between the target face and the terminal), and if it is determined that it is necessary, the corresponding rotation direction, rotation angle, etc. can be further determined, so as to control the rotation of the interface according to the determined rotation direction and rotation angle, etc. For the detailed description of the determination of the target face angle, reference can also be made to the relevant content in other embodiments above.

[0108] The solution provided in this embodiment, by responding to the terminal posture change event, will start the front camera of the terminal to collect data information, and when it is determined that there are multiple faces based on this data information, a target face can be selected from the multiple faces according to the distance information included in the data information, and then it can be determined whether to control the rotation of the interface displayed by the terminal according to the target face. It can be seen that this solution can solve the problem of interface rotation control in the multi-user face scenario, which helps to improve the user experience of using the terminal.

[0109] For the specific implementation description of each step described above in this embodiment, reference can be made to the relevant content in other embodiments above. In addition to the above steps, other steps may also be included in the embodiments of this application. For the other steps that may be included and their specific implementation description, reference can also be made to the relevant content in other embodiments above.

[0110] Figure 6 shows a schematic flowchart of an interface rotation control method provided by another embodiment of this application. As Figure 6 shown, this interface rotation control method includes the following steps:

[0111] 301. In response to a terminal posture change event, obtain a grayscale image and a corresponding depth image through the front camera of the terminal, where at least one face is included in the grayscale image, and the distance from each face to the terminal is included in the depth image;

[0112] 302. Process the grayscale image and the depth image to determine the target face angle;

[0113] 303. Determine whether to trigger the rotation of the interface displayed by the terminal according to the target face angle;

[0114] Wherein, the target face is one of the at least one face; if any face among the at least one face has a defect, during processing, data supplementation can be performed on the information related to the face with the defect in the grayscale image and the depth image, so as to determine the target face angle based on the supplemented face-related information.

[0115] In the above, a preset processing model can be used to process the grayscale image and the depth image to determine a face heat map, face angle information, etc., and then based on the face heat map, a target face is determined from at least one face, and the angle corresponding to the target face is obtained from the face angle information, and the angle corresponding to the target face is the target face angle.

[0116] For the specific implementation descriptions of the above steps 301 to 303 and the above-mentioned processing model, reference can be made to the relevant content in other embodiments above. In addition to the above steps, other steps may also be included in the embodiments of the present application. For the other steps that may be included and their specific implementation descriptions, reference can also be made to the relevant content in other embodiments above.

[0117] In the technical solution provided in this embodiment, in response to a terminal attitude change event, a grayscale image including at least one face and a corresponding depth image can be obtained through the front camera of the terminal. The depth image includes the distance from each face to the terminal. Then, by processing the grayscale image and the depth image, the angle between the target face and the terminal can be obtained, so as to determine whether to trigger the rotation of the interface displayed by the terminal according to the angle between the target face and the terminal. Wherein, the above target face is one of the at least one face; if any face among the at least one face has a defect, during processing the grayscale image and the depth image, data supplementation can be performed on the information related to the face with the defect in the grayscale image and the depth image, so as to determine the target face angle based on the supplemented face-related information. By adopting this solution, even in scenarios where the user's entire face cannot be collected, high accuracy of face recognition detection and angle calculation between the face and the terminal can be ensured through data supplementation, so as to effectively optimize the interface rotation control. In addition, the problem of interface rotation control in a multi-face scenario can also be solved. In summary, the solution of the present application has a wider applicable scenario and less limitation, which helps to improve the user experience of the terminal.

[0118] To facilitate understanding of the solution of the present application, the following combines Figures 7 to 9 Taking the terminal 6 as a smart phone (hereinafter referred to as a mobile phone) as an example, several application scenarios are combined to describe the technical solution of the present application.

[0119] Application scenario 1: The user changes from lying prone and looking at the mobile phone to lying on the side and looking at the mobile phone

[0120] As shown in Figure 7, in the default state, the auto - rotation function of the mobile phone is in the off state. When the mobile phone receives an instruction input by the user clicking on the auto - rotation control 60 in the control center interface, the auto - rotation control 60 is in a selected state, and the function of rotating the screen interface (i.e., the interface displayed on the screen) is enabled. Assume that under the condition that the function of rotating the screen interface of the mobile phone is enabled, the user initially lies prone to look at the mobile phone (such as lying prone to watch a video) and the mobile phone is in a vertical state relative to the ground, and then changes to lying on the side to look at the mobile phone (lying on the side to watch a video) and the mobile phone is at a certain inclination angle relative to the ground (roughly in a horizontal state). When the user changes to lying on the side to look at the mobile phone, the mobile phone detects that its own posture has changed, activates the front - facing camera, and uses the front - facing camera to collect data information in front of the mobile phone (i.e., the direction where the mobile phone screen faces). Based on this data information, a face image of the user and the corresponding depth map can be obtained. The face image is as Figure 1b shown, which is a grayscale image and contains a small part of a user's face. Take this face image and depth map as Figure 3 the input of the convolutional neural network model shown in, trigger the execution of the convolutional neural network model, and a face heat map Heat2 and the roll angle of the face as shown in Figure 4 on the right side will be obtained. The heat map Heat2 reflects a face (i.e., there is a user in front of the mobile phone screen). At this time, it is possible to directly determine whether it is necessary to trigger the rotation of the interface F currently displayed on the mobile phone screen and the corresponding rotation direction according to the roll angle of the face. For example, the roll angle of the face is 278.25° - 293°, indicating that when the user lies on the side, the face turns 67° - 81.75° to the left. At this time, the angle of the face turning to the left is greater than the preset angle (such as 45°), it is determined that it is necessary to trigger the rotation of the currently displayed interface F, and the rotation direction is the clockwise direction. Based on the above determination result, the currently displayed interface F can be controlled to rotate 90° clockwise. Figure 7 The interface F’ shown in is the interface F after rotating 90° clockwise.

[0121] Or, take this face image and depth map as Figure 3 the input of the convolutional neural network model shown in, trigger the execution of the convolutional neural network model, and obtain a face heat map Heat2 and the facial feature points of the face as shown in Figure 4 on the right side. Assume that the facial feature points include a specific point c1 of the left eye and a specific point c2 of the right eye (determined by data supplementation), then: as shown in Figure 8, in a coordinate system established with a corner endpoint of the mobile phone as the origin, the line parallel to the horizontal side of the mobile phone screen as the X-axis, and the line parallel to the vertical side of the mobile phone screen as the Y-axis, there is an angle θ between the line connecting the specific point c1 of the left eye and the specific point c2 of the right eye on the user's face and the positive direction of the Y-axis. This angle θ is used as the angle between the face and the terminal. Based on this angle θ, it can be determined whether to trigger the rotation of the interface F currently displayed on the mobile phone screen and the corresponding rotation direction. For example, when the angle θ is in the interval [-45°, -90°], the currently displayed interface F is controlled to rotate clockwise by 90°.

[0122] Application scenario two: Multiple people watching the mobile phone together

[0123] As shown in Figure 9 , the automatic rotation function of the mobile phone is in the on state. Users u1 and u2 initially sit together and watch the mobile phone (such as watching a video). After a period of time, the two users change to lying on their sides together to watch the mobile phone. At this time, the mobile phone detects a change in its own posture and activates the front camera to obtain the face image and the corresponding depth image of the users in front of the mobile phone using the front camera. Among them, the face image contains at least part of the faces of users u1 and u2 respectively. Taking this face image and depth image as Figure 3 the input of the convolutional network model shown in

[0124] Here, it should be supplemented and explained that: The solutions provided in the embodiments of the present application can be applied not only to the interface rotation control scenario described above, but also to any other scenarios that require face detection, such as face number detection, facial feature point detection of the face, and so on.

[0125] Figure 10 shows a schematic structural diagram of an interface rotation control device provided in an embodiment of the present application. As shown in Figure 9 , the interface rotation control device includes: a startup module 31, a processing module 32, and a determination module 33; among them,

[0126] The startup module 31 is used to activate the front camera of the terminal to collect data information in response to a terminal posture change event;

[0127] A processing module 32 processes the data information to determine the target face angle. Wherein, if there is a missing part in the target face, relevant information related to the target face in the data information can be supplemented during processing, so as to determine the target face angle based on the supplemented relevant information of the target face.

[0128] A determination module 33 is configured to determine whether to control the rotation of the interface displayed on the terminal according to the target face angle.

[0129] Further, when the processing module 32 is used to process the data information to determine the target face angle, it is specifically configured to: obtain a face image and a corresponding depth image based on the data information; the face image contains at least one face, and the target face is one of the at least one face, and the depth image contains the distance from each face to the terminal; process the face image and the depth image to obtain a face heat map and face angle information; determine the target face angle based on the face heat map and the face angle information.

[0130] Further, when the processing module 32 is used to process the face image and the depth image to obtain a face heat map and face angle information, it is specifically configured to: obtain a processing model; use the processing model to process the face image and the depth image to obtain the face heat map and the face angle information.

[0131] Further, the processing model is a convolutional network model, and the convolutional network model includes at least two convolutional layers with edge expansion operations, and the operation parameters corresponding to the edge expansion operations of the two convolutional layers are different; and, if there is a missing part in the target face, when using the processing model to process the face image and the depth image, the two convolutional layers are used to supplement the relevant information of the target face in the face image and the depth image.

[0132] Further, when the processing module 32 is used to process the face image and the depth image with the processing model to obtain the face heat map and the face angle information, it is specifically configured to: perform pre-feature extraction processing on the face image and the depth image respectively to obtain corresponding feature map data; perform a convolution operation on the feature map data and then perform an edge expansion operation to obtain a first convolution result; perform a convolution operation on the first convolution result and then perform an edge expansion operation to obtain a second convolution result; perform a convolution operation on the second convolution result to obtain a third convolution result; obtain the face heat map and the face angle information based on the third convolution result.

[0133] Further, when the processing module 32 is used to obtain the face angle information based on the third convolution result, it specifically is used for: based on the third convolution result, determining the target angle classification corresponding to each face from multiple angle classifications, and determining the angle between each face and the terminal according to the target angle classification corresponding to each face; or, based on the third convolution result, determining the facial feature points of each face, and determining the angle between each face and the terminal according to the facial feature points of each face.

[0134] Further, when the processing module 32 is used to determine the target angle classification corresponding to each face from multiple angle classifications based on the third convolution result, and determine the angle between each face and the terminal according to the target angle classification corresponding to each face, it specifically is used for: based on the third convolution result, determining the score corresponding to each of the multiple angle classifications for each face; determining the angle between each face and the terminal according to the set number of angle classifications with the top scores and the corresponding scores.

[0135] Further, when the determining module 33 is used to determine the target face angle according to the face heat map and the face angle information, it specifically is used for: when it is determined based on the face heat map that there are multiple faces, determining the target face with the shortest distance from the terminal to the multiple faces according to the distance information included in the face heat map; obtaining the angle corresponding to the target face from the face angle information.

[0136] Further, the above front camera is a depth camera, and the face image is a grayscale image or a color image.

[0137] It should be noted here that: the interface rotation control device provided in the above embodiment can implement the technical solution described in the above Figure 2 method embodiment shown. The specific implementation principle of each of the above modules or units can be referred to the corresponding content in the above corresponding method embodiment, and will not be elaborated here.

[0138] Another embodiment of the present application further provides an interface rotation control device. The structure of the device can be referred to Figure 11 . Specifically, the interface rotation control device includes: a start module 41, a selection module 42, and a determination module 43; wherein,

[0139] The start module 41 is used to start the front camera of the terminal to collect data information in response to a terminal attitude change event;

[0140] The selection module 42 is used to, when it is determined based on the data information that there are multiple faces, select a target face from the multiple faces according to the distance information included in the data information;

[0141] A determination module 43, configured to determine whether to control the rotation of the interface displayed on the terminal according to the target face.

[0142] Further, when the above-mentioned selection module 42 is configured to select a target face from the multiple faces according to the distance information included in the data information, it is specifically configured to: according to the distance information included in the data information, select the face closest to the terminal among the multiple faces as the target face.

[0143] Further, when the above-mentioned 43 is configured to select the face closest to the terminal among the multiple faces as the target face according to the distance information included in the data information, it may be specifically configured to: based on the data information, obtain a face map including multiple faces and a depth map including the distance information; process the face map and the depth map to obtain a face heat map; determine the face closest to the terminal as the target face according to the distances between the multiple faces included in the face heat map and the terminal; wherein, if any face among the multiple faces is missing, when processing the face map and the depth map, data can be supplemented for the information related to the missing face in the face map and the depth map, so as to determine the face heat map based on the supplemented face-related information.

[0144] It should be noted here that: the interface rotation control device provided in the above embodiment can implement the technical solutions described in the above Figure 5 method embodiment shown. The specific implementation principles of the above-mentioned modules or units can be referred to the corresponding content in the above-mentioned corresponding method embodiment, and will not be elaborated here.

[0145] Another embodiment of the present application further provides an interface rotation control device, and the structure of the device can be referred to Figure 12 . Specifically, the interface rotation control device includes: an acquisition module 51, a processing module 52, and a determination module 53; wherein,

[0146] The acquisition module 51 is configured to, in response to a terminal attitude change event, acquire a grayscale image and a corresponding depth image through the front camera of the terminal; wherein, at least one face is included in the grayscale image, and the distance from each face to the terminal is included in the depth image;

[0147] The processing module 52 is configured to process the grayscale image and the depth image to determine the target face angle;

[0148] The determination module 53 is configured to determine whether to trigger the rotation of the interface displayed on the terminal according to the target face angle;

[0149] Wherein, the target face is one of the at least one face; if any face among the at least one face has a missing part, during processing, data supplementation can be performed on the information related to the face with the missing part in the grayscale image and the depth image, so as to determine the target face angle based on the supplemented face-related information.

[0150] It should be noted here that: the interface rotation control device provided in the above embodiment can implement the Figure 6 technical solutions described in the method embodiment shown above. The specific implementation principles of the above modules or units can be referred to the corresponding content in the corresponding method embodiment above, and will not be elaborated here.

[0151] The internal functions and structures of the interface rotation control device have been described above. In practice, the interface rotation control device can be implemented on a terminal device. The terminal device can be various portable and movable electronic devices such as smart phones, tablet computers, PDAs (Personal Digital Assistants), netbooks, and e-books. Based on this, another embodiment of the present application provides a terminal device. Figure 13 The structural schematic diagram of the terminal device is shown. As Figure 13 described, the terminal device includes:

[0152] A screen 63 for displaying an interface;

[0153] A front camera 61, which is arranged on the side of the terminal device where the screen is located, is used to collect image data in front of the terminal device, and send the image data to the control device, so that the control device can obtain a face image and a corresponding depth image of the user in front of the terminal device based on the image data, wherein the face image is a grayscale image or a color image;

[0154] A control device 62, which is communicatively connected to the front camera, is used to implement the steps in the interface rotation control method provided in the embodiments of the present application.

[0155] The above screen 63 may include an LCD (Liquid Crystal Display) display screen or an OLED (Organic Light-Emitting Diode) display screen for displaying the system of the terminal or the interfaces of various applications. And the above screen 63 may further include a touch panel (TP), etc. If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations.

[0156] The above-mentioned front camera 61 is located on the side where the screen of the terminal device is located, that is, the front (the front side) of the terminal device. For a detailed description of the front camera 61, reference can be made to the relevant content in other embodiments above.

[0157] And the above-mentioned terminal device may further include a memory 64. The memory 64 is used to store computer programs and can be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application program or method for operating the terminal device, etc. The memory 64 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0158] The above control device 62 includes a processor. The processor is coupled to the memory 64 and is used to execute the computer program in the memory 64 to implement the steps or functions in the interface rotation control method provided in the embodiments of the present application.

[0159] Furthermore, as Figure 13 shown, the terminal device further includes: a communication component 65, a power supply component 66, an audio component 67, a sensor component (not shown in the figure), and other components. Among them,

[0160] The above communication component 65 is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0161] The above power supply component 66 provides power for various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0162] The above audio component 67 can be configured to output and / or input an audio signal. For example, the audio component includes a microphone (MIC). When the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component further includes a speaker for outputting an audio signal.

[0163] The above sensor component may include, but is not limited to, an attitude sensor and the like described in other embodiments above.

[0164] Figure 13 Only some components of the terminal device are schematically shown, which does not mean that the terminal device only includes Figure 13 the components shown.

[0165] Accordingly, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, it can implement each step executable by the terminal device in the above method embodiment.

[0166] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer-usable storage media includes, but is not limited to, disk memories, CD-ROMs, optical memories, etc.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one or more flows in the flowchart and / or one or more blocks in the block diagram.

[0168] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the function specified in one or more blocks of a flowchart and / or one or more blocks of a block diagram.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing steps for implementing the function specified in one or more blocks of a flowchart and / or one or more blocks of a block diagram when executed on the computer or other programmable apparatus.

[0170] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0171] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0172] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0173] It should also be noted that the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0174] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. An interface rotation control method, characterized in that, it includes: Collect data information in response to a terminal attitude change event; Process the data information to determine the target face angle; wherein, if the target face is missing, when processing, data supplementation is performed on the information related to the target face in the data information, and the target face angle is determined based on the supplemented information related to the target face; According to the target face angle, determine whether to control the rotation of the interface displayed by the terminal.

2. The method according to claim 1, characterized in that, Processing the data information to determine the target face angle includes: Based on the data information, obtain a face image and a corresponding depth map; the face image contains at least one face, the target face is one of the at least one face, and the depth map contains the distance from each face to the terminal; Process the face image and the depth map to obtain a face heat map and face angle information; Based on the face heat map and the face angle information, determine the target face angle.

3. The method according to claim 2, characterized in that, Processing the face image and the depth map to obtain a face heat map and face angle information includes Obtain a processing model; Use the processing model to process the face image and the depth map to obtain the face heat map and the face angle information.

4. The method according to claim 3, characterized in that, The processing model is a convolutional network model, and the convolutional network model includes at least two convolutional layers with edge expansion operations, and the operation parameters corresponding to the edge expansion operations of the two convolutional layers are different; and If the target face is missing, when using the processing model to process the face image and the depth map, the two convolutional layers are used to supplement the information related to the target face in the face image and the depth map.

5. The method according to claim 4, characterized in that, Using the processing model to process the face image and the depth map to obtain the face heat map and the face angle information includes: Perform pre-feature extraction processing on the face image and the depth map respectively to obtain corresponding feature map data; Perform a convolution operation on the feature map data and then perform an edge expansion operation to obtain a first convolution result; Perform a convolution operation on the first convolution result and then perform an edge expansion operation to obtain a second convolution result; Perform a convolution operation on the second convolution result to obtain a third convolution result; Based on the third convolution result, obtain the face heat map and the face angle information.

6. The method according to claim 5, characterized in that, Based on the third convolution result, obtaining the face angle information includes: Based on the third convolution result, determine the target angle classification corresponding to each face from multiple angle classifications; according to the target angle classification corresponding to each face, determine the angle between each face and the terminal; or Based on the third convolution result, determine the facial feature points of each face; according to the facial feature points of each face, determine the angle between each face and the terminal.

7. The method according to claim 6, wherein, based on the third convolution result, determining a target angle classification corresponding to each face from multiple angle classifications, and determining the angle between each face and the terminal according to the target angle classification corresponding to each face, includes: based on the third convolution result, determining scores corresponding to each of the multiple angle classifications for each face; determining the angle between each face and the terminal according to a set number of angle classifications with the top - ranked scores and the corresponding scores.

8. The method according to any one of claims 2 to 7, wherein, determining the target face angle based on the face heat map and the face angle information, includes: when it is determined that there are multiple faces based on the face heat map, determining a target face with the shortest distance from the multiple faces to the terminal according to the distances between the multiple faces included in the face heat map and the terminal; obtaining the angle corresponding to the target face from the face angle information.

9. The method according to any one of claims 2 to 7, wherein, the front - facing camera is a depth camera, and the face image is a grayscale image or a color image.

10. An interface rotation control method, wherein, includes: responding to a terminal posture change event, collecting data information; when it is determined that there are multiple faces based on the data information, selecting a target face from the multiple faces according to the distance information included in the data information; determining whether to control the rotation of the interface displayed on the terminal according to the target face; wherein, if any one of the multiple faces has a missing part, by supplementing the data of the information related to the face with the missing part in the data information, determining the target face based on the supplemented information related to the face.

11. The method according to claim 10, wherein, selecting a target face from the multiple faces according to the distance information included in the data information, includes: selecting the face with the shortest distance to the terminal from the multiple faces as the target face according to the distance information included in the data information.

12. The method according to claim 11, wherein, selecting the face with the shortest distance to the terminal from the multiple faces as the target face according to the distance information included in the data information, includes: based on the data information, obtaining a face image including multiple faces and a depth image including the distance information; processing the face image and the depth image to obtain a face heat map; determining the face with the shortest distance to the terminal as the target face according to the distances between the multiple faces included in the face heat map and the terminal; wherein, if any one of the multiple faces has a missing part, when processing the face image and the depth image, the data of the information related to the face with the missing part in the face image and the depth image can be supplemented to determine the face heat map based on the supplemented information related to the face.

13. An interface rotation control method, wherein, includes: In response to a terminal attitude change event, obtain a grayscale image and a corresponding depth image; wherein, at least one face is included in the grayscale image, and the distance from each face to the terminal is included in the depth image; Process the grayscale image and the depth image to determine the target face angle; Determine whether to trigger the rotation of the interface displayed by the terminal according to the target face angle; Wherein, the target face is one of the at least one face; if any face among the at least one face is missing, during processing, data supplementation is performed on the information related to the missing face in the grayscale image and the depth image, and the target face angle is determined based on the supplemented face-related information.

14. A terminal device, characterized in that, it includes: a screen for displaying an interface; a front camera disposed on the side where the screen of the terminal device is located; a control device communicatively connected to the front camera for implementing the steps in the interface rotation control method according to any one of claims 1 to 9, or the steps in the interface rotation control method according to any one of claims 10 to 12, or implementing the steps in the interface rotation control method according to claim 13.

Images