Determination system, electronic system, and determination method

Abstract

The invention provides a determination system, an electronic system, a determination method and a display method. The judgment method comprises the following steps: receiving an image by a feature acquisition module, and acquiring a first key point coordinate, a second key point coordinate and a size of a face frame of a user based on the image; and executing the following steps by the judgment module: obtaining a judgment value based on the ordinate of the first key point coordinate, the ordinate of the second key point coordinate and the size of the face frame; and sending a rotation signal in response to the judgment value satisfying the rotation condition.

Description

Technical Field

[0001] The present disclosure relates to the technical field of electronic system display methods, and in particular to the technology of applying key points of users in images. Background Art

[0002] In recent years, many technology products have been introduced into smart applications, and some functions rely on the use of cameras, such as human presence detection (HPD), face recognition, gesture recognition, gaze detection, etc. However, depending on the usage scenario, there will be situations where the screen needs to be rotated, and the performance of these technologies will often be affected. Although using sensors such as accelerometers or gyroscopes to obtain rotation information can help improve, the judgment results of the sensors may not necessarily be conducive to smart applications. Summary of the invention

[0003] In view of this, some embodiments of the present invention provide a determination system, an electronic system, a determination method and a display method to improve the problems of the prior art.

[0004] Some embodiments of the present invention provide a judgment system, comprising a feature acquisition module and a judgment module; the feature acquisition module is configured to receive an image, and obtain a first key point coordinate, a second key point coordinate, and a size of a face frame of a user based on the image; the judgment module is configured to perform the following steps: obtain a judgment value based on the ordinate of the first key point coordinate, the ordinate of the second key point coordinate, and the size of the face frame; and send a rotation signal in response to the judgment value satisfying a rotation condition.

[0005] Some embodiments of the present invention provide an electronic system, comprising the aforementioned determination system and a display module; the display module is configured to change the orientation direction of the screen display content in response to receiving the aforementioned rotation signal.

[0006] Some embodiments of the present invention provide a judgment method, comprising: receiving an image by a feature acquisition module, and obtaining a first key point coordinate, a second key point coordinate, and a size of a face frame of a user based on the image; and the judgment module executing the following steps: obtaining a judgment value based on the vertical coordinate of the first key point coordinate, the vertical coordinate of the second key point coordinate, and the size of the face frame; and sending a rotation signal in response to the judgment value satisfying a rotation condition.

[0007] Some embodiments of the present invention provide a display method using the aforementioned determination method, comprising: a display module changes the orientation direction of screen display content in response to receiving a rotation signal.

[0008] Based on the above, the judgment system, electronic system, judgment method and display method provided by the embodiments of the present invention take into account the use of the user's key points to determine whether to send a rotation signal. They are suitable for applications such as human presence detection and face recognition that require user detection. Through the prediction of the model itself, it is possible to know whether there is a need for rotation and switch to the model of the corresponding situation without having to obtain information from the sensor, thereby reducing the computing time cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of a determination system according to some embodiments of the present invention. Figure 2 It is a schematic diagram of the operation of a feature acquisition module according to some embodiments of the present invention. Figure 3 It is a schematic diagram of the operation of a feature acquisition module according to some embodiments of the present invention. Figure 4A is a block diagram of a neural network module according to some embodiments of the present invention. Figure 4B is a block diagram of an output feature tensor generation module according to some embodiments of the present invention. Figure 5 is a block diagram of a fusion module according to some embodiments of the present invention. Figure 6 is a schematic diagram of a prediction module structure according to some embodiments of the present invention. Figure 7 is a schematic diagram of an information tensor structure according to some embodiments of the present invention. Figure 8 is a block diagram of an electronic system according to some embodiments of the present invention. Fig. 9 It is a schematic diagram of the operation of an electronic system according to some embodiments of the present invention. Fig.10 It is a schematic diagram of the structure of an electronic device according to some embodiments of the present invention. Fig.11 It is a flow chart of a determination method according to some embodiments of the present invention. Fig.12 is a flow chart of face frame size calculation according to some embodiments of the present invention. Fig.13 is a flow chart of determination value calculation according to some embodiments of the present invention. Fig.14 It is a flow chart of a determination method according to some embodiments of the present invention. Fig.15 is a flow chart of a display method according to some embodiments of the present invention. DETAILED DESCRIPTION

[0010] The aforementioned and other technical contents, features and effects of the present invention will be clearly presented in the detailed description of the embodiments with reference to the accompanying drawings below. The thickness or size of each element in the accompanying drawings is expressed in an exaggerated, omitted or approximate manner for those skilled in the art to understand and read, and the size of each element is not completely its actual size, and is not used to limit the limiting conditions that the present invention can implement, so it has no technical substantive significance, and any structural modification, change in proportional relationship or adjustment of size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope of the technical content disclosed in the present invention. The same reference numerals will be used to represent the same or similar elements in all the drawings. In this article, the words similar to ordinal numbers such as "first" or "second" described are used to distinguish or refer to elements or structures associated with the same or similar elements, and do not necessarily imply the order of these elements in the system. It should be understood that in some cases or configurations, ordinal words can be used interchangeably without affecting the implementation of the present invention.

[0011] Figure 1 is a block diagram of a determination system according to some embodiments of the present invention. Figure 2 It is a schematic diagram of the operation of a feature acquisition module according to some embodiments of the present invention. Figure 3 is a schematic diagram of the operation of the feature acquisition module according to some embodiments of the present invention. Figures 1 to 3 The judging system 100 includes a feature acquisition module 101 and a judging module 102. The feature acquisition module 101 is configured to receive an image 103 (e.g. Figure 2 Image 201 or Figure 3 301), and obtain the first key point of the user based on the image 103 (e.g. Figure 2 The first key point of 2016, Figure 3 The first key point coordinate (x 1 ,y 1 ), the second key point (e.g. Figure 2 The second key point of 2017, Figure 3 The second key point coordinates (x 2 ,y 2 ) and face frames (such as Figure 2 Face frame 2011, Figure 3 The size of the face frame 3011).

[0012] It is worth noting that the aforementioned coordinates are pixel coordinates of the image 103, that is, the coordinates of the upper left corner vertex of the image 103 are set to (0, 0), the first coordinate component of the coordinates is the position counted from left to right based on the upper left corner vertex of the image 103, and the second coordinate component of the coordinates is the position counted from top to bottom based on the upper left corner vertex of the image 103.

[0013] The first component of the coordinate is called the abscissa, and the second component of the coordinate is called the ordinate. Therefore, the coordinates of the first key point (x 1 ,y 1 ) and the coordinates of the second key point (x 2 ,y 2 ) are x 1 and x 2 , the coordinates of the first key point (x 1 ,y 1 ) and the coordinates of the second key point (x 2 ,y 2 ) are y 1 and 2 .

[0014] In some embodiments of the present invention, the first key point and the second key point are selected to be two points symmetrical to the midline of the human body. Figure 2 as well as Figure 3 As shown, the first key point is selected as the right shoulder point (such as Figure 2 The first key point of 2016, Figure 3 The second key point is selected as the left shoulder point (eg Figure 2 The second key point of 2017, Figure 3 The position indicated by the second key point 3017).

[0015] In some implementations of the present invention, the aforementioned face frame (e.g. Figure 2 Face frame 2011, Figure 3 The size of the face frame 3011 is selected as the height h of the face frame (e.g. Figure 2 as well as Figure 3 The height h of the indicated face frame).

[0016] The following will describe in detail the determination methods of some embodiments of the present invention and how the modules of the determination system 100 work in coordination with the accompanying drawings.

[0017] Fig.11 is a flow chart of a determination method according to some embodiments of the present invention. Figure 1 , Figure 2 , Figure 3as well as Fig.11 .exist Fig.11 In the illustrated embodiment, the determination method includes steps S1101 and S1102. In step S1101, the feature acquisition module 101 receives the image 103, and obtains the coordinates of the first key point of the user (eg, Figure 2 as well as Figure 3 (x 1 ,y 1 )), the second key point coordinates (e.g. Figure 2 as well as Figure 3 (x 2 ,y 2 )) and face frames (e.g. Figure 2 Face frame 2011, Figure 3 In step S1102, the judging module 102 executes steps S11021 and S11022. In step S11021, the judging module 102 determines the size of the face frame 3011 based on the coordinates of the first key point (e.g. Figure 2 as well as Figure 3 (x 1 ,y 1 ))'s ordinate, the coordinates of the second key point (e.g. Figure 2 as well as Figure 3 (x 2 ,y 2 )) and the size of the face frame to obtain a judgment value. In step S11022, the judgment module 102 confirms whether the judgment value meets the rotation condition. If the judgment module 102 confirms that the judgment value meets the rotation condition, the judgment module 102 executes step S11023. In step S11023, in response to the judgment value meeting the rotation condition, the judgment module 102 sends a rotation signal. If the judgment module 102 confirms that the judgment value does not meet the rotation condition, it returns to step S1101, waits for the feature acquisition module 101 to receive the next image, and then continues to execute steps S1101 and S1102.

[0018] Fig.12 is a flowchart of face frame size calculation according to some embodiments of the present invention. In some embodiments of the present invention, the feature acquisition module 101 acquires the aforementioned face frame (e.g. Figure 2 Face frame 2011, Figure 3 The upper left point of the face frame 3011 (e.g. Figure 2 The upper left point 2012 of the face frame 2011, Figure 3 The upper left point coordinates of the face frame 3011 (upper left point 3012) and the lower right point (e.g. Figure 2 The lower right point of the face frame 2011 2013, Figure 3The above-mentioned step S11011 includes steps S1201 and S1202. The feature acquisition module 101 obtains the face frame (e.g. Figure 2 Face frame 2011, Figure 3 In step S1201, the feature acquisition module 101 subtracts the ordinate of the lower right point of the face frame from the ordinate of the upper left point of the face frame to obtain a difference, and the difference is the height h of the face frame. In step S1202, the feature acquisition module 101 sets the face frame (e.g. Figure 2 Face frame 2011, Figure 3 That is, the feature acquisition module 101 uses the above difference as the size of the face frame.

[0019] It is worth noting that in the aforementioned embodiment, the feature acquisition module 101 subtracts the ordinate of the lower right point of the face frame from the ordinate of the upper left point of the face frame to obtain the height h of the face frame. Of course, the feature acquisition module 101 may also subtract the ordinate of the upper right point of the face frame (e.g. Figure 2 The upper right point of the face frame 2011 2014, Figure 3 The vertical coordinate of the upper right point coordinate of the face frame 3011 (the upper right point 3014) minus the lower left point of the face frame (for example Figure 2 The lower left point of the face frame 2011 2015, Figure 3 The ordinate of the lower left point coordinate of the lower left point 3015 of the face frame 3011 is calculated to obtain the height h of the face frame, which is not limited in the present invention.

[0020] It is also worth noting that in some embodiments of the present invention, the aforementioned face frame (e.g. Figure 2 Face frame 2011, Figure 3 The size of the face frame 3011 is selected as the width w of the face frame (eg Figure 2 as well as Figure 3 The feature acquisition module 101 subtracts the horizontal coordinate of the lower left point of the face frame from the horizontal coordinate of the upper right point of the face frame to obtain the width w of the face frame.

[0021] Fig.13 is a flowchart of the determination value calculation according to some embodiments of the present invention. Fig.13 ,exist Fig.13 In the illustrated embodiment, the aforementioned step S11021 includes steps S1301 and S1302. In step S1301, the determination module 102 calculates the coordinates of the second key point (eg Figure 2 as well as Figure 3 (x2 ,y 2 )) and the coordinates of the first key point (e.g. Figure 2 as well as Figure 3 (x 1 ,y 1 )) is the absolute value of the difference in the ordinates. Figure 2 as well as Figure 3 Taking the illustrated embodiment as an example, the determination module 102 calculates |y 2 -y 1 In step S1302, the judgment module 102 sets the judgment value to the coordinates of the second key point (eg Figure 2 as well as Figure 3 (x 2 ,y 2 )) and the coordinates of the first key point (e.g. Figure 2 as well as Figure 3 (x 1 ,y 1 )) and the absolute value of the difference between the ordinates of the face frame (e.g. Figure 2 Face frame 2011, Figure 3 The ratio of the size of the face frame 3011 (for example, the height h of the face frame). That is, the determination module 102 sets

[0022] Following the above Fig.13 In the illustrated embodiment, in some embodiments of the present invention, the rotation condition in the aforementioned step S11022 is that the judgment value is greater than or equal to a preset value.

[0023] It is worth noting that in the above-mentioned embodiment, the formula for determining the value mainly uses the size of the face frame (e.g., the height h of the face frame), the ordinate (y) of the first key point coordinate (e.g., the right shoulder point), and the ordinate (y) of the first key point coordinate (e.g., the right shoulder point). 1 ) and the ordinate (y 2 ). Since the first key point and the second key point are selected as two points symmetrical to the midline of the human body, in general, the vertical coordinates of the first key point and the second key point (such as the left and right shoulder points) will not have a large difference, and the ratio of the difference between the two to the scale of the face frame should be very small. Furthermore, in the above embodiment, since the proportion of the face frame is close to a square, even if a rotation is performed (such as Figure 2 , Figure 3 ), the height h of the face frame has only a slight change (i.e. Figure 2 h and Figure 3There is only a slight change in h of the first key point and the second key point (such as the left and right shoulder points), but there is an obvious difference in the vertical coordinates. At this time, the ratio of the height h of the face frame will increase significantly. When it exceeds a certain threshold (such as greater than or equal to the aforementioned preset value), the determination module 102 sends a rotation signal.

[0024] This invention is particularly suitable for applications such as human presence detection and facial recognition, which require user detection. Through the prediction of the model itself, it is possible to know whether there is a need for rotation and switch to the model of the corresponding situation. There is no need to obtain information from sensors, thus reducing the time cost of calculation.

[0025] Figure 4A is a block diagram of a neural network module according to some embodiments of the present invention. Figures 1 to 3 as well as Figure 4A In some embodiments of the present invention, the feature acquisition module 101 includes a neural network module 400. The neural network module 400 is configured to receive the image 103 and output the coordinates of the first key point of the user (e.g. Figure 2 as well as Figure 3 (x 1 ,y 1 )), the second key point coordinates (e.g. Figure 2 as well as Figure 3 (x 2 ,y 2 The aforementioned step S1101 includes receiving the image 103 by the neural network module 400, and outputting the first key point coordinates, the second key point coordinates and the size of the face frame of the aforementioned user by the neural network module 400.

[0026] Various implementations of the neural network module 400 are further described below. In some embodiments of the present invention, the neural network module 400 includes an output feature tensor generation module 401 and prediction modules 402-1 to 402-M, where M is a positive integer greater than or equal to 2. The output feature tensor generation module 401 is configured to generate a plurality of output feature tensors of different sizes based on the image 103. Each of the prediction modules 402-1 to 402-M is configured to receive one of the aforementioned output feature tensors to generate a corresponding information tensor, that is, each of the prediction modules 402-1 to 402-M generates an information tensor. The information tensor is configured to indicate a face frame (e.g. Figure 2 Face frame 2011, Figure 3 The face frame 3011) position information, confidence score information, category information and the first key point coordinates (eg Figure 2 as well as Figure 3 (x 1 ,y 1)) location information, the coordinates of the second key point (e.g. Figure 2 as well as Figure 3 (x 2 ,y 2 The feature acquisition module 101 outputs the first key point coordinates, the second key point coordinates and the size of the face frame of the user based on all the information tensors generated by each of the prediction modules 402-1 to 402-M.

[0027] Fig.14 is a flow chart of a determination method according to some embodiments of the present invention. Fig.14 ,exist Fig.14 In the illustrated embodiment, the aforementioned step S1101 includes steps S1401 to S1403. In step S1401, the output feature tensor generation module 401 generates a plurality of output feature tensors of different sizes based on the image 103. In step S1402, each of the prediction modules 402-1 to 402-M receives one of the aforementioned plurality of output feature tensors to generate an information tensor accordingly. As described above, the information tensor is configured to indicate a face frame (e.g., Figure 2 Face frame 2011, Figure 3 In step S1403, the feature acquisition module 101 outputs the user's first key point coordinates, second key point coordinates, and the size of the face frame based on all information tensors generated by each of the prediction modules 402-1 to 402-M.

[0028] Figure 4B is a block diagram of an output feature tensor generation module according to some embodiments of the present invention.

[0029] Please also see Figure 4A as well as Figure 4B In the following description, for the sake of convenience, M=3 is used for explanation.

[0030] The output feature tensor generation module 401 includes a backbone module 4011 and a feature pyramid module 4012. The image 103 is, for example, a tensor with a dimension of 256×256×3 or a tensor with a dimension of 256×256×1.

[0031] In some embodiments of the present invention, the backbone module 4011 includes backbone layers 40111 to 40114 of different sizes. The backbone module 4011 is configured to generate a plurality of feature tensors of different sizes and having a first order based on the image 103 through the backbone layers 40111 to 40114. Figure 4B As shown, the aforementioned multiple feature tensors are: the output tensor of the backbone layer 40112, the output tensor of the backbone layer 40113, and the output tensor of the backbone layer 40114. The first order is the order in which these feature tensors are arranged from large to small according to their sizes. It is worth noting that, although in this embodiment, the backbone module 4011 only includes 4 backbone layers, ordinary technicians in the technical field of the invention can use other numbers of backbone layers according to needs, and the present invention is not limited to this. It is also worth noting that, although in Figure 4B In the illustrated embodiment, backbone layers 40111 to 40114 are connected in series, but backbone layers 40111 to 40114 may also be connected in series and in parallel, which is not limited in the present invention. Feature pyramid module 4012 is configured to perform feature fusion on the aforementioned feature tensors to obtain multiple output feature tensors.

[0032] Please also see Figure 4B as well as Fig.14 In some embodiments of the present invention, the aforementioned step S1401 includes the following first step and second step. In the first step, the backbone module 4011 generates multiple feature tensors of different sizes and having a first order based on the image 103 through the backbone layer 40111 to the backbone layer 40114, wherein the first order is the order in which these feature tensors are arranged from large to small according to their sizes. In the second step, the feature pyramid module 4012 performs feature fusion on the aforementioned feature tensors of different sizes and having a first order generated by the backbone module 4011 through the backbone layer 40111 to the backbone layer 40114 based on the image 103 to obtain an output feature tensor.

[0033] Please refer to Figure 4B In some embodiments of the present invention, the feature pyramid module 4012 includes a fusion module 40121-1 and a fusion module 40121-2. The feature pyramid module 4012 is configured to perform the following steps to perform feature fusion on the aforementioned feature tensors to obtain multiple output feature tensors.

[0034] First, the feature pyramid module 4012 sets the minimum feature tensor corresponding to the last one in the first order as one of the temporary feature tensor sets. Figure 4B Taking the illustrated embodiment as an example, the minimum feature tensor is the output tensor of the backbone layer 40114, and the minimum feature tensor is stored in the temporary feature tensor 40122-3 as one of the temporary feature tensor sets.

[0035] Next, the feature pyramid module 4012 performs an upsampling operation on the temporary feature tensor 40122-3 via the fusion module 40121-1 to obtain the upsampled temporary feature tensor 40122-3 of the same size as the output tensor of the backbone layer 40113. The feature pyramid module 4012 then performs feature fusion on the upsampled temporary feature tensor 40122-3 and the output tensor of the backbone layer 40113 via the fusion module 40121-1 to obtain the temporary feature tensor 40122-2 of the same size as the output tensor of the convolution layer of the backbone layer 40113. The feature pyramid module 4012 then performs feature fusion on the upsampled temporary feature tensor 40122-2 and the output tensor of the convolution layer of the backbone layer 40112 via the fusion module 40121-2 to obtain the temporary feature tensor 40122-1 of the same size as the output tensor of the convolution layer of the backbone layer 40112.

[0036] The feature pyramid module 4012 outputs the temporary feature tensor 40122-3, the temporary feature tensor 40122-2, and the temporary feature tensor 40122-1 as multiple output feature tensors of the aforementioned feature pyramid module 4012. It is worth noting that in the aforementioned embodiment, the generation of three output feature tensors of different sizes is used as an example for explanation, and a person skilled in the art of the present invention can arbitrarily increase the number of backbone layers in the backbone module 4011 and the number of fusion modules in the feature pyramid module 4012 according to the aforementioned public technology to obtain any number of output feature tensors of different sizes, and the present invention is not limited to the generation of three output feature tensors of different sizes.

[0037] Figure 5is a block diagram of a fusion module according to some embodiments of the present invention. In some embodiments of the present invention, the structures of the fusion module 40121-1 and the fusion module 40121-2 are as shown in the fusion module 500, and the fusion module 500 includes an upsampling module 501, a point-by-point convolution layer 502, and a point-by-point addition module 503. The upsampling module 501 is configured to perform an upsampling operation on the input of the upsampling module 501. In some embodiments of the present invention, the upsampling operation is to repeat the operation twice in the height axis and width axis directions of the input of the upsampling module 501 so that the size of the input of the upsampling module 501 is converted to twice the original size. The point-by-point convolution layer 502 performs a point-by-point convolution operation, wherein the point-by-point convolution operation is defined as performing a convolution operation on a tensor with a convolution kernel of dimension 1×1×C, and C is the number of channels of the input of the point-by-point convolution layer 502. The point-by-point addition module 503 is configured to perform point-by-point addition operations on the two received input tensors to obtain an output tensor of the point-by-point addition module 503. It is worth noting that the upsampling operation used in this embodiment is an implementation method of upsampling, and the upsampling module 501 can use other upsampling methods, which are not limited by the present invention.

[0038] Figure 6 is a schematic diagram of a prediction module structure according to some embodiments of the present invention. Figure 7 is a schematic diagram of an information tensor structure according to some embodiments of the present invention. Figure 6 as well as Figure 7 In this embodiment, the structures of prediction modules 402-1 to 402-3 are shown in the prediction module 600. The prediction module 600 is configured to include t W p ×H p ×128 convolutional layers - convolutional layers 601-1 to 601-t, and a W p ×H p ×PA's convolutional layer - convolutional layer 602, where t is a positive integer, indicating the number of convolutional layers 601-1 to 601-t; W p and H p is a positive integer, indicating the dimensions of the width axis and height axis of the convolutional layers 601-1 to 601-t; A is a positive integer, indicating the number of detection boxes (anchors); P is a positive integer. It is worth noting that the above-mentioned W p ×H p The convolution layer marked with ×128 means that the convolution layer performs convolution operation on the input tensor with 128 convolution kernels, and concatenates the tensors obtained by convolution operation on the input tensor in sequence to obtain a width axis number W. p , the number of height axes is Hp And the output tensor with the number of channels in the channel axis being 128, such an output tensor has a dimension of W p ×H p ×128 tensor. Similarly, W p ×H p A convolutional layer of ×PA means that the convolutional layer performs convolution operations on the input tensor with PA convolution kernels, and stacks the tensors obtained by convolution operations on the input tensor with PA convolution kernels in sequence to obtain a width axis number W. p , the number of height axes is H p and the number of channels of the channels axis for the output tensor of PA.

[0039] The neural network module 400 sets A detection boxes of different sizes on the aforementioned multiple output feature tensors. In this embodiment, the value of P is 4+1+the number of all categories+3, where 4 represents the number of tensor elements required to describe the position coordinates of a vertex of the detection box and the detection width and height, 1 represents the possibility of describing the detection target in the detection box and the accuracy of the detection box with 1 tensor element, and 3 represents the number of tensor elements required to describe the first angle, second angle, and third angle of the face. p , H p The values ​​of , P, A, and t can be set by the user according to the needs, and the present invention does not limit them. It is worth noting that since the output feature tensors received by the prediction modules 402-1 to 402-M have different sizes, the W of each prediction module 402-1 to 402-M is p and H p The values ​​are different.

[0040] The prediction module 600 is configured to receive any one of the aforementioned multiple output feature tensors. After the output feature tensor passes through the convolutional layers 601-1 to 601-t and the convolutional layer 602 of the category prediction module 600, an information tensor 701 can be obtained. The information tensor 701 includes sub-information tensors 701-1 to 701-A, and each of the sub-information tensors 701-1 to 701-A corresponds to a detection frame in the aforementioned A detection frames. Each of the sub-information tensors 701-1 to 701-A includes W p ·H p P-dimensional vector. Figure 7As shown, each P-dimensional vector includes tensor elements 7021-1, 7021-2, 7022-1, 7022-2...702N-1, 702N-2, 703~708, wherein tensor elements 7021-1, 7021-2, 7022-1, 7022-2...702N-1, 702N-2 indicate the horizontal coordinates and vertical coordinates of N key points of the user, that is, 7021-1 indicates the horizontal coordinate of the 1st key point, tensor element 7021-2 indicates the vertical coordinate of the 1st key point, 7022-1 indicates the horizontal coordinate of the 2nd key point, tensor element 7022-2 indicates the vertical coordinate of the 2nd key point, and so on. It is worth noting that, in some embodiments of the present invention, the aforementioned N is 2, 7021-1 indicates the horizontal coordinate of the aforementioned first key point (e.g., right shoulder point), tensor element 7021-2 indicates the vertical coordinate of the aforementioned first key point (e.g., right shoulder point), tensor element 7022-1 indicates the horizontal coordinate of the aforementioned second key point (e.g., left shoulder point), and tensor element 7022-2 indicates the vertical coordinate of the aforementioned second key point (e.g., left shoulder point). In some embodiments of the present invention, the aforementioned N is greater than 2, and tensor elements 7021-1, 7021-2, 7022-1, 7022-2...702N-1, 702N-2 include the vertical coordinates and horizontal coordinates of the first key point and the second key point (e.g., left and right shoulder points).

[0041] Tensor element 703 includes multiple sub-tensor elements, each of which indicates the probability that the object in the detection frame belongs to each category, tensor element 704 indicates a confidence score, which indicates the possibility that there is a detection target in the detection frame and the accuracy of the detection frame, tensor element 705 indicates the height of the detection frame, tensor element 706 indicates the width of the detection frame, and tensor elements 707 and 708 indicate the coordinates of the detection frame. Tensor elements 7021-1, 7021-2, 7022-1, 7022-2...702N-1, 702N-2 indicate the horizontal coordinates and vertical coordinates of N key points of the user (including the first key point and the second key point of the user), the aforementioned detection frame coordinates, the height of the detection frame and the width of the detection frame are the position information of the aforementioned face frame, the probability that the object in the detection frame belongs to each category is the aforementioned category information, and the aforementioned confidence score is the aforementioned confidence score information. The aforementioned N key points of the user (including the first key point and the second key point of the user) refer to the positions on the user (such as the right shoulder point and the left shoulder point) that are pre-set by the feature acquisition module 101 to be acquired from the image 103 .

[0042] The feature acquisition module 101 integrates all the information tensors generated by each of the prediction modules 402-1 to 402-M to obtain the horizontal and vertical coordinates of the user's N key points (including the user's first key point and second key point). The feature acquisition module 101 then outputs the horizontal and vertical coordinates of the key points required for subsequent processing (such as the user's right shoulder point and left shoulder point).

[0043] The feature acquisition module 101 integrates all the information tensors generated by each of the prediction modules 402-1 to 402-M, and can also obtain the position and category of the face frame. The feature acquisition module 101 can directly calculate the height h and width w of the face frame based on the information of the height and width of the detection frame. The feature acquisition module 101 can also first calculate the four vertices of the face frame (for example Figure 2 The upper left point 2012, the lower right point 2013, the upper right point 2014, the lower left point 2015 of the face frame 2011 and Figure 3 The upper left point 3012, the lower right point 3013, the upper right point 3014, and the lower left point 3015 of the face frame 3011 are calculated, and then the height h or the width w of the face frame is calculated.

[0044] It is worth noting that, although in this embodiment, the contents indicated by each tensor element of the aforementioned P-dimensional vector are arranged as described above, the arrangement order of the contents indicated by each tensor element of the P-dimensional vector is not limited to the aforementioned embodiment.

[0045] It is worth noting that in training Figure 4A , Figure 4B as well as Figure 5 to Figure 7 When the neural network module 400 in the illustrated embodiment is used, the trained neural network module 400 can be obtained by simply adding the horizontal and vertical coordinate data of N key points (including the user's first key point and second key point) to the training set and then training it using the training method of the object detection model.

[0046] In training Figure 4A , Figure 4B as well as Figure 5 to Figure 7In the illustrated embodiment, the neural network module 400 can simultaneously obtain the position and category of an object (a face frame in the aforementioned embodiment) based on all the information tensors generated by each of the prediction modules 402-1 to 402-M, that is, a neural network can simultaneously detect the position of an object and recognize the object. Such an architecture and method is called one-stage object detection. The prediction modules 402-1 to 402-M are referred to as network heads in the technical field of the present invention. The prediction modules 402-1 to 402-M disclosed in the aforementioned embodiments can replace the network heads of other one-stage object detection models so that other one-stage object detection models can output a face frame (e.g. Figure 2 Face frame 2011, Figure 3 The face frame 3011) includes the position information, confidence score information, category information, and the position information of the first key point coordinates and the position information of the second key point coordinates. For example, the prediction modules 402-1 to 402-M disclosed in the aforementioned embodiment may replace the network head of the YOLO model. The present invention is not limited to the aforementioned backbone module 4011 and feature pyramid module 4012.

[0047] Figure 8 is a block diagram of an electronic system according to some embodiments of the present invention. Fig. 9 is a schematic diagram of the operation of an electronic system according to some embodiments of the present invention. Figure 8 as well as Fig. 9 The electronic system 800 includes the judgment system 100 and the display module 801 described in the above embodiment. The display module 801 is configured to change the orientation direction of the screen display content 902 in response to receiving the rotation signal sent by the judgment system 100. The orientation direction of the screen display content 902 refers to the direction in which the display module 801 displays the screen display content 902 on the screen 901.

[0048] by Fig. 9 Taking the illustrated embodiment as an example, the display module 801 initially displays the screen display content 902 on the screen 901 in the orientation direction 903. When the display module 801 receives the rotation signal sent by the determination system 100, the display module 801 changes the orientation direction of the screen display content 902, and the display module 801 displays the screen display content 902 on the screen 901 in the orientation direction 904.

[0049] The following will describe in detail the display methods of some embodiments of the present invention and how the modules of the electronic system 800 cooperate with each other in conjunction with the accompanying drawings. Fig.15 is a flow chart of a display method according to some embodiments of the present invention. Figure 8as well as Fig.15 ,exist Fig.15 In the illustrated embodiment, the display method includes step S1501 . In step S1501 , the display module 801 changes the orientation direction of the screen display content 902 (eg, from the orientation direction 903 to the orientation direction 904 ) in response to receiving the rotation signal.

[0050] Fig.10 FIG. 1 is a schematic diagram of the structure of an electronic device according to some embodiments of the present invention. Fig.10 As shown, at the hardware level, the electronic device 1000 includes processing units 1001-1 to 1001-R, where R is a positive integer. The electronic device 1000 includes an internal memory 1002, a non-volatile memory 1003, and a display element 1004. The internal memory 1002 is, for example, a random access memory (RAM). The non-volatile memory 1003 is, for example, at least one disk storage, etc. The display element 1004 is, for example, a liquid crystal display, a plasma display, a computer display (for example, a video graphics array (VGA) display, a super VGA display, and a cathode ray tube display) and other similar types of display devices, but is not limited thereto. Of course, the electronic device 1000 may also include hardware required for other functions.

[0051] The internal memory 1002 and the non-volatile memory 1003 are used to store programs, which may include program codes, and the program codes include computer operation instructions. The internal memory 1002 and the non-volatile memory 1003 provide instructions and data to the processing units 1001-1 to 1001-R. The processing units 1001-1 to 1001-R read the corresponding computer programs from the non-volatile memory 1003 to the internal memory 1002 and then run them, forming the judgment system 100 and the electronic system 800 at the logical level, and the processing units 1001-1 to 1001-R are specifically used to execute Figure 11 to Figure 15 Of course, the modules of the determination system 100 and the electronic system 800 can also be implemented in hardware, and the present invention is not limited thereto. The display element 1004 can include the aforementioned screen 901.

[0052] The processing units 1001-1 to 1001-R may be an integrated circuit chip with signal processing capabilities. In the implementation process, the methods and steps disclosed in the aforementioned embodiments can be completed by hardware integrated logic circuits or software instructions in the processing units 1001-1 to 1001-R. The processing units 1001-1 to 1001-R can be general-purpose processors, including central processing units (CPU), tensor processing units (Tensor Processing Unit), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, which can implement or execute the methods and steps disclosed in the aforementioned embodiments.

[0053] Examples of computer storage media include, but are not limited to, phase change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other internal memory technology, compact disk read only memory (CD-ROM), digital video disk (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0054] Based on the above, the judgment system, electronic system, judgment method and display method provided by the embodiments of the present invention take into account the use of the user's key points to determine whether to send a rotation signal. They are suitable for applications such as human presence detection and face recognition that require user detection. Through the prediction of the model itself, it is possible to know whether there is a need for rotation and switch to the model of the corresponding situation without having to obtain information from the sensor, thereby reducing the computing time cost.

[0055] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the appended claims.

Claims

1. A judgment system, comprising: a feature acquisition module configured to receive an image and obtain first key point coordinates, second key point coordinates, and a size of a face frame of the user based on the image; and The determination module is configured to perform the following steps: (a) obtaining a determination value based on the ordinate of the first key point coordinates, the ordinate of the second key point coordinates, and the size of the face frame; and (b) issuing a rotation signal in response to the judgment value satisfying the rotation condition.

2. The judgment system as described in claim 1, wherein the coordinates of the first key point are the coordinates of the right shoulder point of the user, and the coordinates of the second key point are the coordinates of the left shoulder point of the user.

3. The determination system of claim 1 , wherein the feature acquisition module is configured to obtain the size of the face frame based on the following steps: subtracting the ordinate of the lower right point coordinate of the face frame from the ordinate of the upper left point coordinate of the face frame to obtain a difference; and The size of the face frame is set to the difference.

4. The determination system as claimed in claim 1, wherein the aforementioned step (a) comprises: Calculating the absolute value of the difference between the ordinate of the second key point coordinate and the ordinate of the first key point coordinate; and The determination value is set to be a ratio of the absolute value of the difference to the size of the face frame. 5 . The judgment system as claimed in claim 4 , wherein the rotation condition is that the judgment value is greater than or equal to a preset value.

6. The judgment system according to claim 1, wherein: The feature acquisition module includes a neural network module, which is configured to receive the image and output the first key point coordinates, the second key point coordinates and the size of the face frame of the user.

7. The judgment system of claim 6, wherein the neural network module comprises an output feature tensor generation module and a plurality of prediction modules, wherein the output feature tensor generation module is configured to generate a plurality of output feature tensors of different sizes based on the image; each of the plurality of prediction modules is configured to receive one of the plurality of output feature tensors to correspondingly generate an information tensor, wherein the information tensor is configured to indicate position information, confidence score information, category information, and position information of the first key point coordinates and position information of the second key point coordinates of the face frame; and the feature acquisition module outputs the first key point coordinates, the second key point coordinates and the size of the face frame of the user based on all the information tensors generated by each of the plurality of prediction modules.

8. An electronic system comprising the judgment system according to any one of claims 1 to 7, comprising: The display module is configured to change the orientation direction of the screen display content in response to receiving the rotation signal.

9. A judgment method, comprising: (a) receiving an image by a feature acquisition module, and obtaining first key point coordinates, second key point coordinates and a size of a face frame of a user based on the image; and (b) The judgment module executes the following steps: (b1) obtaining a determination value based on the ordinate of the first key point coordinates, the ordinate of the second key point coordinates, and the size of the face frame; and (b2) issuing a rotation signal in response to the judgment value satisfying the rotation condition.

10. The determination method as claimed in claim 9, wherein the coordinates of the first key point are the coordinates of the right shoulder point of the user, and the coordinates of the second key point are the coordinates of the left shoulder point of the user.

Images