Method, device and computer host for determining whether a person is paying attention to an object

Abstract

The application discloses a method and device for judging whether a person is paying attention to a target object and a computer host device, wherein when the target attention judgment is performed, the distance between a face in a to-be-recognized image and a camera and the coordinate of a face center point are calculated first, and then the actual space coordinate of the face relative to the camera is obtained through the two data, which is equivalent to obtaining the actual space position of the face, and then the observation angle threshold interval corresponding to the face can be calculated through the actual space position of the face, and finally, whether the current posture angle of the face is in the observation angle threshold interval can be compared, so that whether the face is paying attention to the target object can be judged, and through the foregoing design, when the target attention judgment is performed, the observation angle threshold corresponding to each face can be calculated, so that whether the face at any space position is paying attention to the target object can be accurately judged, and the judgment accuracy is improved.

Description

Technical Field

[0001] This invention belongs to the field of behavior recognition technology, specifically relating to a method, apparatus, and computer main unit for determining whether a person is paying attention to a target object. Background Technology

[0002] In many industries today, it is necessary to promptly perceive the audience's focus, thereby understanding their interests and judgments in order to improve design solutions and promote industry development. For example, in elevators, the level of audience attention can be statistically analyzed by determining whether viewers are paying attention to advertising devices. Based on the statistical results, the most attention-grabbing advertisements can be matched to improve the effectiveness of the advertisements.

[0003] Currently, to determine whether an audience is paying attention to a target, eye-tracking technology is typically used to estimate the point of focus in near-field scenarios, such as helmet-mounted displays. However, in far-field scenarios (above 0.5 meters), current technology struggles to track eye features, making it difficult to estimate the point of focus. Therefore, head pose estimation technology is usually used in far-field scenarios to determine target attention. The principle is to calculate the actual pose angle of the face and then compare it with the set yaw (yaw angle around the Y-axis of the Cartesian coordinate system) and pitch (pitch angle around the x-axis of the Cartesian coordinate system) values ​​to determine whether the face is paying attention to the target.

[0004] However, the aforementioned head pose estimation techniques have the following shortcomings: the yaw and pitch values ​​are preset and cannot be changed during the judgment, while the face pose angle changes with the face's spatial position. Therefore, the aforementioned estimation methods have large errors and cannot accurately determine whether a face at any spatial position is focused on the target. Thus, it is urgent to provide a target focus judgment method with high accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, and computer main unit for determining whether a person is paying attention to a target object, so as to solve the problem that existing target object estimation methods cannot accurately determine whether a face at any spatial location is paying attention to a target object.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a method for determining whether a person is paying attention to a target object, comprising:

[0008] Obtain an image to be identified, wherein the image to be identified contains at least one human face;

[0009] Based on the image to be identified, the distance between the face in the image to be identified and the camera, as well as the coordinates of the center point of the face in the image to be identified are obtained, wherein the camera is mounted on the target object, and the center point of the face is the center point of the outline of the face in the image to be identified;

[0010] Based on the distance between the face and the camera in the image to be identified and the coordinates of the center point of the face, the actual spatial coordinates of the face relative to the camera are obtained;

[0011] Based on the actual spatial coordinates of the face relative to the camera, an observation angle threshold range is obtained, wherein the observation angle threshold range is used to characterize the range of pose angles in which the face can focus on the target object;

[0012] Based on the image to be identified, the current pose angle of the face is obtained;

[0013] Determine whether the current attitude angle is within the observation angle threshold range;

[0014] If so, it is determined that the person corresponding to the face is paying attention to the target object.

[0015] Based on the above-disclosed content, when performing target attention determination, this invention first calculates the distance between the face and the camera in the image to be identified, as well as the coordinates of the face's center point. Then, using the aforementioned two data, it obtains the actual spatial coordinates of the face relative to the camera. This step is equivalent to obtaining the actual spatial position of the face. Next, the observation angle threshold range corresponding to the face can be calculated using the actual spatial position of the face. Finally, by comparing whether the current pose angle of the face is within the observation angle threshold range, it can be determined whether the face is paying attention to the target object. Through the aforementioned design, when performing target attention determination, this invention can calculate the corresponding observation angle threshold for each face, thereby accurately determining whether a face at any spatial location is paying attention to the target object, improving the accuracy of the determination.

[0016] In one possible design, the actual spatial coordinates of the face relative to the camera are obtained based on the distance between the face in the image to be recognized and the camera, as well as the coordinates of the center point of the face. This includes:

[0017] The vertical distance between the face in the image to be identified and the camera in the vertical direction, and the vertical distance between the face and the camera in the horizontal direction are obtained and denoted as the first distance and the second distance, respectively.

[0018] Based on the distance between the face and the camera, the first distance, the second distance, and the coordinates of the center point of the face, the actual offset distance of the face relative to the camera in the vertical direction is obtained;

[0019] Based on the actual vertical offset distance of the face relative to the camera, the first distance, the second distance, and the coordinates of the center point of the face, the actual horizontal offset distance of the face relative to the camera is obtained;

[0020] The horizontal distance between the face and the camera is obtained based on the actual vertical offset distance of the face relative to the camera, the first distance, and the coordinates of the center point of the face.

[0021] The actual spatial coordinates are obtained based on the actual vertical offset distance of the face relative to the camera, the actual horizontal offset distance of the face relative to the camera, and the horizontal distance.

[0022] Based on the above disclosure, this invention discloses the specific calculation process of the actual spatial coordinates of a face. That is, the actual offset distance of the face relative to the camera in the vertical direction is equivalent to the horizontal coordinate value in the actual spatial coordinates, the actual offset distance of the face relative to the camera in the horizontal direction is equivalent to the vertical coordinate value in the actual spatial coordinates, and the horizontal distance is equivalent to the value on the z-axis. Thus, the aforementioned three distances constitute the actual spatial coordinates of the face.

[0023] In one possible design, based on the distance between the face and the camera, the first distance, the second distance, and the coordinates of the center point of the face, the actual offset distance of the face relative to the camera in the vertical direction is obtained, including:

[0024] The actual offset distance h0 of the face in the vertical direction relative to the camera is calculated using the following formula:

[0025]

[0026] In the above formula, Where α represents the vertical offset angle of the face relative to the camera, β represents the horizontal offset angle of the face relative to the camera, h represents the horizontal coordinate of the center point of the face, d1 represents the first distance, d2 represents the second distance, and s represents the distance between the face and the camera.

[0027] In one possible design, the actual offset distance of the face relative to the camera in the horizontal direction is obtained based on the actual offset distance of the face in the vertical direction relative to the camera, the first distance, the second distance, and the coordinates of the center point of the face, including:

[0028] The actual offset distance w0 of the face relative to the camera in the horizontal direction is calculated using the following formula;

[0029]

[0030] In the above formula, Where h0 represents the actual vertical offset distance of the face relative to the camera, α represents the vertical offset angle of the face relative to the camera, β represents the horizontal offset angle of the face relative to the camera, h represents the horizontal coordinate of the center point of the face, d1 represents the first distance, w represents the vertical coordinate of the center point of the face, d2 represents the second distance, and s represents the distance between the face and the camera.

[0031] In one possible design, the horizontal distance between the face and the camera is obtained based on the actual vertical offset distance of the face relative to the camera, the first distance, and the coordinates of the center point of the face, including:

[0032] The horizontal distance d0 is calculated using the following formula;

[0033]

[0034] In the above formula, Where h0 represents the actual offset distance of the face relative to the camera in the vertical direction, α represents the offset angle of the face relative to the camera in the vertical direction, h represents the horizontal coordinate of the center point of the face, and d1 represents the first distance.

[0035] In one possible design, obtaining the vertical distance between the face in the image to be identified and the camera in the vertical direction includes:

[0036] Obtain the wide-angle parameters of the camera and the maximum shooting distance in the vertical direction;

[0037] Based on the wide-angle parameters of the camera and the maximum shooting distance in the vertical direction, the vertical distance between the face in the image to be identified and the camera in the vertical direction is obtained.

[0038] In one possible design, the observation angle threshold range is obtained based on the actual spatial coordinates of the face relative to the camera, including:

[0039] Obtain the height and length of the target object;

[0040] The observation angle threshold range is calculated based on the actual spatial coordinates, the height of the target object, and the length of the target angle.

[0041] Based on the above-disclosed information, since the attitude angle of a human face includes pitch and yaw angles, the observation angle threshold range also includes pitch and yaw angle threshold ranges. The pitch angle is the observation angle of the human face in the vertical direction. Therefore, by adding or subtracting the actual offset distance of the human face in the vertical direction from the height of the target object, we obtain the two observation limit distances of the human face in the vertical direction. At the same time, the human face position and the endpoints corresponding to the two observation limit distances will form a right triangle. The angle between the human face position and the endpoints of the two observation limit distances are the maximum and minimum pitch angles. Finally, using the horizontal distance and the two observation limit distances, and according to the inverse trigonometric function, the two pitch angles can be calculated, thus obtaining the pitch angle threshold range. Similarly, the yaw angle is obtained using the same principle.

[0042] In one possible design, obtaining the distance between the face in the image to be identified and the camera includes:

[0043] Obtain the interpupillary distance of the face in the image to be identified;

[0044] Obtain the trained interpupillary distance fitting regression model, wherein the trained interpupillary distance fitting regression model is used to characterize the functional relationship between the interpupillary distance of a face in any image and the distance from the face to the camera.

[0045] Based on the trained interpupillary distance fitting regression model and the interpupillary distance of the face in the image to be identified, the distance between the face in the image to be identified and the camera is obtained.

[0046] Based on the above-disclosed content, the present invention first constructs an interpupillary distance fitting regression model, that is, by collecting the interpupillary distance of the face in multiple images and the distance from the face to the camera, and using a nonlinear logistic regression model for linear fitting, thereby obtaining the trained interpupillary distance fitting regression model. Finally, by obtaining the interpupillary distance of the face in the image to be recognized, the distance between the face in the image to be recognized and the camera can be obtained.

[0047] In a second aspect, the present invention provides a device for determining whether a person is paying attention to a target object, comprising:

[0048] An acquisition unit is used to acquire an image to be identified, wherein the image to be identified contains at least one human face;

[0049] The parameter calculation unit is used to obtain the distance between the face in the image to be identified and the camera, and the coordinates of the center point of the face in the image to be identified, based on the image to be identified. The camera is mounted on the target object, and the center point of the face is the center point of the outline of the face in the image to be identified.

[0050] The face position calculation unit is used to obtain the actual spatial coordinates of the face relative to the camera based on the distance between the face in the image to be identified and the camera, as well as the coordinates of the center point of the face.

[0051] The observation angle calculation unit is used to obtain the observation angle threshold range based on the actual spatial coordinates of the face relative to the camera, wherein the observation angle threshold range is used to characterize the range of pose angles in which the face can focus on the target object;

[0052] The pose angle calculation unit is used to obtain the current pose angle of the face based on the image to be recognized.

[0053] The judgment unit is used to determine whether the current attitude angle is within the observation angle threshold range, and if the determination is yes, it determines that the person corresponding to the face is paying attention to the target object.

[0054] Thirdly, the present invention provides another device for determining whether a person is paying attention to a target object. Taking a computer main device as an example, the device includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the method for determining whether a person is paying attention to a target object as described in the first aspect or any possible design of the first aspect.

[0055] Fourthly, the present invention provides a storage medium storing instructions that, when executed on a computer, perform the method for determining whether a person is concerned about a target object, as described in the first aspect or any possible design of the first aspect.

[0056] Fifthly, the present invention provides a computer program product comprising instructions that, when executed on a computer, cause the computer to perform the method for determining whether a person is concerned with a target object, as described in the first aspect or any possible design of the first aspect. Attached Figure Description

[0057] Figure 1 A flowchart illustrating the steps of the method for determining whether a person is paying attention to a target object provided by the present invention;

[0058] Figure 2 A schematic diagram illustrating the pose angle of a human face provided by the present invention;

[0059] Figure 3 A longitudinal diagram illustrating the positional relationship between the face and the camera provided by this invention;

[0060] Figure 4 A horizontal diagram illustrating the positional relationship between the face and the camera provided by this invention;

[0061] Figure 5 A schematic diagram of the position of the face tilt angle provided by the present invention;

[0062] Figure 6 This is a schematic diagram of the position of the face yaw angle provided by the present invention;

[0063] Figure 7 A schematic diagram of the device for determining whether a person is paying attention to a target object, provided by the present invention;

[0064] Figure 8 A schematic diagram of the structure of the computer main device provided by the present invention. Detailed Implementation

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be noted that while the description of these embodiments is intended to aid in understanding the invention, it does not constitute a limitation thereof. The specific structural and functional details disclosed herein are merely for describing exemplary embodiments of the invention. However, the invention can be embodied in many alternative forms and should not be construed as being limited to the embodiments described herein.

[0066] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0067] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0068] Example

[0069] like Figure 1As shown, the method for determining whether a person is paying attention to a target object provided in this embodiment can be applied to determining whether a person is paying attention to an advertisement, such as whether a person in an elevator is paying attention to an advertisement playing on an advertising device. Of course, it can also be used for determining attention to a target object in other fields, such as determining whether students are paying attention to the teaching equipment during online teaching, or whether students are paying attention to the blackboard during teaching. In application, this method can calculate the observation angle threshold range corresponding to the face in each image. Thus, it can accurately determine whether a face at any spatial location is paying attention to a target object, improving the accuracy of the judgment and making it suitable for large-scale promotion and application.

[0070] In this embodiment, to facilitate real-time acquisition and judgment of whether a person is paying attention to the target object, for example, a camera is installed on the target object to acquire facial images of the person. At the same time, the model corresponding to this method can be deployed in the camera (it can be deployed in an AI (Artificial Intelligence) camera, but is not limited to) for real-time attention judgment. Of course, the acquired images or videos can also be uploaded to a server, and the server can execute this method to realize the judgment of the person's attention to the target object. The following uses an advertising device in an elevator as an application scenario to illustrate the specific situation, as shown in steps S1 to S7 below.

[0071] S1. Obtain the image to be identified, wherein the image to be identified contains at least one human face.

[0072] Step S1 is the process of acquiring facial images. In practice, a camera can be used to capture video of people in the elevator in real time to obtain facial images of the people, so as to determine whether the people are paying attention to the target object (i.e., the advertising device).

[0073] In specific implementation, this embodiment first calculates the actual spatial position of the face in the elevator based on the distance from the face to the camera in the acquired face image (i.e., the image to be recognized). Then, based on the actual spatial position of the face, it calculates the observation angle threshold range corresponding to the face when it is focused on the target object, that is, it obtains the range of pose angles that the face can focus on the target object. Finally, by calculating the actual pose angle of the face and comparing it with the aforementioned observation angle threshold range, it can be determined whether the face is focused on the target object. The calculation process of the distance from the face to the camera in the image is shown in step S2.

[0074] S2. Based on the image to be identified, obtain the distance between the face in the image to be identified and the camera, and the coordinates of the center point of the face in the image to be identified, wherein the camera is mounted on the target object, and the center point of the face is the center point of the outline of the face in the image to be identified; in specific implementation, the coordinates of the center point of the face in the image to be identified can be obtained based on a face detection algorithm, such as using methods based on Active Shape Model (ASM) and Active Appearance Model (AAM), Cascaded poseregression (CPR), or deep learning to detect two-dimensional face bounding boxes (i.e., outlines), two-dimensional face key regions, and / or two-dimensional face key points in the face image, thereby obtaining the coordinates of the center point of the face.

[0075] Meanwhile, for the distance between the face and the camera in the image to be identified, the distance between the face and the camera in the image to be identified can be predicted by fitting the interpupillary distance of the face in the image and the distance from the face to the camera. In specific implementation, the following steps can be used to fit and regress the interpupillary distance of the face and the distance from the face to the camera.

[0076] Step 1: Obtain the training dataset, which includes multiple face images and the distance from the face to the camera in each face image. The distance from the face to the camera in each face image is recorded as the attention distance.

[0077] Step 2: Obtain the interpupillary distance of the face in each face image. The interpupillary distance is represented by pixel distance, that is, the distance between the two pupils can be obtained by counting the pixels in the image.

[0078] Step 3: Input the interpupillary distance and the attention distance corresponding to each face image into the nonlinear logistic regression model for linear fitting to obtain the fitting logistic regression function based on the interpupillary distance and attention distance of the face. This is the fitting logistic regression function obtained after training with a large amount of data. Finally, the distance between the face and the camera in any image to be recognized can be predicted based on this function. The specific prediction steps are shown in steps S21 to S23.

[0079] S21. Obtain the interpupillary distance of the face in the image to be identified.

[0080] S22. Obtain the trained interpupillary distance fitting regression model, wherein the trained interpupillary distance fitting regression model is used to characterize the functional relationship between the interpupillary distance of a face in any image and the distance from the face to the camera.

[0081] S23. Based on the interpupillary distance fitting regression model and the interpupillary distance of the face in the image to be identified, the distance between the face in the image to be identified and the camera is obtained.

[0082] The principle of steps S21 to S23 is as follows: by obtaining the interpupillary distance of the face in the image to be recognized, and then inputting the interpupillary distance into the trained interpupillary distance fitting regression model (that is, the fitting logistic regression function described above), the distance between the face and the camera can be obtained. The interpupillary distance of the face is equivalent to the variable in the function. After inputting it into the function, the function value can be obtained, which is the distance between the face and the camera.

[0083] After obtaining the coordinates of the center point of the face in the image to be recognized and the distance from the face to the camera, the actual spatial position of the face in the elevator can be calculated so that the observation angle threshold range corresponding to the face can be calculated based on the actual spatial position. The calculation process of the actual spatial position of the face is shown in step S3 below.

[0084] S3. Based on the distance between the face and the camera in the image to be identified and the coordinates of the center point of the face, the actual spatial coordinates of the face relative to the camera are obtained. In specific implementation, the distance of the face relative to the camera in three dimensions can be calculated respectively. That is, a rectangular coordinate system is established with the camera as the origin. The calculation of the actual spatial position of the face is to calculate its distance relative to the camera on the x-axis, y-axis and z-axis. The specific steps are shown in steps S31 to S35 below.

[0085] S31. Obtain the vertical distance between the face in the image to be recognized and the camera in the vertical direction, and the vertical distance between the face and the camera in the horizontal direction, denoted as the first distance and the second distance, respectively. In actual scenarios, the person may not be facing the camera directly, and the person's height may be higher or lower than the camera's installation position. At the same time, the person's standing position may also be at a certain angle to the camera. Therefore, the first distance is to calculate the height difference between the face and the camera, and the second distance is to calculate the horizontal distance between the face and the camera. In specific implementation, this can be achieved by constructing a position map of the face and the camera and using trigonometric functions. For the specific calculation process, please refer to the following steps S311 to S312.

[0086] S311. Obtain the wide-angle parameters of the camera and the maximum shooting distance in the vertical direction.

[0087] S312. Based on the wide-angle parameters of the camera and the maximum shooting distance in the vertical direction, obtain the vertical distance between the face in the image to be identified and the camera in the vertical direction.

[0088] See Figure 3As shown in the figure, α1 represents the wide-angle parameter of the camera, that is, the wide-angle angle of the lens. Figure 3 In this context, h1 represents the maximum vertical shooting distance of the camera. Figure 3 Points O (where the camera is located), A, and B (the edge point captured by the camera in the vertical direction) form a right triangle. Therefore, given the lengths of angle AOB and the right-angled side AB, the length of the right-angled side OA, which is the first length, can be calculated using inverse trigonometric functions.

[0089] Similarly, the calculation principle for the second distance is the same as that for the first distance, which can be found in [reference needed]. Figure 4 As shown, Figure 4 In the diagram, w1 represents the maximum shooting angle of the camera in the horizontal direction, where points C and D are the edge points of the camera's shooting in the horizontal direction, and β1 also represents the wide-angle angle of the camera. Therefore, based on the right triangle OCD, the length of the right-angled side QD, which is the second distance, can be calculated using inverse trigonometric functions.

[0090] After obtaining the vertical distance of the face in the image to be recognized relative to the camera in the vertical and horizontal directions, the actual spatial position (i.e., actual spatial coordinates) of the face in the elevator can be calculated, as shown in steps S32 to S35 below.

[0091] S32. Based on the distance between the face and the camera, the first distance, the second distance, and the coordinates of the center point of the face, the actual offset distance of the face relative to the camera in the vertical direction is obtained.

[0092] Step S32 is essentially calculating the actual height difference between the face and the camera. In practice, this can also be calculated using trigonometric functions based on the position map of the face and the camera. The calculation formula is as follows:

[0093]

[0094] See Figure 3 and Figure 4 As shown, and combined with the above equation (1), it can be seen that, Where α represents the vertical offset angle of the face relative to the camera, β represents the horizontal offset angle of the face relative to the camera, h represents the horizontal coordinate of the face center point (i.e., the vertical offset of the face relative to the camera in the image to be recognized), and d1 represents the first distance. Figure 3 The length of OA, d2 represents the second distance, which is... Figure 3 The length of OD, and s represents the distance between the face and the camera, i.e. Figure 4 and Figure 3 The length of the OE (Optical Point E) is given, where point E represents the position of the face in the image to be recognized. The actual position of the face is generally different from its position in the image. This is due to the camera lens; the actual position of the face is usually in front of the face in the captured image, but both are on the same straight line. Figure 3 and Figure 4 Point F in the diagram represents the actual position of the face in the elevator. See Figure 3. The line segments between points O, E, and A form a right triangle OEA, and angle AOE is angle α. Therefore, the ratio of the vertical offset of the face relative to the camera in the image to be recognized to the first distance is the tangent of angle α. Similarly, see... Figure 4 Since triangle OED is a right triangle and angle EOD is angle β, the cosine of angle β is the ratio of d² to s.

[0095] Using the aforementioned equation (1), the actual vertical offset distance of the face relative to the camera can be calculated, which is the actual height difference between the two. Figure 3 After obtaining the height difference in the vertical direction, h0 in the figure can be used to calculate the actual offset distance of the face relative to the camera in the horizontal direction, as shown in step S33.

[0096] S33. Based on the actual offset distance of the face relative to the camera in the vertical direction, the first distance, the second distance, and the coordinates of the center point of the face, the actual offset distance of the face relative to the camera in the horizontal direction is obtained; in specific implementation, it can be calculated by, but is not limited to, the following formula (2):

[0097]

[0098] In the above formula, Where h0, α, β, h, d1, d2, and s have the same meaning as in equation (1) above, while w in equation (2) represents the ordinate of the center point of the face, which is the horizontal offset of the face in the image to be recognized relative to the camera. See [link to relevant documentation]. Figure 4 As shown, the tangent of angle β can also be derived from the right triangle OED, which will not be elaborated here.

[0099] Similarly, the distance of the face relative to the camera on the z-axis can be obtained through the following step S34.

[0100] S34. Based on the actual vertical offset distance of the face relative to the camera, the first distance, and the coordinates of the center point of the face, the horizontal distance between the face and the camera is obtained; in specific implementations, it can be calculated using, but is not limited to, the following formula:

[0101]

[0102] The meanings of the parameters in equation (3) above can be found in equations (1) and (2), and will not be elaborated here.

[0103] Thus, through the aforementioned steps S32, S33 and S33, the distances of the face relative to the camera on the x-axis, y-axis and z-axis can be obtained. Therefore, by combining the aforementioned three distances, the actual spatial position of the face can be obtained, as shown in step S35.

[0104] S35. Based on the actual offset distance of the face relative to the camera in the vertical direction, the actual offset distance of the face relative to the camera in the horizontal direction, and the horizontal distance, the actual spatial coordinates are obtained, where h0 is the component on the x-axis, w0 is the component on the y-axis, and d0 is the component on the z-axis, that is, the actual spatial coordinates of the face are (h0, w0, d0).

[0105] After calculating the actual spatial coordinates of the face in the elevator, the observation angle threshold range corresponding to the face can be calculated based on these coordinates. In other words, the range of pose angles that the face can focus on is calculated. For details, please refer to step S4 below.

[0106] S4. Based on the actual spatial coordinates of the face relative to the camera, obtain the observation angle threshold interval. This observation angle threshold interval characterizes the range of pose angles within which the face can focus on the target object. In practice, the observation angle threshold interval can also be calculated using trigonometric functions. Furthermore, since the face's pose angles include pitch angle (rotation around the x-axis), yaw angle (rotation around the y-axis), and roll angle (rotation around the z-axis), see [link to relevant documentation]. Figure 2 As shown, the observation angle threshold range should also be set accordingly. Since there is no rotation around the z-axis when focusing on the target object, the observation angle threshold range should include both the pitch angle threshold range and the yaw angle threshold range.

[0107] Optionally, but not limited to, the aforementioned pitch angle threshold range and yaw angle threshold range may be calculated using the following steps S41 and S42.

[0108] S41. Obtain the height and length of the target object, wherein the height and length of the target object are preset by the staff.

[0109] S42. The observation angle threshold range is calculated based on the actual spatial coordinates, the height of the target object, and the length of the target angle.

[0110] The following combination Figure 5 and Figure 6To explain the principles behind steps S41 and S42 mentioned above:

[0111] The reason for the pitch angle of a face relative to the camera is that the height of a person may be higher or lower than the camera. Therefore, after calculating the actual vertical offset distance of the face relative to the camera, the pitch angle of the face can be calculated by combining it with the height of the target object.

[0112] See Figure 5 As shown, points G and H represent the highest and lowest points of the target object, respectively. The length of GH represents the height of the target object. The height corresponding to the maximum pitch angle of the face is the height of the target object plus the actual vertical offset distance of the face relative to the camera, which is GH + h0. Similarly, the height corresponding to the minimum pitch angle of the face is GH - h0. Meanwhile, line segment OH represents the horizontal distance between the face and the camera, which is d0. Therefore, points G, H, and F form a right triangle. Knowing the two legs, the two limits of angle GFH can be determined using inverse trigonometric functions. Figure 5 The two endpoint values ​​of the pitch angle are: maximum pitch angle pitch = arctan((GH+h0) / d0), and minimum pitch angle pitch = arctan((GH-h0) / d0).

[0113] See Figure 6 As shown, the principle for calculating the yaw angle threshold range is the same as that for the pitch angle threshold range. The horizontal distance corresponding to the maximum yaw angle is the length of the target object plus the actual horizontal offset distance of the face relative to the camera, which is IJ + w0. Similarly, the horizontal distance corresponding to the minimum yaw angle is the length of the target object minus the actual horizontal offset distance of the face relative to the camera, which is IJ - w0. Points I, J, and F form a right triangle. Therefore, knowing the two legs, the two extreme values ​​of angle IFJ can be determined using inverse trigonometric functions. Figure 6 The two endpoint values ​​of the yaw angle are: the maximum yaw angle is arctan((IJ+w0) / d0), and the minimum yaw angle is arctan((IJ-w0) / d0).

[0114] Therefore, the aforementioned pitch angle threshold range and yaw angle threshold range can be used to form the observation angle threshold range. Then, it is only necessary to calculate the actual pose angle of the face in the image to be identified and compare it with the observation angle threshold range to determine whether the face is paying attention to the target object, as shown in steps S5 to S7 below.

[0115] S5. Based on the image to be identified, obtain the current pose angle of the face. In specific implementation, for example, but not limited to, obtaining the key points of the face in the image to be identified to obtain the corresponding pose angle, the specific calculation process is shown in steps S51 and S52 below.

[0116] S51 performs facial landmark detection on the image to be identified to obtain the coordinates of n facial landmarks, where n is a positive integer. In this embodiment, facial landmark detection refers to locating the key areas of the face given a face image, including eyebrows, eyes, nose, mouth, and facial contours. In specific implementation, facial landmark detection can be achieved using, but is not limited to, methods based on Active Shape Model (ASM) and Active Appearance Model (AAM), Cascaded pose regression (CPR), or deep learning. That is, it detects two-dimensional face bounding boxes, two-dimensional key facial regions, and / or two-dimensional facial landmarks in the face image.

[0117] After obtaining n first facial key points, coordinate transformation can be performed, that is, the face position in the camera coordinate system can be converted into the face position in the Cartesian coordinate system to obtain the face pose angle. The specific calculation process is shown in step S52 below.

[0118] S52. Based on the coordinates of the n facial key points, obtain the current pose angle of the face; in specific implementation, the facial key point coordinates can be converted into pose angles using the following specific steps, but not limited to: First step: Match the detected facial key point coordinates with the corresponding facial key points in the 3D face model (pre-set by the user); Second step: Solve the transformation relationship matrix between the original facial key points and the corresponding 3D facial key points (the solvePnP function of OpenCV (a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source)) to obtain the rotation vector, and then obtain the rotation relationship matrix through the rotation vector); Third step: Solve the three Euler angles of the face relative to the camera coordinate system based on the rotation relationship matrix (i.e., the second pose angle corresponding to the current face image, which includes pitch, yaw and roll).

[0119] Meanwhile, the coordinates of key facial points for n individuals may include, but are not limited to, the coordinates of the left corner of the eye, the right corner of the eye, the tip of the nose, the left corner of the mouth, the right corner of the mouth, and the jaw. Of course, the setting of these key points can be preset according to the tolerance for recognition accuracy, and is not limited to including only the key points listed above.

[0120] After obtaining the current pose angle of the face in the image to be identified, it can be compared with the observation angle threshold range for judgment, as shown in steps S6 and S7.

[0121] S6. Determine whether the current attitude angle is within the observation angle threshold range.

[0122] S7. If so, it is determined that the person corresponding to the face is paying attention to the target object.

[0123] In practice, when the pitch angle of a face is within the pitch angle threshold range and the yaw angle is within the yaw angle threshold range, it can be determined that the person corresponding to the face is paying attention to the target object.

[0124] Therefore, through the method for determining whether a person is paying attention to a target object as described in the aforementioned steps S1 to S7, the present invention can calculate the observation angle threshold range corresponding to the face in each image. Thus, it can accurately determine whether a face at any spatial location is paying attention to a target object, improving the accuracy of the judgment and making it suitable for large-scale promotion and application.

[0125] like Figure 7 As shown, the second aspect of this embodiment provides a hardware device for implementing the method for determining whether a person is paying attention to a target object as described in the first aspect of the embodiment, comprising:

[0126] An acquisition unit is used to acquire an image to be identified, wherein the image to be identified contains at least one human face.

[0127] The parameter calculation unit is used to obtain the distance between the face in the image to be identified and the camera, and the coordinates of the center point of the face in the image to be identified, based on the image to be identified, wherein the camera is mounted on the target object, and the center point of the face is the center point of the outline of the face in the image to be identified.

[0128] The face position calculation unit is used to obtain the actual spatial coordinates of the face relative to the camera based on the distance between the face in the image to be identified and the camera, as well as the coordinates of the center point of the face.

[0129] The observation angle calculation unit is used to obtain the observation angle threshold range based on the actual spatial coordinates of the face relative to the camera, wherein the observation angle threshold range is used to characterize the range of pose angles in which the face can focus on the target object.

[0130] The pose angle calculation unit is used to obtain the current pose angle of the face based on the image to be recognized.

[0131] The judgment unit is used to determine whether the current attitude angle is within the observation angle threshold range, and if the determination is yes, it determines that the person corresponding to the face is paying attention to the target object.

[0132] The working process, working details and technical effects of the hardware device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0133] like Figure 8 As shown, the third aspect of this embodiment provides another device for determining whether a person is paying attention to a target object. Taking a computer main device as an example, the device includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the method for determining whether a person is paying attention to a target object as described in the first aspect of the embodiment.

[0134] Specifically, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array).

[0135] At the same time, a processor can also include a main processor and a coprocessor. The main processor is the processor used to process data in the wake-up state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RSIC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0136] The working process, working details and technical effects of the computer main device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0137] The fourth aspect of this embodiment provides a storage medium that stores instructions for a method of determining whether a person is paying attention to a target object as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when the instructions are run on a computer, execute the method of determining whether a person is paying attention to a target object as described in the first aspect.

[0138] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0139] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0140] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method for determining whether a person is paying attention to a target object as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0141] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of determining whether a person is paying attention to an object, characterized by, The method comprises the following steps: acquiring a to-be-identified image, wherein the to-be-identified image contains at least one face; obtaining the distance between the face in the to-be-identified image and a camera and the coordinates of the center point of the face in the to-be-identified image according to the to-be-identified image, wherein the camera is installed on a target object and the center point of the face is the center point of the contour frame of the face in the to-be-identified image; obtaining the actual spatial coordinates of the face relative to the camera according to the distance between the face in the to-be-identified image and the camera and the coordinates of the center point of the face; obtaining an observation angle threshold interval according to the actual spatial coordinates of the face relative to the camera, wherein the observation angle threshold interval is used to represent the range of the attitude angle of the target object that can be observed by the face; obtaining the current attitude angle of the face according to the to-be-identified image; judging whether the current attitude angle is within the observation angle threshold interval; if yes, determining that the person corresponding to the face is paying attention to the target object; obtaining the actual spatial coordinates of the face relative to the camera according to the distance between the face in the to-be-identified image and the camera and the coordinates of the center point of the face, comprising: obtaining the vertical distance between the face in the to-be-identified image and the camera in the vertical direction and the vertical distance between the face and the camera in the horizontal direction, respectively denoted as the first distance and the second distance; obtaining the actual offset distance of the face in the vertical direction relative to the camera according to the distance between the face and the camera, the first distance, the second distance and the coordinates of the center point of the face; obtaining the actual offset distance of the face in the horizontal direction relative to the camera according to the actual offset distance of the face in the vertical direction relative to the camera, the first distance, the second distance and the coordinates of the center point of the face; obtaining the horizontal distance between the face and the camera according to the actual offset distance of the face in the vertical direction relative to the camera, the first distance and the coordinates of the center point of the face; obtaining the actual spatial coordinates according to the actual offset distance of the face in the vertical direction relative to the camera, the actual offset distance of the face in the horizontal direction relative to the camera and the horizontal distance; obtaining the actual offset distance of the face in the vertical direction relative to the camera according to the distance between the face and the camera, the first distance, the second distance and the coordinates of the center point of the face, comprising: the actual offset distance h0 of the face in the vertical direction relative to the camera is calculated by the following formula: In the above formula, wherein a represents an offset angle of the face in the vertical direction relative to the camera, β represents an offset angle of the face in the horizontal direction relative to the camera, h represents an abscissa in the coordinate of the center point of the face, d1 represents the first distance, d2 represents the second distance, and s represents the distance between the face and the camera.

2. The method of claim 1, wherein, obtaining the actual offset distance of the face in the horizontal direction relative to the camera according to the actual offset distance of the face in the vertical direction relative to the camera, the first distance, the second distance and the coordinates of the center point of the face, comprising: the actual offset distance w0 of the face in the horizontal direction relative to the camera is calculated by the following formula; In the above formula, wherein h0 represents an actual offset distance of the face in the vertical direction relative to the camera, a represents an offset angle of the face in the vertical direction relative to the camera, β represents an offset angle of the face in the horizontal direction relative to the camera, h represents a horizontal coordinate in the coordinate of the face center point, d1 represents the first distance, w represents a vertical coordinate in the coordinate of the face center point, d2 represents the second distance, and s represents the distance between the face and the camera.

3. The method of claim 1, wherein, According to the actual offset distance of the face in the vertical direction relative to the camera, the first distance, and the coordinates of the face center point, a horizontal distance between the face and the camera is obtained, including: The horizontal distance d0 is calculated by the following formula: In the above formula, wherein h0 represents an actual offset distance of the face in the vertical direction relative to the camera, a represents an offset angle of the face in the vertical direction relative to the camera, h represents the horizontal coordinate in the coordinate of the center point of the face, and d1 represents the first distance.

4. The method of claim 1, wherein, A vertical distance between the face in the to-be-identified image and the camera in the vertical direction is obtained, including: A wide-angle parameter of the camera and a maximum shooting distance in the vertical direction are obtained; According to the wide-angle parameter of the camera and the maximum shooting distance in the vertical direction, the vertical distance between the face in the to-be-identified image and the camera in the vertical direction is obtained.

5. The method of claim 1, wherein, According to the actual spatial coordinates of the face relative to the camera, an observation angle threshold interval is obtained, including: The height and length of the target object are obtained; According to the actual spatial coordinates, the height of the target object, and the length of the target object, the observation angle threshold interval is calculated.

6. The method of claim 1, wherein, According to the to-be-identified image, a distance between the face in the to-be-identified image and the camera is obtained, including: The interpupillary distance of the face in the to-be-identified image is obtained; A trained interpupillary distance fitting regression model is obtained, wherein the trained interpupillary distance fitting regression model is used to represent a functional relationship between the interpupillary distance of the face in any image and the distance from the face to the camera; According to the trained interpupillary distance fitting regression model and the interpupillary distance of the face in the to-be-identified image, the distance between the face in the to-be-identified image and the camera is obtained.

7. A device for determining whether a person is paying attention to a target object, characterized in that, Including: An acquisition unit is configured to acquire a to-be-identified image, wherein the to-be-identified image contains at least one face; A parameter calculation unit is configured to obtain, according to the to-be-identified image, a distance between the face in the to-be-identified image and a camera and coordinates of a face center point in the to-be-identified image, wherein the camera is installed on a target object, and the face center point is a center point of an outline frame of the face in the to-be-identified image; A face position calculation unit is configured to obtain, according to the distance between the face in the to-be-identified image and the camera and the coordinates of the face center point, actual spatial coordinates of the face relative to the camera; An observation angle calculation unit is configured to obtain, according to the actual spatial coordinates of the face relative to the camera, an observation angle threshold interval, wherein the observation angle threshold interval is used to represent a range of attitude angles of the target object that can be paid attention to by the face; An attitude angle calculation unit is configured to obtain, according to the to-be-identified image, a current attitude angle of the face; A judgment unit is configured to judge whether the current attitude angle is within the observation angle threshold interval, and determine that a person corresponding to the face is paying attention to the target object when the judgment is yes. According to the distance between the face in the to-be-identified image and the camera and the coordinates of the face center point, actual spatial coordinates of the face relative to the camera are obtained, including: A vertical distance between the face in the to-be-identified image and the camera in the vertical direction and a vertical distance between the face in the to-be-identified image and the camera in the horizontal direction are obtained, respectively denoted as a first distance and a second distance; According to the distance between the face and the camera, the first distance, the second distance, and the coordinate of the face center point, an actual offset distance of the face in the vertical direction relative to the camera is obtained; According to the actual offset distance of the face in the vertical direction relative to the camera, the first distance, the second distance, and the coordinate of the face center point, an actual offset distance of the face in the horizontal direction relative to the camera is obtained; According to the actual offset distance of the face in the vertical direction relative to the camera, the first distance, and the coordinate of the face center point, a horizontal distance between the face and the camera is obtained; According to the actual offset distance of the face in the vertical direction relative to the camera, the actual offset distance of the face in the horizontal direction relative to the camera, and the horizontal distance, the actual space coordinate is obtained; According to the distance between the face and the camera, the first distance, the second distance, and the coordinate of the face center point, an actual offset distance of the face in the vertical direction relative to the camera is obtained, comprising: The actual offset distance h0 of the face in the vertical direction relative to the camera is calculated by using the following formula: In the above formula, wherein a represents an offset angle of the face in the vertical direction relative to the camera, β represents an offset angle of the face in the horizontal direction relative to the camera, h represents an abscissa in the coordinate of the center point of the face, d1 represents the first distance, d2 represents the second distance, and s represents the distance between the face and the camera.

8. A computer host device, characterized by comprising: The memory is used for storing a computer program, the transceiver is used for transmitting and receiving messages, and the processor is used for reading the computer program and executing the method for judging whether a person pays attention to a target object according to any one of claims 1-6.

Images