User attention information acquisition method, storage medium, chip and electronic equipment
By acquiring and analyzing the video information of the user in the area to be detected, determining the user's position and orientation, and combining this information to determine the target attention focus, the problem of low accuracy in the acquisition of attention information in the prior art is solved, and higher accuracy and fewer misjudgments are achieved.
Patent Information
- Application Number
- CN202311301554.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-09
- Publication Date
- 2025-04-11
Smart Images
Figure CN119810700A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of education and training, and specifically to a method for obtaining user attention information, a storage medium, a chip and an electronic device. Background Art
[0002] In the classroom, it is necessary to know where students are paying attention. By determining students' attention in class, it helps teachers improve their grasp of the class, adjust the lecture rhythm in time, and create a better teaching atmosphere. It also provides education managers with a dimension and means to evaluate teaching quality.
[0003] Good attention is often achieved during class. When the teacher is lecturing, students should focus on the teacher or the blackboard. If students doze off or look out the window absentmindedly, we can determine that their attention is distracted. In existing technical solutions, a camera is often used to capture the image of a student at a certain moment, and then the position of the head is obtained through a head detection method, and then the angle direction of the head is calculated or regressed; or the position of the face and its key points are obtained through face detection, and the angle of the face is calculated through the key points, or the angle of the face is directly regressed to determine the direction of the head. The above methods will eventually calculate the angle and direction of the head, and by setting a certain head deflection range, it is determined whether the student's attention is distracted in the current state. For example, in current technology, a camera is often installed in front of the classroom, and then the angle of the head is detected. If the head angle deflects too much (such as turning to the window), it is determined to be distracted. The above method simply determines whether an individual is distracted by the angle of his or her head (face), which is very limited. For example, when students in class bow their heads to study or bow their heads to sleep, the pitch angles of these two situations are relatively large. If the above method is used to judge, it will be easy to determine that the student is distracted, resulting in a decrease in the accuracy of obtaining attention information from the students. Summary of the invention
[0004] One purpose of the embodiments of the present application is to provide a method for obtaining user attention information, a storage medium, a chip and an electronic device, aiming to improve the existing technical problem of low accuracy in obtaining user attention information.
[0005] In a first aspect, an embodiment of the present application provides a method for obtaining user attention information, the method for obtaining user attention information comprising:
[0006] Acquire the first video in the area to be detected;
[0007] If the first video in the first detection time period contains audio information of the first user, obtaining a first video image of the area to be detected in the first detection time period from the first video;
[0008] Determine the position information of k users to be detected and the orientation information of k users to be detected in the area to be detected by using the first video image, where k is an integer;
[0009] Determine the target attention focus point position information according to the orientation information of the k users to be detected;
[0010] The attention information of the k users to be detected is determined based on the position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the target attention focus point.
[0011] In a possible implementation, determining the target attention focus position information according to the orientation information of the k users to be detected includes:
[0012] Determine n reference attention focus point position information according to the orientation information of the k users to be detected;
[0013] Obtaining the number of users to be detected corresponding to the n reference attention focus position information respectively, to obtain n first numbers;
[0014] The reference attention focus point position information corresponding to the maximum value among the n first quantities is determined as the target attention focus point position information.
[0015] In a possible implementation, determining n reference attention focus point position information according to the orientation information of the k users to be detected includes:
[0016] Determine n head reference attention focus point position information according to the head orientation information in the orientation information of the k users to be detected;
[0017] Determine n facial reference attention focus point position information according to facial orientation information in the orientation information of the k users to be detected;
[0018] Determine n reference eye attention focus point position information according to head orientation information in the orientation information of the k users to be detected;
[0019] Obtaining a first reconciliation factor corresponding to a head reference attention focus point, obtaining a second reconciliation factor corresponding to a face reference attention focus point, and obtaining a third reconciliation factor corresponding to an eye reference attention focus point;
[0020] According to the first harmonization factor, the second harmonization factor and the third harmonization factor, the position information of the n head reference attention focus point position information, the face reference attention focus point position information and the n eye reference attention focus point position information are harmonized to obtain the corresponding n reference attention focus point position information.
[0021] In a possible implementation, the step of performing position information harmonization on the n pieces of head reference attention focus point position information, the face reference attention focus point position information, and the n pieces of eye reference attention focus point position information according to the first harmonization factor, the second harmonization factor, and the third harmonization factor to obtain corresponding n pieces of reference attention focus point position information includes:
[0022] According to the first reconciliation factor, the second reconciliation factor and the third reconciliation factor, the n pieces of the head reference attention focus point position information, the face reference attention focus point position information and the n pieces of the eye reference attention focus point position information are reconciled to obtain m pieces of first attention focus point position information, where m is an integer greater than or equal to n;
[0023] Obtain the types of attention focus points corresponding to the position information of the m first attention focus points respectively;
[0024] The first attention focus point position information whose attention focus point type is a preset attention focus type is determined as the reference attention focus point position information to obtain n reference attention focus point position information.
[0025] In a possible implementation, determining the attention information of the k users to be detected according to the position information of the k users to be detected, the orientation information of the k users to be detected, and the position information of the target attention focus point includes:
[0026] Determine the attention directions corresponding to the k users to be detected respectively according to the position information of the k users to be detected and the position information of the target attention focus point;
[0027] Obtaining angles between directions indicated by direction information of k users to be detected and corresponding attention directions, to obtain k first angles;
[0028] The attention information of the corresponding user to be detected is determined according to the k first angles.
[0029] In a possible implementation, the step of obtaining angles between k orientation information of the to-be-detected users and corresponding attention orientations to obtain k first angles includes:
[0030] Obtaining the angle between the head orientation and the attention orientation in the orientation information of k users to be detected, so as to obtain k first reference angles;
[0031] Obtaining angles between facial orientations in orientation information of k users to be detected and the attention orientation to obtain k second reference angles;
[0032] Obtaining the angle between the eye orientation and the attention orientation in the orientation information of k users to be detected, so as to obtain k third reference angles;
[0033] The maximum angle among the first reference angle, the second reference angle and the third reference angle respectively corresponding to the k users to be detected is determined as the first angle corresponding to the k users to be detected.
[0034] In a possible implementation, the method further includes:
[0035] Obtaining position change information of the target attention focus point position information within the first detection time period;
[0036] If the position change information is preset position change information, it is determined that the same attention focusing event exists within the first detection time period.
[0037] In a second aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes a method for acquiring user attention information as described in any one of the first aspects.
[0038] In a third aspect, an embodiment of the present application provides a chip, including:
[0039] at least one processor; and,
[0040] a memory communicatively connected to the at least one processor; wherein,
[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user attention information acquisition method as described in any one of the first aspects.
[0042] In a fourth aspect, an embodiment of the present application provides an electronic device, including:
[0043] Audio and video collector;
[0044] at least one processor, connected in communication with the audio and video collector; and
[0045] a memory communicatively connected to the at least one processor; wherein,
[0046] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user attention information acquisition method as described in any one of the first aspects.
[0047] In the method for obtaining user attention information provided in the embodiment of the present application, by obtaining a first video in the area to be detected, if there is audio information of the first user in the first video within the first detection time period, then the first video image of the area to be detected within the first detection time period is obtained from the first video, and the position information of k users to be detected and the orientation information of the k users to be detected in the area to be detected are determined through the first video image, and the target attention focus point position information is determined according to the orientation information of the k users to be detected, and the attention information of the k users to be detected is determined according to the position information of the k users to be detected, the orientation information of the k users to be detected and the target attention focus point position information, so that the attention information of the k users to be detected can be determined through the target attention focus point position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the k users to be detected, so that the attention information of the k users to be detected can be determined through the target attention focus point position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the k users to be detected, so that the attention information of the users to be detected is determined in combination with the group attention focus of the k users to be detected, thereby improving the accuracy of determining the attention information of the users to be detected. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0049] Figure 1 A schematic diagram of the structure of an information acquisition system provided in an embodiment of the present application;
[0050] Figure 2A A schematic diagram of a process for obtaining user attention information provided in an embodiment of the present application;
[0051] Figure 2B A schematic diagram of an attention space provided in an embodiment of the present application;
[0052] Figure 3 A schematic diagram of the circuit structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0054] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in this application do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.
[0055] In order to better understand a method for obtaining user attention information provided by an embodiment of the present application, the following first briefly introduces the existing method for obtaining user attention information. In the existing scheme, when obtaining the user's attention information, for example, when obtaining the attention information of students in the classroom, it is usually necessary to capture the image of the student at a certain moment through a camera, and then obtain the position of the head through a head detection method, and then calculate or regress the angle direction of the head, and finally determine the student's attention information based on the angle direction of the head and the pre-set head deflection range. The attention information may include whether the student is concentrating, for example, the student's attention is focused on the teacher's instructions, and the student's attention is distracted (for example, not focused on the content of the teacher's instructions). The content of the teacher's instructions can be, for example, what the teacher writes on the blackboard, etc., or for example, the teacher can instruct the student to study on his own, then the content of the teacher's instructions can be the student reading a book, etc. However, determining the student's attention information only by the deflection angle of the student's head is prone to errors. For example, when a teacher instructs a student to study on his own, the student's head deflection angle should usually be closely related to the desk. For example, the head should face the desk after the deflection. However, since students may discuss problems when studying on their own, the student's head deflection angle may be weakly related to the desk. For example, a student turns his head to discuss with a student at the next table. At this time, the above-mentioned existing scheme will determine that the student's attention information is distracted, which will lead to misjudgment, resulting in reduced accuracy in the final determination of the attention information.
[0056] This application aims to solve the above problems and proposes a method for obtaining user attention information, which determines the attention information of k users to be detected (for example, students, etc.) through the target attention focus position information of k users to be detected (for example, students, etc.), the orientation information of k users to be detected and the position information of k users to be detected, thereby combining the group attention focus of k users to be detected to determine the attention information of the users to be detected, thereby improving the accuracy of determining the attention information of the users to be detected. The above scheme is explained by taking students as an example, and of course it can also be applied to other scenarios, such as conference scenarios, etc.
[0057] The present application embodiment provides an information acquisition system, such as Figure 1 As shown, the information acquisition system 100 includes an audio and video acquisition device 1 and a processing device 2. The audio and video acquisition device 1 and the processing device 2 can be integrated into one device or can be separate devices. If they are separate devices, the audio and video acquisition device 1 and the processing device 2 can be connected via WIFI, or can be connected wirelessly via other frequency bands such as 2.4GHz, or can be other wireless connection methods, such as connection based on other IoT wireless protocols, etc., and of course can also be wired connection methods. The audio and video acquisition device can be a video acquisition camera, a device integrating audio acquisition and video acquisition, etc., and the processing device can be a computer, server, tablet computer, mobile computer, etc., and of course can also be other electronic devices.
[0058] As another aspect of the present application, the present application provides a method for obtaining user attention information, which is applied to an electronic device. Figure 2A , user attention information acquisition includes:
[0059] S201: Acquire a first video in the area to be detected.
[0060] Among them, the area to be detected may be an area where user attention information needs to be obtained, for example, in a classroom area where lectures are given, in a conference area where meetings are held, etc. There are multiple users to be detected in the area to be detected, and it is necessary to obtain the attention information of the multiple users to be detected. The attention information may include concentration or distraction, etc. Concentration can be understood as the range direction of the attention of the group in the area to be detected where the concentration point of the user to be detected is within the area to be detected, and distraction can be understood as the range direction of the attention of the group in the area to be detected where the concentration point of the user to be detected is not within the area to be detected, etc.
[0061] The first video may include video images and audio in the area to be detected. Specifically, the first video may be acquired through an audio and video acquisition device.
[0062] Alternatively, the video image in the first video may be acquired through a camera, the audio in the to-be-detected area may be acquired through a microphone, and then the video image and the audio in the first video may be synchronously fused to obtain the first video. Of course, it is also possible to directly use the audio and video image for subsequent recognition processing without synchronously fusion processing of the video image and the audio in the first video.
[0063] S202: If the first video within the first detection time period contains audio information of the first user, obtain a first video image of the area to be detected within the first detection time period from the first video.
[0064] The first user may be a pre-set user, for example, a teacher, a conference speaker, etc. The first detection time period may be a pre-set time period for acquiring user attention information, for example, a certain time period after the teacher starts class, specifically, from the tenth minute to the twelfth minute after the teacher starts class, etc. The first detection time period may also be a randomly selected time period, which is not specifically limited here.
[0065] The audio in the first video can be identified to determine whether there is audio of the first user in the first detection time period. For example, here, the first user is a teacher. The teacher's voice detection can use a deep learning model to train a large number of samples of the teacher's voice recordings during class, so that the model can learn the teacher's voice characteristics. Other technical means can also be used, such as eliminating environmental noise, using a speech recognition method, and judging whether the current speaker is a teacher based on the recognition of the current speaker's speech content.
[0066] S203. Determine, through the first video image, position information of k users to be detected and orientation information of k users to be detected in the area to be detected, where k is an integer.
[0067] There may be multiple first video images of the area to be detected within the first detection time period, and each first video image may be processed to obtain the orientation information of the user to be detected corresponding to each first video image.
[0068] The position information of the user to be detected may be obtained through a preset position information acquisition model, and the orientation information of the user to be detected may be obtained through a preset orientation information acquisition model. The orientation information may include head orientation information, face orientation information and eye orientation information of the user to be detected.
[0069] For example, a 3D reconstruction method can be used to first reconstruct the spatial information of the head and face of the user to be detected, and then use a multi-view composed of a monocular or binocular or more cameras, or a depth camera, or a laser radar and other devices to capture 3D structural information to reconstruct the head of the user to be detected. After reconstruction, the head orientation information, facial orientation information and eye orientation information of the user to be detected are obtained according to the reconstructed image. A deep learning method can also be used to obtain angle information by estimating key points. Of course, it can also be that for the image of the head (face) position, a residual network is used to regress the key points of the head (face), map the obtained key points to the Euclidean space, calculate the Euler angle, and finally obtain the head (face) direction angle, thereby obtaining the head orientation information and facial orientation information. Using the residual network, the eyeballs to be detected are classified, the deviation of the line of sight is determined, and the direction angle of the eye line of sight of the user to be detected is finally obtained in combination with the direction angle of the head (face), thereby obtaining the eye orientation information of the user to be detected.
[0070] S204: Determine the target attention focus position information according to the orientation information of the k users to be detected.
[0071] When acquiring the target attention focus point position information, clustering can be performed based on the orientation information of the k users to be detected to obtain n reference attention focus point position information. The reference attention focus point corresponding to each reference attention focus point position information is obtained by clustering the orientation information of multiple users to be detected. Therefore, the reference attention focus point position information with the largest number of reference users to be detected can be determined as the target attention focus point position information. In this way, the reference attention focus point corresponding to the largest number of users to be detected can be determined as the target attention focus point, which can improve the accuracy of subsequent processing.
[0072] Of course, when the n reference attention focus point position information is determined, the reference attention focus point corresponding to the reference attention focus point position information can also be subjected to a legality judgment to judge whether it is a legal attention focus point, thereby obtaining n reference attention focus point position information. For example, the reference attention focus point position information can be used to judge whether the reference attention focus point is in a preset position area, specifically, the attention focus points in different position areas have their corresponding attention focus point types, and then the reference attention focus point type is judged by judging whether it is a preset type. If it is a preset type, it is a legal attention focus point. If it is not a preset type, it is an illegal attention focus point, and it needs to be deleted at this time. By judging the legality of the reference attention focus point position information, the accuracy of subsequent processing can be improved, thereby improving the accuracy of obtaining the attention information of the user to be detected.
[0073] S205. Determine the attention information of the k users to be detected based on the position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the target attention focus point.
[0074] Among them, the attention directions corresponding to the k users to be detected can be determined according to the position information of the k users to be detected and the position information of the target attention focus point, and the attention information of the user to be detected can be determined according to the angle between the attention direction and the direction information of the user to be detected. For example, if the angle is less than a preset angle value, the attention information can be focused attention, and if the angle is greater than or equal to the preset angle value, the attention information can be distracted attention.
[0075] In this example, by obtaining a first video in the area to be detected, if there is audio information of the first user in the first video within a first detection time period, then the first video image of the area to be detected within the first detection time period is obtained from the first video, and the position information of k users to be detected and the orientation information of the k users to be detected in the area to be detected are determined through the first video image, and the target attention focus position information is determined according to the orientation information of the k users to be detected, and the attention information of the k users to be detected is determined according to the position information of the k users to be detected, the orientation information of the k users to be detected and the target attention focus position information, so that the attention information of the k users to be detected can be determined through the target attention focus position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the k users to be detected, so that the attention information of the k users to be detected can be determined by the target attention focus position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the k users to be detected, so that the attention information of the users to be detected is determined in combination with the group attention focus of the k users to be detected, thereby improving the accuracy of determining the attention information of the users to be detected.
[0076] In a possible implementation, a method for determining the position information of a target attention focus point according to the orientation information of k users to be detected includes:
[0077] A1. Determine n reference attention focus point position information according to the orientation information of k users to be detected;
[0078] A2. Obtain the number of users to be detected corresponding to the n reference attention focus position information, and obtain n first numbers;
[0079] A3. Determine the reference attention focus point position information corresponding to the maximum value among the n first quantities as the target attention focus point position information.
[0080] Among them, n reference attention focus point position information can be determined based on the facial orientation information, head orientation information, and eye orientation information in the orientation information.
[0081] The reference attention focus point corresponding to each reference attention focus point position information is determined by the orientation information of multiple users to be detected. Therefore, the multiple users to be detected can be determined as the users to be detected corresponding to the reference attention focus point position information, so that the number of users to be detected corresponding to the reference attention focus point position information can be obtained, and the first number can be obtained.
[0082] The reference attention focus point position information corresponding to the maximum value of the first quantity can be determined as the target attention focus point position information. For example, if the first quantity is 9, 10, 20, and 8, then the 20 corresponding reference attention focus point position information can be determined as the target attention focus point position information. The target attention focus point position information can be represented by position coordinates.
[0083] In this example, by determining the reference attention focus point position information corresponding to the maximum value among the n first quantities as the target attention focus point position information, the accuracy of determining the target attention focus point position information can be improved.
[0084] In a possible implementation, a method for determining n reference attention focus position information based on orientation information of k users to be detected includes:
[0085] B1. determining n head reference attention focus point position information according to the head orientation information in the orientation information of the k users to be detected;
[0086] B2. determining n facial reference attention focus point position information according to the facial orientation information of the k users to be detected;
[0087] B3, determining n reference eye attention focus point position information according to the head orientation information in the orientation information of the k users to be detected;
[0088] B4, obtaining a first reconciliation factor corresponding to a head reference attention focus point, obtaining a second reconciliation factor corresponding to a face reference attention focus point, and obtaining a third reconciliation factor corresponding to an eye reference attention focus point;
[0089] B5. Harmonize the n pieces of head reference attention focus point position information, face reference attention focus point position information and n pieces of eye reference attention focus point position information according to the first harmonization factor, the second harmonization factor and the third harmonization factor to obtain corresponding n pieces of reference attention focus point position information.
[0090] The cross product of each two pieces of head orientation information can be obtained to obtain a cross product set, and the cross product set is clustered to obtain a clustering result of the head orientation information (head reference attention focus point position information). The method of clustering the cross product set can be to use a general clustering method to obtain the head reference attention focus point position information.
[0091] The head reference attention focus position information can be determined by the method shown in the following formula:
[0092]
[0093] Among them, f h (x, y, z) is the position information (coordinates) of the head reference attention focus point, cluster is the preset attention algorithm, is the head orientation information of the i-th user to be detected, is the head orientation information of the jth user to be detected, M h is the number of users to be detected, M h is equal to n, and × is a cross product operation. The head orientation information can be represented by the head direction angle, for example, Represents the polar distance, polar angle, and azimuth of the head in 3D polar coordinates.
[0094] When determining the facial reference attention focus point position information and the eye reference attention focus point position information, the method for determining the head reference attention focus point position information can be referred to, which will not be repeated here.
[0095] Specifically, the facial reference attention focus position information can be expressed by the following formula:
[0096]
[0097] Among them, f f (x, y, z) is the facial reference attention focus point location information (coordinates), cluster is the preset attention algorithm, is the facial orientation information of the i-th user to be detected, is the facial orientation information of the jth user to be detected, M f is the number of users to be detected, M f Equal to n, × is the cross product operation.
[0098] The facial reference attention focus position information can be expressed by the following formula:
[0099]
[0100] Among them, f e(x, y, z) is the reference eye focus point location information (coordinates), cluster is the preset attention algorithm, is the eye orientation information of the i-th user to be detected, is the eye orientation information of the jth user to be detected, M e is the number of users to be detected, M e Equal to n, × is the cross product operation.
[0101] The method for obtaining the first harmonization factor corresponding to the head reference attention focus point may be to determine the preset harmonization factor corresponding to the head reference attention focus point as the first harmonization factor; the method for obtaining the second harmonization factor corresponding to the facial reference attention focus point may be to determine the preset harmonization factor corresponding to the facial reference attention focus point as the second harmonization factor; and the method for obtaining the third harmonization factor corresponding to the eye reference attention focus point may be to determine the preset harmonization factor corresponding to the eye reference attention focus point as the third harmonization factor.
[0102] The reference attention focus position information is determined according to the method shown in the following formula:
[0103] f(x, y, z) = af h (x,y,z)+βf f (x,y,z)+γf e (x, y, z),
[0104] Among them, α is the first harmonic factor, β is the second harmonic factor, γ is the third harmonic factor, and f h (x, y, z) is the position information of the head reference attention focus point, f f (x, y, z) is the location information of the facial reference attention focus point, f e (x, y, z) is the reference eye focus point position information (coordinates), so that the reference eye focus point position information can be determined.
[0105] In this example, the reference attention focus point position information of the head, the reference attention focus point position information of the face, and the reference attention focus point position information of the eyes are respectively determined, and then harmonized through corresponding harmonization factors to obtain the reference attention focus point position information, so as to improve the accuracy of determining the reference attention focus point position information.
[0106] In one possible implementation, a method for performing position information harmonization on n pieces of head reference attention focus point position information, face reference attention focus point position information, and n pieces of eye reference attention focus point position information according to the first harmonization factor, the second harmonization factor, and the third harmonization factor to obtain corresponding n pieces of reference attention focus point position information includes:
[0107] C1, reconciling the n pieces of the head reference attention focus point position information, the face reference attention focus point position information and the n pieces of the eye reference attention focus point position information according to the first reconciliation factor, the second reconciliation factor and the third reconciliation factor to obtain m pieces of first attention focus point position information, where m is an integer greater than or equal to n;
[0108] C2, obtaining the types of attention focus points corresponding to the position information of the m first attention focus points;
[0109] C3. Determine the first attention focus point position information whose attention focus point type is a preset attention focus type as the reference attention focus point position information to obtain n reference attention focus point position information.
[0110] Different first attention focus point position information has its corresponding attention focus point type, for example, Figure 2B As shown, here, the user to be detected is a student, and the area to be detected is a classroom. If the position indicated by the first attention focus point position information is in the forward attention ROI space, then its attention focus point type can be a forward type; if the position indicated by the first attention focus point position information is in the lateral attention ROI space, then its attention focus point type can be a lateral type; if the position indicated by the first attention focus point position information is in the bottom attention ROI space, then its attention focus point type can be a bottom type. If the position indicated by the first attention focus point position information is not in the above three spaces, then its attention focus point type is an illegal attention focus point. The preset attention focus type can be a forward type, a lateral type, and a bottom type. Among them, the forward attention ROI refers to the individual attention F (x, y, z) projected to the cubic space formed by the dotted line. The lateral attention ROI refers to the individual attention F (x, y, z) projected to the side wall of the classroom, but does not belong to the remaining space of the forward attention ROI space. The bottom attention ROI refers to the remaining space where the individual attention F(x, y, z) is projected onto the classroom floor but does not belong to the forward attention ROI space.
[0111] In this example, by screening the legitimacy of m first attention focus point position information, n reference attention focus point position information is obtained, which can improve the accuracy of subsequent determination of attention information.
[0112] In a possible implementation, a method for determining the attention information of k users to be detected based on the position information of the k users to be detected, the orientation information of the k users to be detected, and the position information of the target attention focus point includes:
[0113] D1, determining the attention directions corresponding to the k users to be detected respectively according to the position information of the k users to be detected and the position information of the target attention focus point;
[0114] D2. Obtaining angles between directions indicated by direction information of k users to be detected and corresponding attention directions, to obtain k first angles;
[0115] D3. Determine the attention information of the corresponding user to be detected based on the k first angles.
[0116] The position information of the user to be detected and the position information of the target attention focus point can be used to determine a first vector, and the direction of the first vector is from the position indicated by the position information of the user to be detected to the position indicated by the position information of the target attention focus point. Thus, the direction of the first vector can be determined as the attention direction corresponding to the user to be detected.
[0117] Since the orientation information includes head orientation information, facial orientation information and eye orientation information, the angles between the direction indicated by the head orientation information, the direction indicated by the facial orientation information and the direction indicated by the eye orientation information and the attention direction can be obtained respectively, and the maximum angle is determined as the first angle.
[0118] The first angle may be compared with a preset angle value. If the first angle is smaller than the preset angle value, the attention information may be focused attention. If the first angle is greater than or equal to the preset angle value, the attention information may be distracted attention.
[0119] In this example, the attention information is determined by the angle between the attention direction and the direction indicated by the direction information of the user to be detected and a preset angle value, which can improve the accuracy of determining the attention information.
[0120] In a possible implementation, a method for obtaining angles between k orientation information of the to-be-detected users and corresponding attention orientations to obtain k first angles includes:
[0121] E1. Obtaining the angle between the head orientation and the attention orientation in the orientation information of k users to be detected, so as to obtain k first reference angles;
[0122] E2. Obtaining the angle between the facial orientation in the orientation information of k users to be detected and the attention orientation to obtain k second reference angles;
[0123] E3, obtaining the angle between the eye orientation and the attention orientation in the orientation information of k users to be detected, so as to obtain k third reference angles;
[0124] E4. Determine the maximum angle among the first reference angle, the second reference angle and the third reference angle respectively corresponding to the k users to be detected as the first angle corresponding to the k users to be detected.
[0125] The first reference angle, the second reference angle and the third reference angle may be acquired through a general direction angle determination method.
[0126] The maximum angle among the first reference angle, the second reference angle and the third reference angle is determined as the first angle corresponding to the user to be detected. At this time, it can be reflected that the user to be detected is most likely to be distracted. If the attention information obtained at this time is focused, then the attention of the user to be detected is very likely to be focused, thereby improving the accuracy of determining the attention information.
[0127] In a possible implementation, the attention focusing event in the first detection time period may also be identified, specifically:
[0128] F1. Obtaining position change information of the target attention focus point position information within the first detection time period;
[0129] F2. If the position change information is preset position change information, it is determined that the same attention focusing event exists within the first detection time period.
[0130] The method for obtaining the position change information of the target attention focus point position information within the first detection time period may be:
[0131] Obtain the target attention focus point position information at the first moment of the first detection time period, and obtain the target attention focus point position information at every t time intervals after the first moment, and determine the distance value as the position change information based on the distance value between the positions indicated by each target attention focus point position information.
[0132] The preset position change information may be a distance value less than a preset distance value. Specifically, it can be understood that, since the distance values between the positions indicated by each target attention focus point position information are less than the preset distance value in the first detection time period, it indicates that the group attention of the k users to be detected has not changed significantly, so that it can be determined that the same attention focus event exists in the first detection time period. If the position change information is not the preset position change information, it indicates that there are multiple attention focus events in the first detection time period, and it is necessary to re-acquire the target attention focus point position information to improve the accuracy of subsequent attention information acquisition.
[0133] It should be noted that, in each of the above-mentioned embodiments, there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present application, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.
[0134] See also Figure 3 , Figure 3 The following is a schematic diagram of the circuit structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 300 includes an audio and video collector 301, one or more processors 302, and a memory 303. The processor 302, the audio and video collector 301, and the memory 303 are communicatively connected.
[0135] The memory 303 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for establishing a Bluetooth connection in the embodiment of the present application. The processor 302 executes the functions of the user attention information acquisition method provided in the above method embodiment by running the non-volatile software programs, instructions and modules stored in the memory 303.
[0136] The memory 303 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 303 may optionally include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the processor 302 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0137] The program instructions / modules are stored in the memory 303, and when executed by the one or more processors 302, the method for obtaining user attention information in any of the above method embodiments is executed.
[0138] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example Figure 3 A processor 302 in the embodiment may enable the one or more processors to execute the method for obtaining user attention information in any of the above method embodiments.
[0139] An embodiment of the present application also provides a chip, which includes at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user attention information acquisition method in any of the above method embodiments.
[0140] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes any one of the user attention information acquisition methods described.
[0141] The above described device or equipment embodiments are merely illustrative, wherein the unit modules described as separate components may or may not be physically separated, and the components displayed as module units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network module units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.
[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for obtaining user attention information, characterized in that: The method for obtaining user attention information comprises: Acquire the first video in the area to be detected; If the first video in the first detection time period contains audio information of the first user, obtaining a first video image of the area to be detected in the first detection time period from the first video; Determine the position information of k users to be detected and the orientation information of k users to be detected in the area to be detected by using the first video image, where k is an integer; Determine the target attention focus point position information according to the orientation information of the k users to be detected; The attention information of the k users to be detected is determined based on the position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the target attention focus point.
2. The method for obtaining user attention information according to claim 1, characterized in that: The step of determining the target attention focus point position information according to the orientation information of the k users to be detected includes: Determine n reference attention focus point position information according to the orientation information of the k users to be detected; Obtaining the number of users to be detected corresponding to the n reference attention focus position information respectively, to obtain n first numbers; The reference attention focus point position information corresponding to the maximum value among the n first quantities is determined as the target attention focus point position information.
3. The method for obtaining user attention information according to claim 2, characterized in that: The step of determining n reference attention focus point position information according to the orientation information of the k users to be detected includes: Determine n head reference attention focus point position information according to the head orientation information in the orientation information of the k users to be detected; Determine n facial reference attention focus point position information according to facial orientation information in the orientation information of the k users to be detected; Determine n reference eye attention focus point position information according to head orientation information in the orientation information of the k users to be detected; Obtaining a first reconciliation factor corresponding to a head reference attention focus point, obtaining a second reconciliation factor corresponding to a face reference attention focus point, and obtaining a third reconciliation factor corresponding to an eye reference attention focus point; According to the first harmonization factor, the second harmonization factor and the third harmonization factor, the position information of the n head reference attention focus point position information, the face reference attention focus point position information and the n eye reference attention focus point position information are harmonized to obtain the corresponding n reference attention focus point position information.
4. The method for obtaining user attention information according to claim 3, characterized in that: The step of reconciling the n pieces of head reference attention focus point position information, the face reference attention focus point position information, and the n pieces of eye reference attention focus point position information according to the first reconciliation factor, the second reconciliation factor, and the third reconciliation factor to obtain corresponding n pieces of reference attention focus point position information includes: According to the first reconciliation factor, the second reconciliation factor and the third reconciliation factor, the n pieces of the head reference attention focus point position information, the face reference attention focus point position information and the n pieces of the eye reference attention focus point position information are reconciled to obtain m pieces of first attention focus point position information, where m is an integer greater than or equal to n; Obtain the types of attention focus points corresponding to the position information of the m first attention focus points respectively; The first attention focus point position information whose attention focus point type is a preset attention focus type is determined as the reference attention focus point position information to obtain n reference attention focus point position information.
5. The method for obtaining user attention information according to any one of claims 1 to 4, characterized in that: The step of determining the attention information of the k users to be detected according to the position information of the k users to be detected, the orientation information of the k users to be detected and the position information of the target attention focus point comprises: Determine the attention directions corresponding to the k users to be detected respectively according to the position information of the k users to be detected and the position information of the target attention focus point; Obtaining angles between directions indicated by direction information of k users to be detected and corresponding attention directions, to obtain k first angles; The attention information of the corresponding user to be detected is determined according to the k first angles.
6. The method for obtaining user attention information according to claim 5, characterized in that: The step of obtaining angles between k orientation information of the to-be-detected users and corresponding attention orientations to obtain k first angles includes: Obtaining the angle between the head orientation and the attention orientation in the orientation information of k users to be detected, so as to obtain k first reference angles; Obtaining angles between facial orientations in orientation information of k users to be detected and the attention orientation to obtain k second reference angles; Obtaining the angle between the eye orientation and the attention orientation in the orientation information of k users to be detected, so as to obtain k third reference angles; The maximum angle among the first reference angle, the second reference angle and the third reference angle respectively corresponding to the k users to be detected is determined as the first angle corresponding to the k users to be detected.
7. The method for obtaining user attention information according to claim 6, characterized in that: The method further comprises: Obtaining position change information of the target attention focus point position information within the first detection time period; If the position change information is preset position change information, it is determined that the same attention focusing event exists within the first detection time period.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the user attention information acquisition method as described in any one of claims 1 to 7.
9. A chip, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user attention information acquisition method as described in any one of claims 1 to 7.
10. An electronic device, characterized in that: include: Audio and video collector; At least one processor, communicatively connected to the audio and video collector; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user attention information acquisition method as described in any one of claims 1 to 7.