User portrait data determination method and electronic equipment

Abstract

The invention is suitable for the technical field of image processing, and provides a user portrait data determination method and electronic device.The method is applied to terminal equipment and comprises the steps that under the condition that an application runs, a scene image collected by the terminal equipment is obtained; the scene image comprises an image of a user capable of seeing a display picture of the terminal equipment, detecting a face posture of the user in the scene image, determining whether the user is in a concerned state according to the face posture of the user, and when it is determined that the user is in the concerned state, sending the scene image to the terminal equipment. And determining portrait data of the user according to the content of the application. The accuracy of the determined user portrait data can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method, device, electronic device and computer-readable storage medium for determining user portrait data. Background Art

[0002] User portrait refers to a model or tool that classifies and describes users based on their behaviors, interests, preferences, etc. User portraits usually include data on the user's basic information, interests, habits, consumption behaviors, etc. User portraits are widely used in personalized recommendations, user behavior analysis, and other fields.

[0003] With the development of data analysis technology, it is possible to collect behavioral data as user portrait data when users use products or browse content, and to construct corresponding user portraits through the user portrait data, so as to conduct data analysis, recommendations, etc. on the user. For example, when a user uses video software, a user portrait can be constructed based on behavioral data such as the type of videos the user browses, the time period in which the video is viewed, etc. However, since the content that a user browses is not necessarily the content that the user is interested in, the user portrait data determined by the above method is not accurate enough. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for determining user portrait data, which can improve the accuracy of determined user portrait data.

[0005] In a first aspect, an embodiment of the present application provides a method for determining user portrait data, which is applied to a terminal device, including:

[0006] When the application is running, obtaining a scene image captured by the terminal device; the scene image includes an image of a user who can see a display screen of the terminal device;

[0007] Detecting a user's facial posture in the scene image;

[0008] Determining whether the user is in a focused state according to the user's facial posture;

[0009] When it is determined that the user is in a focused state, the portrait data of the user is determined according to the content of the application.

[0010] In a second aspect, an embodiment of the present application provides a device for determining user portrait data, the device being applied to a terminal device, comprising:

[0011] A scene image acquisition module, used to acquire a scene image collected by the terminal device when the application is running; the scene image includes an image of a user who can see the display screen of the terminal device;

[0012] A face gesture detection module, used to detect the face gesture of the user in the scene image;

[0013] A state judgment module, used to determine whether the user is in a focused state according to the user's facial posture;

[0014] The portrait data determination module is used to determine the portrait data of the user according to the content of the application when it is determined that the user is in a focused state.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the user portrait data determination method described in the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for determining user portrait data described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the user portrait data determination method described in any one of the first aspects above.

[0018] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0019] When the application is running, the present application obtains the scene image collected by the terminal device, determines whether the user is in a focused state according to the user's facial posture in the scene image, and then determines the user's portrait data according to the content of the application when it is determined that the user is in a focused state. Since the scene image includes an image of a user who can see the display screen of the terminal device, and the user's facial posture in the scene portrait can reflect whether the user is focused on the display screen of the terminal device, and further can reflect whether the user is interested in the content of the application in the display screen, after determining that the user is in a focused state through the user's facial posture, the valid data that the user is really interested in can be determined according to the content of the application, thereby improving the accuracy of determining the user portrait data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a method for determining user portrait data provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of determining whether a user is in a focused state according to an embodiment of the present application;

[0023] Figure 3 It is a structural diagram of a device for determining user portrait data provided in an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0026] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0031] When using user portraits for personalized recommendations, the accuracy of the user portrait data in the user portrait will directly affect the accuracy of the personalized recommendations. Therefore, accurate determination of the user portrait data is crucial to building user portraits.

[0032] At present, recommendation algorithms can learn users’ browsing data, behavior data, etc., and use the learned data as user portrait data to build user portraits, and then make corresponding recommendations. For example, when recommending videos to users, mainstream recommendation algorithms mainly build user portraits by extracting behavioral data such as the types of videos watched and the time periods when users watch videos when using video software.

[0033] However, since users do not always pay attention when using video software, it is possible to learn behavioral data of users who inadvertently slide or stay on the software. Therefore, this solution for building user portraits may lead to certain learning errors, making the determination of user portrait data inaccurate.

[0034] Therefore, the above method has the problem of being unable to accurately determine user portrait data.

[0035] In order to improve the accuracy of determining user portrait data, an embodiment of the present application provides a method for determining user portrait data based on user facial posture. In this method, when an application is running, a scene image captured by a terminal device is obtained, and the user's facial posture in the scene image is detected. According to the user's facial posture, it is determined whether the user is paying attention to the content of the application. If it is determined that the user is paying attention to the content of the application, the user's portrait data is determined according to the content of the application.

[0036] Figure 1A flow chart of a method for determining user portrait data provided by an embodiment of the present application is shown. The method for determining user portrait data is applied to a terminal device and is described in detail as follows:

[0037] S11. When the application is running, obtain a scene image captured by the terminal device; the scene image includes an image of a user who can see a screen displayed by the terminal device.

[0038] The terminal device refers to a device with which a user can interact directly, such as a computer, a smart phone, a tablet computer, a television, or other device that can access network services and run applications. The application refers to an application running in a terminal device, such as a video software client APP, a browser, etc. The scene image refers to an image captured by the terminal device directly in front of the display screen of the terminal device. For example, the scene image is an image captured by a camera in the terminal device or a camera connected to the terminal device directly in front of the display screen of the terminal device.

[0039] Specifically, assuming that the terminal device includes a camera, when it is detected that an application in the terminal device is running, it means that the user may be using the application or browsing the content of the application. At this time, the camera of the terminal device can be enabled to capture the scene image in front of the display screen of the terminal device, and then the scene image captured by the camera of the terminal device is obtained. At the same time, since the captured scene image is the image in front of the display screen of the terminal device, the scene image can include an image of the user who can see the display screen of the terminal device.

[0040] In an embodiment of the present application, when the application is running, a scene image captured by a terminal device is obtained. When the user uses the application or browses the content of the application, an image of a user who can see the display screen of the terminal device directly in front of the display screen of the terminal device can be obtained, thereby improving the accuracy of scene image acquisition.

[0041] S12: Detect the user's facial posture in the scene image.

[0042] The user face gesture refers to a face attribute that reflects the relative position and direction of the user's face or head, and may include face gestures such as frontal face gesture and side face gesture. For example, when the user face gesture is a frontal face gesture, it means that the user is facing the display screen of the terminal device, and when the user face gesture is a side face gesture, it means that the user is not facing the display screen of the terminal device. The user face gesture can be used to indicate whether the user is paying attention to the display screen of the terminal device, and further reflect whether the user is paying attention to the content of the application.

[0043] Specifically, the facial gesture included in the scene image can be directly detected by one or more pre-trained deep learning neural network models, and then the detected facial gesture is used as the user facial gesture. For example, the user facial gesture in the scene image can be detected by one or more pre-trained models such as a face detection model, a face gesture recognition model, or a face tracking model. Alternatively, after detecting the facial gesture included in the scene image, the facial gesture of a specific user can be selected as the user facial gesture by a face recognition algorithm.

[0044] For example, assuming that the facial posture of user A is detected from the scene image, the facial posture of user A can be directly used as the above-mentioned user facial posture; or, if the facial postures of user A and user B are simultaneously detected from the scene image, and user B is a pre-stored user, then only the facial posture of user B can be used as the above-mentioned user facial posture.

[0045] It should also be noted that the information collection process (such as the scene image collection process) / feature extraction process (such as the user face posture detection process, face recognition process, etc.) involved in the embodiments of the present application is performed with the user's knowledge and permission, that is, the information collection process / feature extraction process complies with relevant regulations and does not constitute an act that harms the public interest.

[0046] In the embodiment of the present application, since the scene image is the image directly in front of the display screen of the terminal device, the above scene image may include users who pay attention to the application, and may also include users who do not pay attention to the application. By detecting the user's facial posture in the above scene image, it can be determined based on the user's facial posture whether the user pays attention to the content of the application, thereby improving the accuracy of determining the user portrait data.

[0047] S13: Determine whether the user is in a focused state according to the user's facial posture.

[0048] The above-mentioned attention state refers to the state in which the user is in when paying attention to the content of the application, that is, the state in which the user is in when being interested in the content of the application on the display screen of the terminal device.

[0049] Specifically, since the above-mentioned user facial gestures may include different facial gestures of the user, and different facial gestures can reflect whether the user is paying attention to the display screen of the terminal device, and further reflect whether the user is interested in the content of the application in the display screen, it is possible to determine whether the user is paying attention to the content of the above-mentioned application based on the duration of a specific facial gesture among the above-mentioned user facial gestures. If the duration of the specific facial gesture is greater than or equal to the preset duration threshold, it means that the user is paying attention to the content of the application stably, that is, the user is interested in the content of the application, and it can be determined that the user is in a state of attention; if the duration of the specific facial gesture is less than the preset duration threshold, it means that the user is not paying attention to the content of the application stably, that is, the user is not interested in the content of the application, and it can be determined that the user is not in a state of attention. In addition, after obtaining the user facial gesture, it is also possible to combine the user's facial expression, user operation data, etc. to determine whether the user is interested in the content of the application, so as to accurately determine whether the user is in a state of attention.

[0050] For example, assuming that the above-mentioned specific facial gesture is a frontal gesture, if the duration of the frontal gesture is greater than or equal to the preset duration threshold, it means that the user continues to pay attention to the content of the application, that is, the user is interested in the content of the application, and it can be determined that the user is in a focused state. Alternatively, when the user's facial gesture is a frontal gesture, it can also be determined whether the user is in a focused state based on the duration of the user's facial expression (such as a smile) and the size of the user's operation data (such as the user's operation frequency).

[0051] In the embodiment of the present application, whether the user is interested in the content of the application can be determined through the user's facial posture, thereby accurately determining whether the user is paying attention to the content of the application.

[0052] S14. When it is determined that the user is in a focused state, determine the portrait data of the user according to the content of the application.

[0053] Specifically, when it is determined that the user is paying attention to the content of the application, the time period in which the user is paying attention can be determined first, and the content of the application actually browsed by the user during the time period and / or the behavior data generated by the user when browsing the content of the application can be used as the user's portrait data. Since the user is in a state of attention, it means that the user continues to pay attention to the content of the application, the behavior of the user inadvertently sliding or staying can be excluded, so that more valid data can be determined from the content of the above application, and the accuracy of determining the user portrait data can be improved.

[0054] For example, assuming that the above application is a video software APP, and the time period when the above user is in the attention state is 10:05-10:10, then one or more data such as the time period, the type of videos browsed by the user in the time period, the videos clicked by the user in the time period, the videos shared by the user in the time period, the time the user stays on different videos in the time period, etc. can be used as the user's portrait data.

[0055] When the application is running, the present application obtains the scene image collected by the terminal device, determines whether the user is in a focused state according to the user's facial posture in the scene image, and then determines the user's portrait data according to the content of the application when it is determined that the user is in a focused state. Since the scene image includes an image of a user who can see the display screen of the terminal device, and the user's facial posture in the scene portrait can reflect whether the user is focused on the display screen of the terminal device, and further can reflect whether the user is interested in the content of the application in the display screen, after determining that the user is in a focused state through the user's facial posture, the valid data that the user is really interested in can be determined according to the content of the application, thereby improving the accuracy of determining the user portrait data.

[0056] In some embodiments, due to the difference in computing power of different terminal devices, the strategy for obtaining the image captured by the terminal device may be adjusted according to actual conditions to better meet different business needs. Therefore, the above-mentioned obtaining the scene image captured by the terminal device includes:

[0057] Acquire the scene image captured by the terminal device in real time, or acquire the scene image captured by the terminal device from a preset scene image sequence; wherein the scene image is an image captured by the terminal device according to a preset acquisition interval.

[0058] Specifically, if the terminal device has strong computing power (for example, real-time image processing will not affect the normal operation of the application), or if it is necessary to collect the user's portrait data in real time, the above-mentioned scene images collected by the terminal device in real time can be directly obtained, that is, the subsequent image processing process can be directly performed after each scene image is collected; if the terminal device has poor computing power (for example, real-time image processing may affect the normal operation of the application), or if it is not necessary to collect the user's portrait data in real time, the above-mentioned scene images collected by the terminal device can be arranged in the order of collection to obtain a scene image sequence, and then the scene images in the sequence can be obtained in turn for subsequent image processing.

[0059] Optionally, since the above scene images are images collected by the terminal device according to a preset collection interval, the strategy for obtaining the scene images collected by the terminal device can also be adjusted according to the size of the preset collection interval. If the preset collection interval is small, due to the presence of a large number of continuous scene images, each collected scene image may not be processed in time. In this case, the scene image sequence can be determined based on the scene image, and then the scene images can be obtained from the scene image sequence in sequence; if the preset collection interval is large, since the interval time of each collected scene image is large, each collected scene image can be processed in time. In this case, the subsequent image processing process can be performed directly after each scene image is collected.

[0060] In the embodiments of the present application, different strategies for acquiring scene images can be used to better adapt to different business scenarios, thereby improving the accuracy of determining user portrait data.

[0061] In some embodiments, when a pre-trained deep learning neural network model is used to perform user face detection, the above-mentioned detecting the user face posture in the above-mentioned scene image includes:

[0062] The face detection model and the face posture recognition model are used to detect the user's face posture in the above scene image.

[0063] The face detection model is used to detect faces in scene images, and the face posture recognition model is used to determine the face posture of the user in the faces.

[0064] In an optional embodiment of the present application, the face detection model may be a pre-trained YOLO model, and the face posture recognition model may be a RetinaFace model. Of course, other pre-trained neural network models may also be used, which are not limited here.

[0065] In the embodiment of the present application, the face detection model and the face posture recognition model can quickly and accurately detect the user's face posture in the scene image.

[0066] In some embodiments, the above-mentioned using a face detection model and a face gesture recognition model to detect the user's face gesture in the above-mentioned scene image includes:

[0067] Identify the user's face area in the scene image according to the face detection model;

[0068] The user's facial posture in the user's facial area is detected according to the facial posture recognition model; the user's facial posture includes a frontal posture or a non-frontal posture.

[0069] The user face area includes the face of a user who may be paying attention to the display screen of the terminal device.

[0070] Specifically, since the above-mentioned scene image may include users who are not paying attention to the display screen of the terminal device, in order to improve the accuracy of user facial gesture recognition, a face detection model can be used to first identify at least one user face area in the above-mentioned scene image, that is, first determine at least one face of a user who may pay attention to the display screen of the terminal device, and then detect the user face gesture in the above-mentioned at least one user face area according to the face gesture recognition model. At the same time, in order to better reflect whether the user is paying attention to the display screen of the terminal device according to the user face gesture, the detected user face gesture may only include a frontal face gesture or a non-frontal face gesture, wherein a user face gesture of a frontal face gesture indicates that the user is paying attention to the display screen of the terminal device, and a user face gesture of a non-frontal face gesture indicates that the user is not paying attention to the display screen of the terminal device.

[0071] Optionally, assuming that the face detection model identifies a user face region from the scene image, indicating that a user may be watching the display screen of the terminal device at this time, the user face posture in the user face region can be detected according to the face posture recognition model. If the user face posture is a frontal face posture, it indicates that the user is paying attention to the display screen of the terminal device. If the user face posture is a non-frontal face posture, it indicates that the user is not paying attention to the display screen of the terminal device. Assuming that the face detection model identifies more than one user face region from the scene image, the user face region of a specific user can be selected from them through face recognition, wherein the number of user face regions of the specific user is not greater than the number of user face regions recognized by the face detection model, and then the face posture recognition model is used to recognize the user face posture in the user face region of the specific user, so that the user face posture of the specific user can be determined in a targeted manner.

[0072] In some embodiments, the identifying the user face region in the scene image according to the face detection model includes:

[0073] Performing face detection on the scene image according to the face detection model to obtain at least one detection frame;

[0074] An image region corresponding to at least one of the detection frames is determined as a user face region in the scene image according to the area of ​​the detection frames.

[0075] Specifically, when the above-mentioned face detection model is used to identify the user face area in the scene image, at least one detection frame can be obtained. Since the user generally faces the terminal device (such as a mobile phone, computer, etc.) directly at a close distance when using the application or browsing the application content, the area of ​​the detection frame corresponding to the user's face in the scene image is relatively large. At this time, a preset number of detection frames can be selected from the above-mentioned at least one detection frame as the user face area according to the area size. Of course, the detection frame with the largest area can also be directly selected from the above-mentioned at least one detection frame as the above-mentioned user face area, which is not limited here.

[0076] For example, suppose that N detection frames are obtained when the face detection model is used to identify the user face area in the scene image, assuming that b 1 , b 2 , ..., b N , where N is a positive integer. At this time, the detection frame b with the largest area can be selected from the N detection frames, and its corresponding image area I b As the user's face area.

[0077] In an embodiment of the present application, when face detection is performed on a scene image through a face detection model, since users who are closer are generally users who use the application or browse the application content, that is, users who may pay attention to the application content, the user face area in the scene image is determined based on the area of ​​the obtained detection frame, which can improve the accuracy of determining the user face area.

[0078] In another optional embodiment of the present application, in order to improve the accuracy of user face gesture recognition, the above-mentioned face detection model and the above-mentioned face gesture recognition model are both verified models, and the verification process of the above-mentioned face detection model and the above-mentioned face gesture recognition model includes:

[0079] Acquire test data; the test data is video data including facial postures, wherein the facial postures in the video data include frontal facial postures and non-frontal facial postures;

[0080] Using the face detection model and the face posture recognition model to perform face classification on the test data, and obtain face classification results; the face classification results include classification results of the face frontal posture and classification results of the face non-frontal posture;

[0081] The above-mentioned face detection model and the above-mentioned face posture recognition model are verified according to the above-mentioned face classification results.

[0082] Specifically, when using test data to verify the above-mentioned face detection model and the above-mentioned face posture recognition model, the classification indicators of the face frontal posture and the face non-frontal posture can be calculated according to the classification results of the face frontal posture identified by the face detection model and the face posture recognition model, as well as the classification results of the above-mentioned face non-frontal posture, and when the classification index is greater than a preset indicator threshold, it is determined that the model verification has passed.

[0083] In an optional embodiment of the present application, the above classification indicators may include one or more of accuracy, recall rate, precision rate, etc.

[0084] For example, suppose the above test data is video data shot by the subject holding a mobile phone using the front camera, wherein A / 2 video data of some of the subjects’ faces not facing the mobile phone (for example, turning the head in any other posture) are collected, indicating that the subject is not interested in the video content at this time, and A / 2 video data of some of the subjects keeping their faces facing the mobile phone (with slight rotation) are collected, indicating that the subject is interested in the video content at this time, and the length of each video is fixed to T seconds (T can be set). Taking a video data as an example, suppose the video itself is x, and whether the subject’s face is facing the mobile phone is y1 (where y1=0 when the face is not facing the mobile phone, otherwise y1=1), and x is input into the algorithm (i.e., the face detection model and the face posture recognition model), and the algorithm will output the classification result y2: If y1=y2, it is determined that the model has correctly classified the test data, otherwise it is considered to be classified incorrectly. Assume that there are a total of b frontal face videos that are correctly classified, b ’ Frontal face videos are misclassified as non-frontal faces, c non-frontal face videos are correctly classified, c ’ If a non-frontal face video is misclassified as a frontal face, the classification accuracy acc can be calculated as acc = (b + c) / A, and the recall rate of the frontal face video is recall = b / (b + b ’ ), the accuracy of the frontal face video is precision = b / (b+c ’ ), if the business end's preset indicator thresholds for accuracy, recall, and precision are all 90%, then the verification is considered passed when all three values ​​are greater than 90%, otherwise the model structure can be adjusted or the model can be retrained until the verification is passed.

[0085] In some embodiments, determining whether the user is in a focused state according to the user's facial posture includes:

[0086] Determine whether the user's facial posture is a frontal posture;

[0087] If the user's face posture is a frontal posture, then the user's user stay time is increased; the user stay time is used to indicate the length of time the user pays attention to the content of the application;

[0088] Determining whether the user is in a focused state according to the relationship between the increased user stay time and a preset stay time threshold;

[0089] If the user's facial posture is a non-frontal posture, whether the user is in an attention state is determined according to the number of retention times of the user's stay time.

[0090] The above-mentioned user stay time refers to the duration of the user's face being displayed.

[0091] Specifically, since the terminal device will collect scene images according to the preset collection interval, it means that the facial posture recognition of the scene image is continuous. For the above-mentioned user facial posture in any frame of the scene image, if it is judged to be a frontal posture, it means that the user is facing the display screen of the terminal device, that is, the user is in use at this time, and is not inadvertently sliding or staying. In this case, the user's stay time can be increased, and the relationship between the increased user stay time and the preset stay time threshold (such as a size relationship, a ratio relationship, etc.) can be used to determine whether the user is paying attention to the content of the application. In addition, if the user's face posture in a certain frame of scene image is judged as a non-frontal posture, it means that the user is not facing the display screen of the terminal device. The user may be inadvertently sliding or staying at this time. However, if the above behavior occurs after the frontal posture is continuously detected, it may lead to inaccurate determination of the attention state. For example, the user's face posture in a certain frame of scene image is judged as a non-frontal posture, and the user's face posture in the scene images before and after the frame is all frontal postures or most of them are frontal postures. At this time, the user's actual frontal viewing time is also long. If the user's face posture in the frame of scene image is directly cleared when the user's face posture is judged as a non-frontal posture, the determined user's stay time may be short, thereby leading to inaccurate determination of the attention state. Therefore, when the user's face posture in a certain frame of scene image is judged as a non-frontal posture, the user's stay time can be retained first, and then the user can be determined whether the user is in the attention state according to the number of retention times of the user's stay time, thereby improving the accuracy of the attention state determination.

[0092] It should be noted that the user stay time can be the real time calculated by the timestamp. For example, at 10:01, the user's face posture is detected as a frontal posture, and at 10:02, the user's face posture is also detected as a frontal posture. Then, the increased user stay time at 10:02 is 1s. Of course, the user stay time can also be a standardized parameter duration. For example, the increased user stay time t ’ =t+1, wherein t is the previous user stay time, and 1 is the increased user stay time. The expression form of the user stay time is not limited here.

[0093] In an embodiment of the present application, when it is determined that the user's facial posture is a frontal posture, the user's user dwell time is increased. Since the above-mentioned user dwell time can reflect the length of time that the user pays attention to the content of the application, whether the user is in a state of attention can be more accurately determined based on the increased user dwell time; at the same time, when it is determined that the user's facial posture is not a frontal posture, by retaining the user dwell time, and then determining whether the user is in a state of attention to the content of the application based on the number of times the above-mentioned user dwell time is retained, it is equivalent to introducing a fault-tolerant mechanism, and also improves the accuracy of determining the user's attention status.

[0094] In some embodiments, determining whether the user is in the attention state according to the relationship between the increased user stay time and a preset stay time threshold includes:

[0095] Determine the relationship between the increased user stay time and a preset stay time threshold;

[0096] If the increased user stay time is greater than or equal to the preset stay time threshold, it is determined that the user is in a follow-up state;

[0097] If the increased user stay time is less than the preset stay time threshold, return to the step of obtaining the scene image captured by the terminal device.

[0098] Specifically, only when the user looks at the display screen of the terminal device with his / her face facing forward for a long enough time, does it mean that the user is maintaining a stable focus on the content of the current application, that is, the user is interested in the content of the current application and is in a focusing state. Therefore, the relationship between the increased user stay time and the preset stay time threshold can be determined. If the increased user stay time is greater than or equal to the preset stay time threshold, it means that the user is maintaining a stable focus on the content of the current application. At this time, it can be determined that the user is paying attention to the content of the above application; if the increased user stay time is less than the preset stay time threshold, it means that the time the user spends watching the terminal device display screen with his face forward is not long enough, and it cannot be said that the user is maintaining a stable focus on the content of the current application. However, since the user may continue to watch the terminal device display screen with his face forward in the future, it is necessary to continue to observe the user's behavior, that is, at this time the user is in a state of watching the terminal device display screen with his face forward, but needs to continue to observe. The scene image captured by the terminal device can be reacquired, and then the user's face can be tracked through a face tracking algorithm (such as a staple algorithm, etc.). During the tracking process, the face detection model and the face posture recognition model are used to extract the user's face posture in the scene image, thereby continuously observing the user's behavior until it can be determined whether the user is in a state of attention based on the user's stay time or the number of times the user's stay time is retained.

[0099] For example, assuming that after acquiring a frame of scene image, the user's face posture is detected as a frontal face posture, the user's stay time is set to t = 1, and it is determined whether it is less than the preset stay time threshold τ. If it is less than, continue to acquire a new frame of scene image. If the user's face posture is still detected as a frontal face posture, the user's stay time is set to t ’ =t+1=2, ..., continue the above steps until the increased user stay time t ’ If the time is greater than or equal to the preset dwell time threshold τ, it is confirmed that the user is in the attention state.

[0100] In an embodiment of the present application, by continuously detecting the user's facial posture in the acquired scene image and increasing the user's dwell time when a frontal facial posture is detected, the length of time the user faces the terminal device display screen with a frontal facial posture during the application running process can be counted, and then a preset dwell time threshold can be used to determine whether the user pays attention to the content of the application for a sufficiently long time, and then determine whether the user is interested in the content of the application, so that it can be accurately determined whether the user is in a state of attention.

[0101] In some embodiments, determining whether the user is in a follow-up state according to the number of times the user's stay time is retained includes:

[0102] Increase the retention times of the above user's stay time;

[0103] Determine the relationship between the increased number of retention times and the preset retention threshold;

[0104] If the increased retention times are greater than or equal to the preset retention threshold, it is determined that the user is not in the attention state, the user stay time is cleared, and the process returns to the step of obtaining the scene image captured by the terminal device;

[0105] If the increased number of retention times is less than the preset retention threshold, the user stay time is retained and the process returns to the step of obtaining the scene image captured by the terminal device.

[0106] Specifically, when it is determined that the user's facial posture in a certain frame of scene image is a non-frontal posture, the number of retention times of the user's user stay time can be increased first, and the relationship between the increased number of retention times and the preset retention threshold is determined. If the increased number of retention times is greater than or equal to the preset retention threshold, it indicates that the user has not faced the display screen of the terminal device for a long time. At this time, it can be determined that the user is not paying attention to the content of the application, then the user stay time is cleared, and the scene image collected by the terminal device is re-acquired, and the user behavior is continued to be observed according to the user's facial posture therein; if the increased number of retention times is less than the preset retention threshold, the number of retention times is small at this time, indicating that the user may not have faced the display screen of the terminal device for a short time. Since the user may continue to watch the display screen of the terminal device with his face in the future, it is necessary to continue to observe the user behavior, that is, at this time the user is in a state of not watching the display screen of the terminal device with his face in the front, but needs to continue to observe. By retaining the user stay time and re-acquiring the scene image collected by the terminal device, the user behavior is continued to be observed according to the user's facial posture therein until it can be determined whether the user is in the attention state according to the user stay time or the number of retention times of the user stay time.

[0107] For example, assuming that after acquiring a certain frame of scene image, the user's face posture is detected as a non-frontal posture, and the corresponding user stay time is t=3, the user stay time can be retained, and then the retention number is set to l=1, and it is determined whether it is less than the preset retention threshold ι. If it is less than, continue to acquire a new frame of scene image. If the user's face posture is still detected as a non-frontal posture, the retention number of the user stay time is set to l. ’ = l + 1 = 2, ..., continue the above steps until the increased retention times l ’ If it is greater than or equal to the preset retention threshold ι, it indicates that the user has not faced the display screen of the terminal device for a long time, confirming that the user is not paying attention to the content of the application, and then the user stay time is set to t=0, and a new scene image is acquired again.

[0108] In the embodiment of the present application, when the user's facial posture is detected as a non-frontal posture, the user's stay time is first retained, and the number of retentions corresponding to the user's stay time is increased, that is, the number of times the user faces the terminal device display screen with a non-frontal posture during the application operation process can be counted, and then by introducing a preset retention threshold to determine whether the user has not paid attention to the content of the application for enough times, it can be effectively determined whether the user is not in a state of attention to the content of the application; at the same time, the above-mentioned retention number can also avoid misjudging special situations as the user is not in a state of attention, for example, the above-mentioned special situation can be: the user's facial posture in multiple consecutive frames of scene images is a frontal posture, and then there are a few (for example, 2 frames) frames of scene images in which the user's facial posture is a non-frontal posture, and then the user's facial posture in the subsequent multiple consecutive frames of scene images is a frontal posture. In this case, the actual user also continues to pay attention to the content of the application for a long enough time, that is, the user is interested in the content of the application and is in a state of attention. Therefore, the relationship between the above-mentioned retention number and the preset retention threshold can more accurately determine whether the user is in a state of attention.

[0109] In order to better describe the method for determining user portrait data in the embodiment of the present application, Figure 2 Describes how to determine whether a user is paying attention. Figure 2As shown, during the application operation, the terminal device continuously collects multiple frames of scene images according to a preset collection interval, and then determines whether the user is in a state of attention through the following steps, including: S201, obtaining the i-th frame scene image; S202, detecting the user's facial posture in the i-th frame scene image; S203, judging whether the user's facial posture is a frontal posture, if the user's facial posture is a frontal posture, executing S204, increasing the user's stay time, and then executing S206, judging whether the user's stay time is greater than or equal to a preset stay time threshold, if so, executing S208, determining that the user is in a state of attention, if not, it indicates that the user is in a state of viewing the terminal device display screen with a front face, but needs to continue to observe, that is, it is necessary to continue to observe the user's behavior, executing S212, i=i+1, and returning to S201, and continuing to the next frame of scene image Detection; if the user's face posture is a non-frontal posture, then execute S205, increase the number of retentions corresponding to the user's stay time, then execute S207, determine whether the number of retentions is greater than or equal to the preset retention threshold, if the number of retentions is greater than or equal to the preset retention threshold, then execute S209, determine that the user is not in a state of attention, then execute S211, clear the user's stay time, then execute S212, i=i+1, and return to S201 to continue to detect the next frame of scene image; if the number of retentions is less than the preset retention threshold, it indicates that the user is in a state of not watching the terminal device display screen with a front face, but needs to continue to observe, that is, it is necessary to continue to observe the user's behavior, at this time, S210 can be executed, the user's stay time can be retained, then S212, i=i+1, and return to S201 to continue to detect the next frame of scene image. Of course, after determining that the user is in a state of attention, the next frame of scene image can be continuously detected, so as to continuously determine whether the user is in a state of attention during the application operation.

[0110] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0111] Corresponding to the method for determining user portrait data in the above embodiment, Figure 3 A structural schematic diagram of a user portrait data determination device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0112] Reference Figure 3 The device may be a user portrait data determination device 31, and the user portrait data determination device 31 may include a scene image acquisition module 311, a face posture detection module 312, a state judgment module 313 and a portrait data determination module 314.

[0113] Reference Figure 3 , the 31 includes:

[0114] The scene image acquisition module 311 is used to acquire the scene image collected by the terminal device when the application is running; the scene image includes an image of a user who can see the display screen of the terminal device;

[0115] The above-mentioned face gesture detection module 312 is used to detect the user's face gesture in the above-mentioned scene image;

[0116] The state determination module 313 is used to determine whether the user is in a focused state according to the user's facial posture;

[0117] The portrait data determination module 314 is used to determine the portrait data of the user according to the content of the application when it is determined that the user is in a focused state.

[0118] In some embodiments, due to the difference in computing power of different terminal devices, the strategy for acquiring images captured by the terminal device may be adjusted according to actual conditions to better meet different business needs. Therefore, when acquiring the scene image captured by the terminal device, the scene image acquisition module 311 is used to:

[0119] Acquire the scene image captured by the terminal device in real time, or acquire the scene image captured by the terminal device from a preset scene image sequence; wherein the scene image is an image captured by the terminal device according to a preset acquisition interval.

[0120] In some embodiments, when using a pre-trained deep learning neural network model to perform user face detection, the face gesture detection module 312 is used to:

[0121] The face detection model and the face posture recognition model are used to detect the user's face posture in the above scene image.

[0122] In some embodiments, when the face gesture detection module 312 detects the face gesture of the user in the scene image using the face detection model and the face gesture recognition model, it is used to:

[0123] Identify the user's face area in the scene image according to the face detection model;

[0124] The user's facial posture in the user's facial area is detected according to the facial posture recognition model; the user's facial posture includes a frontal posture or a non-frontal posture.

[0125] In some embodiments, when the face gesture detection module 312 identifies the user face area in the scene image according to the face detection model, it is used to:

[0126] Performing face detection on the scene image according to the face detection model to obtain at least one detection frame;

[0127] An image region corresponding to at least one of the detection frames is determined as a user face region in the scene image according to the area of ​​the detection frames.

[0128] In another optional embodiment of the present application, in order to improve the accuracy of user face gesture recognition, the above-mentioned face detection model and the above-mentioned face gesture recognition model are both verified models, and the above-mentioned user portrait data determination device 31 also includes a model verification module. The verification process of the above-mentioned model verification module on the face detection model and the above-mentioned face gesture recognition model includes:

[0129] Acquire test data; the test data is video data including facial postures, wherein the facial postures in the video data include frontal facial postures and non-frontal facial postures;

[0130] Using the face detection model and the face posture recognition model to perform face classification on the test data, and obtain face classification results; the face classification results include classification results of the face frontal posture and classification results of the face non-frontal posture;

[0131] The above-mentioned face detection model and the above-mentioned face posture recognition model are verified according to the above-mentioned face classification results.

[0132] In some embodiments, when determining whether the user is in a focused state according to the user's facial posture, the state determination module 313 is used to:

[0133] Determine whether the user's facial posture is a frontal posture;

[0134] If the user's face posture is a frontal posture, then the user's user stay time is increased; the user stay time is used to indicate the length of time the user pays attention to the content of the application;

[0135] Determining whether the user is in a focused state according to the relationship between the increased user stay time and a preset stay time threshold;

[0136] If the user's facial posture is a non-frontal posture, whether the user is in an attention state is determined according to the number of retention times of the user's stay time.

[0137] In some embodiments, when determining whether the user is in the attention state according to the relationship between the increased user stay time and the preset stay time threshold, the state determination module 313 is used to:

[0138] Determine the relationship between the increased user stay time and a preset stay time threshold;

[0139] If the increased user stay time is greater than or equal to the preset stay time threshold, it is determined that the user is in a follow-up state;

[0140] If the increased user stay time is less than the preset stay time threshold, return to the step of obtaining the scene image captured by the terminal device.

[0141] The first pending state refers to a state in which the user is viewing the display screen of the terminal device with his / her face facing forward but needs to continue observing.

[0142] In some embodiments, when determining whether the user is in the attention state according to the number of times the user's stay time is retained, the state determination module 313 is used to:

[0143] Increase the retention times of the above user's stay time;

[0144] Determine the relationship between the increased number of retention times and the preset retention threshold;

[0145] If the increased retention times are greater than or equal to the preset retention threshold, it is determined that the user is not in the attention state, the user stay time is cleared, and the process returns to the step of obtaining the scene image captured by the terminal device;

[0146] If the increased number of retention times is less than the preset retention threshold, the user stay time is retained and the process returns to the step of obtaining the scene image captured by the terminal device.

[0147] It should be noted that the information interaction, execution process, etc. between the devices / units are based on the same concept as the method embodiments of the present application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0148] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4The at least one processor 40 includes a memory 41 and a computer program 42 stored in the memory 41 and executable on the at least one processor 40. When the processor 40 executes the computer program 42, the steps in any of the method embodiments are implemented.

[0149] The electronic device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include an input sending device, a network access device, a bus, etc.

[0150] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0151] In some embodiments, the memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. The memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 4. Further, the memory 41 may also include both an internal storage unit of the electronic device 4 and an external storage device. The memory 41 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 41 may also be used to temporarily store data that has been sent or is to be sent.

[0152] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the functional units and modules is used as an example. In practical applications, the function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0153] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the method embodiments when executing the computer program.

[0154] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the various method embodiments can be implemented.

[0155] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the various method embodiments when executing the computer program product.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0157] In the embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for determining user portrait data, characterized in that: Applied to terminal equipment, including: When the application is running, obtaining a scene image captured by the terminal device; the scene image includes an image of a user who can see a display screen of the terminal device; Detecting a user's facial posture in the scene image; Determining whether the user is in a focused state according to the user's facial posture; When it is determined that the user is in a focused state, the portrait data of the user is determined according to the content of the application.

2. The method for determining user portrait data according to claim 1, characterized in that: The acquiring the scene image collected by the terminal device includes: Acquire the scene image captured by the terminal device in real time, or acquire the scene image captured by the terminal device from a preset scene image sequence; wherein the scene image is an image captured by the terminal device according to a preset acquisition interval.

3. The method for determining user portrait data according to claim 1, characterized in that: The detecting the user's facial posture in the scene image includes: The face detection model and the face gesture recognition model are used to detect the user's face gesture in the scene image.

4. The method for determining user portrait data according to claim 3, characterized in that: The method of detecting the user's facial posture in the scene image by using a face detection model and a face posture recognition model includes: Identifying a user's face area in the scene image according to the face detection model; The user's facial posture in the user's facial area is detected according to the facial posture recognition model; the user's facial posture includes a frontal facial posture or a non-frontal facial posture.

5. The method for determining user portrait data according to claim 4, characterized in that: The step of identifying a user face region in the scene image according to the face detection model includes: Performing face detection on the scene image according to the face detection model to obtain at least one detection frame; An image region corresponding to at least one of the detection frames is determined as a user face region in the scene image according to the area of ​​the detection frame.

6. The method for determining user portrait data according to claim 3, characterized in that: The face detection model and the face gesture recognition model are both verified models, and the verification process of the face detection model and the face gesture recognition model includes: Acquire test data; the test data is video data including facial postures, wherein the facial postures in the video data include frontal facial postures and non-frontal facial postures; Using the face detection model and the face posture recognition model to perform face classification on the test data to obtain face classification results; the face classification results include classification results of the frontal face posture of the face and classification results of the non-frontal face posture of the face; The face detection model and the face gesture recognition model are verified according to the face classification result.

7. The method for determining user portrait data according to any one of claims 1 to 6, characterized in that: The determining whether the user is in a focused state according to the user's facial posture includes: Determining whether the user's facial posture is a frontal posture; If the user's facial posture is a frontal posture, then the user's user stay time is increased; the user stay time is used to indicate the length of time the user pays attention to the content of the application; Determining whether the user is in a focused state according to the relationship between the increased user stay time and a preset stay time threshold; If the user's facial posture is a non-frontal posture, whether the user is in a focused state is determined according to the number of times the user's stay time is retained.

8. The method for determining user portrait data according to claim 7, characterized in that: The determining whether the user is in the attention state according to the relationship between the increased user stay time and the preset stay time threshold includes: Determine the relationship between the increased user stay time and a preset stay time threshold; If the increased user stay time is greater than or equal to the preset stay time threshold, it is determined that the user is in a paying attention state; If the increased user stay time is less than the preset stay time threshold, return to the step of obtaining the scene image captured by the terminal device.

9. The method for determining user portrait data according to claim 7, characterized in that: The determining whether the user is in a focused state according to the number of retention times of the user's stay time includes: Increasing the number of retention times of the user's stay time of the user; Determine the relationship between the increased number of retention times and a preset retention threshold; If the increased number of retention times is greater than or equal to the preset retention threshold, it is determined that the user is not in the attention state, the user stay time is cleared, and the process returns to the step of obtaining the scene image captured by the terminal device; If the increased retention times are less than the preset retention threshold, the user stay time is retained and the process returns to the step of obtaining the scene image captured by the terminal device.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images