Video playing processing method and device and storage medium

Abstract

The invention discloses a video playing processing method and device and a storage medium. The method comprises the following steps: acquiring a face image in a predetermined area of a display screen; the visual direction of an object included in the face image is detected, a visual detection result is obtained, and the visual detection result is used for indicating whether the visual direction is the direction facing the display screen or not and indicating the accumulated duration when the visual direction is not the direction facing the display screen; and determining a video playing strategy according to the visual detection result. Through the video playing method and device, the problems of poor playing flexibility, poor user experience and large resource waste existing in video playing in a timed playing or continuous playing mode in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a video playback processing method, device and storage medium. Background Art

[0002] Promotional videos can effectively convey information about products or services, making it easier for audiences to understand and remember through visual and auditory means. In related technologies, when playing videos based on display screens, it is mainly done through timed playback or continuous playback. However, the timed playback method is prone to situations where the video continues to play when the user does not want to watch the video, and the video playback stops when the user wants to watch the video. The playback flexibility is poor, which can easily bring a bad viewing experience to users; while continuous playback can easily cause a waste of resources.

[0003] Currently, no effective solution has been proposed for the problems of poor playback flexibility, poor user experience and large waste of resources in video playback through timed playback or continuous playback in related technologies. Summary of the invention

[0004] The main purpose of the present application is to provide a video playback processing method, device and storage medium to solve the problems of poor playback flexibility, poor user experience and large resource waste in the related art of performing video playback through timed playback or continuous playback.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a video playback processing method is provided. The method comprises: obtaining a face image in a predetermined area of ​​a display screen; detecting the visual direction of an object included in the face image to obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen, and the cumulative length of time that the visual direction is not toward the display screen; and determining a video playback strategy according to the visual detection result.

[0006] Optionally, determining the video playback strategy based on the visual detection result includes: when the visual detection result indicates that the visual direction is toward the display screen, determining the first video playback strategy as: controlling the display screen to play video content; or when the visual detection result indicates that the visual direction is not toward the display screen, determining the second video playback strategy as: controlling the display screen to stop playing the video, and based on the accumulated duration, determining a stopping method for controlling the display screen to stop playing the video.

[0007] Optionally, based on the accumulated duration, a stopping method for controlling the display screen to stop playing the video is determined, including: when the accumulated duration is less than or equal to a preset second duration, determining a first stopping method as: controlling the display screen to display a still picture, wherein the preset second duration is greater than or equal to the preset first duration; or when the accumulated duration is greater than the preset second duration and less than or equal to a preset third duration, determining a second stopping method as: controlling the display screen to enter a sleep state, wherein the preset third duration is greater than the preset second duration; or when the accumulated duration is greater than the preset third duration, determining a third stopping method as: controlling the display screen to shut down.

[0008] Optionally, when the visual detection result indicates that the visual direction is toward the display screen, the first video playback strategy is determined as: after controlling the display screen to play the video content, the method also includes: determining the object type corresponding to the object included in the face image; detecting whether the current video content played on the display screen matches the object type; when the current video content does not match the object type, determining target video content that matches the object type; and controlling the current video content played on the display screen to be switched to the target video content.

[0009] Optionally, when the visual direction of N objects among the objects included in the facial image is toward the display screen, and N is greater than or equal to 2, determining the target video content that matches the object type includes: obtaining M object types and the number of objects corresponding to the M object types based on the object types corresponding to the N objects, wherein M is an integer greater than or equal to 1; and determining the video content corresponding to the object type with the largest number of objects among the M object types as the target video content.

[0010] Optionally, the method also includes: when the visual detection result indicates that the visual direction is toward the display screen, detecting whether the object whose visual direction is toward the display screen is a target object; when the object whose visual direction is toward the display screen is the target object, detecting whether the current video content played on the display screen is the preferred video content of the target object; and when the current video content is not the preferred video content, controlling the current video content played on the display screen to switch to the preferred video content.

[0011] Optionally, detecting whether the object whose visual direction is toward the display screen is the target object includes: matching facial image information of the object whose visual direction is toward the display screen with facial image information pre-stored in a target database, and detecting whether the facial image information of the object whose visual direction is toward the display screen is in the target database; and determining that the object whose visual direction is toward the display screen is the target object if the facial image information of the object whose visual direction is toward the display screen is in the target database.

[0012] Optionally, acquiring a facial image within a predetermined area of ​​the display screen includes: acquiring image information within the predetermined area by means of K image acquisition devices, wherein K is an integer greater than or equal to 2, and the K image acquisition devices correspond to different image acquisition areas; and identifying facial image information in the image information to obtain the facial image within the predetermined area.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a video playback processing device is provided. The device includes: an acquisition module, used to acquire a face image in a predetermined area of ​​a display screen; a detection module, used to detect the visual direction of an object included in the face image, and obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen, and the cumulative length of time that the visual direction is not toward the display screen; and a playback strategy determination module, used to determine a video playback strategy according to the visual detection result.

[0014] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing any one of the above-mentioned video playback processing methods.

[0015] Through this application, the following steps are adopted: obtaining a face image in a predetermined area of ​​a display screen; detecting the visual direction of the object included in the face image to obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen and the cumulative time that the visual direction is not toward the display screen; determining a video playback strategy according to the visual detection result. The purpose of flexibly selecting a video playback strategy based on the visual direction and the cumulative time that the visual direction is not toward the display screen is achieved, and the problems of poor playback flexibility, poor user experience and large resource waste in the related art of video playback through timed playback or continuous playback are solved. This further achieves the effect of improving the flexibility of video playback and reducing the consumption of video playback resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a flowchart of a video playback processing method provided in an embodiment of the present application; and

[0018] Figure 2 is a flowchart of an optional video playback processing method according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a video playback processing device provided according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0025] A collection device refers to a device or system used to collect and store data or materials. In different fields, collection devices have different applications. Collection devices usually include sensors, data collectors, storage devices and other components, through which the required data can be collected and recorded. The data can be images or videos.

[0026] Image frame refers to a still image in a video or animation. Image frame is the smallest unit of video or animation. Video is composed of a series of continuous image frames to create the illusion of motion. Each frame is a still image. When the image frames are played at a fast enough speed, a continuous dynamic effect can be presented. Image frame is the basic element for building videos and animations.

[0027] Facial features refer to the unique shapes, lines and features of a person's face, including the shape and position of the eyes, nose, mouth, chin and other parts, as well as individual unique features such as facial contours, wrinkles, moles and scars. Facial features can be used to identify and distinguish different individuals, and can also be used to judge people's emotions and expressions. In the field of computer vision and artificial intelligence, facial features are often used for tasks such as face recognition.

[0028] The present application is described below in conjunction with preferred implementation steps. Figure 1 is a flowchart of a video playback processing method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0029] Step S101, obtaining a face image in a predetermined area of ​​a display screen;

[0030] Optionally, the display screen is used to play video content. The display screen can be a large LED screen in the venue or a home display, such as a TV. The reserved area of ​​the display screen is the standing area where the content of the display screen can be seen, where the reserved area of ​​the display screen includes but is not limited to the front of the display screen and the left and right sides of the display screen.

[0031] In an optional embodiment, obtaining a facial image within a predetermined area of ​​a display screen includes: acquiring image information within the predetermined area through K image acquisition devices, wherein K is an integer greater than or equal to 2, and the K image acquisition devices correspond to different image acquisition areas; identifying facial image information in the image information to obtain a facial image within the predetermined area.

[0032] Optionally, the image acquisition device is used to acquire image information within a predetermined area. The image acquisition device uses a shooting camera, sets the camera shooting time interval to 0.1 seconds, and outputs multiple frames of images. The acquisition device can also use a video recording device to split the acquired video into images according to a predetermined frame size, save them in the form of images, and perform subsequent recognition.

[0033] Optionally, multiple image acquisition devices are targeted to capture images of different predetermined areas. For example, the display screen is mostly rectangular, and three image acquisition devices are set around the display screen, wherein the first image acquisition device is installed in the middle of the long side above the display screen, and the shooting direction of the first image acquisition device is tilted downward by a preset angle, which is set according to the height of the display screen to ensure that the image directly in front of the display screen can be captured, the second image acquisition device is installed in the middle of the left wide side of the display screen, and tilted to the left by a preset angle to ensure that all images in the area where the display screen content can be seen on the left side of the display screen can be captured, and the third image acquisition device is installed in the middle of the right wide side of the display screen, and tilted to the right by a preset angle to ensure that all images in the area where the display screen content can be seen on the right side of the display screen can be captured. In addition, since the tilt angle and installation height of each image acquisition device are different, it can be ensured that faces at different heights can be captured, thereby improving the comprehensiveness of face image capture.

[0034] Optionally, each image acquisition device is provided with multiple cameras for expanding the shooting range, and the shooting range can be flexibly adjusted. When playing videos in a conference venue, since the venue is large, in order to capture a comprehensive picture, the image acquisition device is set to wide-angle shooting using multiple cameras in the image acquisition device, and different wide-angle degrees can be set according to the size of the venue; when playing videos on a home monitor, there is no need to set a wide-angle because the room is not large.

[0035] Step S102, detecting the visual direction of the object included in the face image to obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen, and the cumulative length of time that the visual direction is not toward the display screen;

[0036] Optionally, the facial information in the multiple frames of images obtained by the image acquisition device is recognized frame by frame, and the facial information can be recognized by using, but not limited to, computer vision technology, wherein the computer vision technology can determine the visual direction by identifying the position of the face, and can also track the direction of the human eye's line of sight through an eye tracking algorithm to determine the visual direction. If the direction of the human eye's line of sight is within the preset area, it proves that the visual direction is on the display screen; if the direction of the human eye's line of sight is not within the preset area, it proves that the visual direction is not on the display screen; when the face recognition result indicates that no face information is detected, it is directly determined that no one is looking at the display screen.

[0037] Optionally, a multi-task cascade convolutional neural network model can be used to extract facial information in an image, and visual direction recognition can be performed by identifying the orientation of the face. Specifically, first, multiple tasks are set for the multi-task cascade convolutional neural network, where each task corresponds to a feature on the face, that is, the mouth corresponds to a task, the nose corresponds to a task, and each feature in the face corresponds to a task, so that the accuracy of subsequent extraction can be guaranteed; multiple frames of images are continuously passed into the multi-task cascade convolutional neural network model, and the multiple frames of images are convolved through the first layer to extract basic features such as edges and textures to obtain a preliminary feature image; the preliminary feature image is passed through a pooling layer to reduce its dimension and reduce the amount of calculation in the subsequent extraction process, and the preliminary feature image is expanded to form a one-dimensional vector, and matched and classified or regressed with multiple tasks through a fully connected layer to identify facial information, where the facial information is the feature in the face. The facial orientation is determined by judging the integrity of the facial features. When the facial features are missing or part of the facial features are blocked, it is determined that the facial orientation is not toward the display screen, that is, the content played on the display screen is not viewed.

[0038] Step S103, determining a video playback strategy according to the visual detection result.

[0039] Optionally, the video playback strategy includes at least one of the following: controlling the display screen to play the video normally when someone is watching the display screen, controlling the display screen to display a still picture when the viewing object is in a predetermined area but the visual direction is not on the display screen, controlling the display screen to enter a sleep state when the viewing object leaves the predetermined area within a predetermined time, and controlling the display screen to shut down when the viewing object leaves the predetermined area beyond the predetermined time.

[0040] In an optional embodiment, a video playback strategy is determined based on the visual detection result, including: when the visual detection result indicates that the visual direction is toward the display screen, determining the first video playback strategy as: controlling the display screen to play the video content; or when the visual detection result indicates that the visual direction is not toward the display screen, determining the second video playback strategy as: controlling the display screen to stop playing the video, and based on the accumulated duration, determining the stopping method of controlling the display screen to stop playing the video.

[0041] Optionally, if the visual direction is toward the display screen, the video content is played; if the visual direction is not toward the display screen, the timing starts from the moment when the visual direction leaves the display screen, wherein there are two situations when the visual direction is not toward the display screen. The first is that the viewing object is in the predetermined area, but the facial features are partially covered, that is, the viewing object turns his head to look in other directions, and the visual direction is offset; the second is that the viewing object leaves the predetermined area. When the viewing object's line of sight temporarily leaves the display screen, where temporarily means within a certain preset time period, the viewing object may be communicating with other viewing objects. In order to avoid the viewing object being unable to connect to the previously seen part when returning due to a long communication time, the video being played is paused, which effectively ensures that the viewing object can watch continuously when returning.

[0042] Optionally, timing starts from the moment when the user leaves the display screen from the visual direction, and the timing method adopts the minute timing method, that is, the count obtained after timing is in minutes. Minute timing only requires time accumulation and does not need to be converted into hours, thereby reducing the calculation time during operation.

[0043] In an optional embodiment, based on the accumulated duration, a stopping method for controlling the display screen to stop playing the video is determined, including: when the accumulated duration is less than or equal to a preset second duration, determining the first stopping method as: controlling the display screen to display a still picture, wherein the preset second duration is greater than or equal to the preset first duration; or when the accumulated duration is greater than the preset second duration and less than or equal to a preset third duration, determining the second stopping method as: controlling the display screen to enter a sleep state, wherein the preset third duration is greater than the preset second duration; or when the accumulated duration is greater than the preset third duration, determining the third stopping method as: controlling the display screen to shut down.

[0044] Optionally, when the viewing object leaves the predetermined area and the time of leaving is short, that is, the time of leaving is less than or equal to the second duration, it is determined that the viewing object has left temporarily. At this time, the viewing object still intends to continue watching, and the video in the display screen is only temporarily stilled, which effectively ensures that the viewing object can be watched continuously when returning to watch. After the viewing object leaves, the time of leaving exceeds the second duration and is less than the third duration. At this time, the viewing object cannot return to watch in time after leaving. If the display screen is in the video screen state for a long time, power resources are always consumed, and resources are wasted. Therefore, the display screen at this time is set to a dormant state. The display screen in the dormant state does not display the video, but the video can be turned on at any time, that is, the standby state. Since the dormant state also consumes power resources, when the viewing object leaves for a long time, that is, the time of leaving the viewing object exceeds the third duration, the display screen is turned to a shutdown state at this time to stop resource consumption. Among them, the resource consumption when the display screen displays a static picture is greater than the resource consumption in the dormant state, and the resource consumption in the dormant state is greater than the resource consumption in the shutdown state.

[0045] Optionally, for the accumulated duration when the visual direction is not toward the display screen, each time the accumulated duration is obtained, it indicates that the viewing object has left the screen once, and the corresponding accumulated times are increased by one. When the accumulated times exceed the predetermined times, it is possible that the viewing object did not watch the video on the display screen, but when doing other things, the visual direction passed the direction toward the display screen. At this time, a prompt is sent to the viewing object to prompt the viewing object to determine whether to continue playing the video; the feedback result of the viewing object is received, if the feedback is to continue playing, the playback is continued; if the feedback is not to continue playing, the playback is stopped. The above method can avoid problems such as video freezes caused by frequent playback stops, which affect the viewing object experience.

[0046] Optionally, when the visual direction is not towards the display screen, the accumulated time is accumulated, and each time the accumulated time is obtained, the accumulated times are increased by one, and the time interval between the accumulated times is obtained. When the visual direction is shifted multiple times within a predetermined time, the accumulated times are increased by one. When the predetermined time is exceeded, the accumulated times are reset to zero, and the times are accumulated again after the visual direction is shifted. This effectively avoids the situation where the viewing object is reminded after leaving only once during subsequent viewing when the viewing time span is large. For example, the accumulated times for prompting the viewing object to determine whether to continue playing the video is N+1 times. After leaving N times in the morning, when continuing to watch in the afternoon, leaving again, a prompt is issued to the viewing object. This effectively avoids the situation where incorrect prompts affect the viewing object's experience.

[0047] In an optional embodiment, detecting whether an object whose visual direction is toward the display screen is a target object includes: matching facial image information of the object whose visual direction is toward the display screen with facial image information pre-stored in a target database, and detecting whether the facial image information of the object whose visual direction is toward the display screen is in the target database; if the facial image information of the object whose visual direction is toward the display screen is in the target database, determining that the object whose visual direction is toward the display screen is the target object.

[0048] Optionally, when the number of viewing objects is less than or equal to the predetermined number range value, or in the case of a home scene, due to the small number of people and the different video contents corresponding to different viewing objects, the target object is identified by matching the preset face database, which is conducive to the correspondence between the multiple groups of viewing object face images stored in the target database and the preferred video content. Specifically, the viewing object is matched with the facial image information pre-stored in the target database using a feature point matching algorithm to obtain the target object; first, the facial information of different viewing objects is extracted using face recognition technology to obtain landmark features, including multiple feature points, contours, colors, textures and other information of the face. Matching is performed by comparing the similarity between multiple landmark features and calculating the distance between multiple landmark features. Specifically, the nearest neighbor matching method is used to calculate the distance between multiple feature points, determine the distance between the feature points as a data set, and match it with the facial image information pre-stored in the preset target database to determine whether the viewing object is the target object, wherein the distance between multiple feature points is the distance between the eyes, the distance between the center of the eyebrows to the mouth, and the distance between the cheekbones of the two cheeks.

[0049] In an optional embodiment, the method further includes: when the visual detection result indicates that the visual direction is toward the display screen, detecting whether the object whose visual direction is toward the display screen is the target object; when the object whose visual direction is toward the display screen is the target object, detecting whether the current video content played on the display screen is the preferred video content of the target object; when the current video content is not the preferred video content, controlling the current video content played on the display screen to switch to the preferred video content.

[0050] Optionally, home displays often have multiple types of program videos, and venue promotional videos will also play different types of videos for different groups. When different viewing objects appear in front of the display screen, the viewing objects need to manually change the type of video, resulting in a poor viewing experience for the viewing objects. Different viewing objects watch different types of videos, and the preferences of different viewing objects are pre-set, or the target videos that each viewing object likes are obtained by big data capture. The facial information is collected through the collection device, and the extracted facial information is matched with the preferred video content of the viewing object. When one viewing object leaves the predetermined area and another viewing object enters the predetermined area, the preferred video content that matches the facial information of the viewing object is automatically played based on the facial information collected from the viewing object. This method effectively plays the matching preferred video content according to different viewing objects, and plays the video in a targeted manner, thereby improving the viewing object's experience.

[0051] In an optional embodiment, when the visual detection result indicates that the visual direction is toward the display screen, the first video playback strategy is determined as: after controlling the display screen to play the video content, the method also includes: determining the object type corresponding to the object included in the face image; detecting whether the current video content played on the display screen matches the object type; when the current video content does not match the object type, determining the target video content that matches the object type; and controlling the current video content played on the display screen to switch to the target video content.

[0052] Optionally, the object type can be the age range, gender, etc. of the viewing object. Taking the age range of the viewing object as an example, different types of videos are played for different groups in the venue promotional video. Different groups are defined as people of different age stages. By identifying the age range of facial information, when the age is less than a first predetermined age range, a video mainly with animation is played; when the age is greater than the first predetermined age range and less than the second predetermined age range, a video mainly with current hot topics is played; and when the age is greater than the second predetermined age range, a video mainly with easy-to-understand content is played.

[0053] Optionally, by identifying the age range of facial information, specifically, first collect a large amount of facial image data, including facial images of different age ranges and corresponding age labels, build an age differentiation model, input the facial image data into the age differentiation model, and pre-process the facial image data, including image denoising, grayscale, normalization and other operations, to facilitate feature extraction and classification. Use feature extraction machine learning algorithm (Haar) to extract facial features from facial image data, use support vector machine (SVM) and decision tree to classify the age range of facial images according to facial features. Use a large amount of collected facial image data to train the age differentiation model to identify facial features of different age groups. According to the facial information extracted by the multi-task cascade convolutional neural network, determine the position and feature points of the face, and match them with the facial features of different age groups, so as to achieve age classification of the facial information obtained by the acquisition device.

[0054] Optionally, in the case of viewing on a home display, there may be a situation where the viewer has not finished watching the video, and still wants to continue watching the unfinished video content during the next viewing process, then the unfinished video content is determined as the target video content, firstly, it is determined based on the facial information of the viewer whether the current video content is the target video content that the viewer has not finished watching, when the video content currently played on the display screen is not the target video that the viewer has not finished watching, the current video is controlled to switch to the target video content, and when it is detected that the viewer has not finished watching the target video content, the preferred video content that matches the facial information of the viewer is automatically played, thereby improving the viewing experience of the viewer.

[0055] In an optional embodiment, when the visual direction of N objects among the objects included in the face image is toward the display screen, and N is greater than or equal to 2, determining the target video content that matches the object type includes: based on the object types corresponding to the N objects, obtaining M object types and the number of objects corresponding to the M object types, wherein M is an integer greater than or equal to 1; and determining the video content corresponding to the object type with the largest number of objects among the M object types as the target video content.

[0056] Optionally, when there is more than one viewing object opposite the display screen, and the viewing objects like different video contents, the number of viewing objects is identified and the viewing objects are classified, and the group with more viewing objects after classification is determined as the target group, and the video content corresponding to the target group is the target video content, wherein the viewing objects can be classified according to age group, according to gender, or according to family members, and the video content is matched and played based on the number of target groups. In the above manner, according to the principle of "the minority is subject to the majority", the target video content is selected to meet the video viewing needs of more viewing objects to the greatest extent.

[0057] Optionally, a video content suitable for all viewers is set as the target video content, and when multiple viewing groups are detected, the target video content suitable for all viewers is played, so as to better meet the video viewing needs of more viewers and improve the viewing experience of the viewers.

[0058] Through the above steps S101 to S103, the purpose of flexibly selecting a video playback strategy according to the visual direction and the cumulative duration that the visual direction is not toward the display screen can be achieved, solving the problem of poor playback flexibility, poor user experience and large resource waste in the related art of playing videos by timed playback or continuous playback. Thus, the effect of improving the flexibility of video playback and reducing the consumption of video playback resources is achieved.

[0059] Based on the above embodiments and optional embodiments, the present application proposes an optional implementation mode: Figure 2 is a flowchart of an optional video playback processing method according to an embodiment of the present application. Figure 2 As shown, the method includes:

[0060] S1, cameras are arranged around the display screen, and video is collected within a predetermined area through the cameras;

[0061] Among them, multiple cameras can be set on the display screen, and each camera corresponds to a different collection area, which is convenient for face collection of people of different heights or different location areas.

[0062] S2, perform face recognition based on the collected video to obtain face recognition results, specifically:

[0063] Computer vision technology can be used, but is not limited to, such as a face detection algorithm based on deep learning such as a multi-task cascade convolutional neural network (MTCNN), to perform face detection on the collected video to obtain a face recognition result, wherein the face recognition result is at least used to locate the face position in the video.

[0064] S3, based on the face recognition result, detect whether someone is looking at the display screen, specifically:

[0065] Detecting whether someone is looking at the display screen based on the face recognition result can be achieved in the following way: when the face recognition result indicates that face information is detected, it proves that someone is looking at the display screen; when the face recognition result indicates that face information is not detected, it proves that no one is looking at the display screen.

[0066] Detecting whether someone is looking at the display screen based on the face recognition result can also be achieved in the following way: when the face recognition result indicates that face information is detected, further based on the face information, use computer vision technologies and algorithms such as face recognition and eye tracking algorithms to detect the visual direction; if the visual direction is within a preset area, it proves that someone is looking at the display screen; if the visual direction is not within the preset area, it proves that no one is looking at the display screen; when the face recognition result indicates that no face information is detected, it is directly determined that no one is looking at the display screen.

[0067] S4, according to whether someone is watching the display screen, determine the display screen playback processing strategy, including:

[0068] S41, if someone is watching the display screen, the display screen is controlled to play the video normally;

[0069] S42, if no one is looking at the display screen, the display screen can be kept still;

[0070] S43, it is also possible to detect the cumulative time when no one is looking at the display screen, and control the display screen to enter a dormant state when the cumulative time reaches a first preset time; and control the display screen to shut down when the cumulative time reaches a second preset time.

[0071] S5, if someone is watching the display screen, the age information of the face in the video can be further detected based on the face recognition result, and the appropriate video can be selected for switching and playing according to the age information, including:

[0072] S51, deep learning technology such as convolutional neural network (CNN) can be used to extract features of the detected face, and key features of the face, such as facial contour, eye position, lip shape, etc. can be extracted;

[0073] S52, determining the age range of the face according to the extracted key features of the face;

[0074] S53, selecting a suitable video for switching and playing according to the age information.

[0075] When a video includes multiple faces, it is necessary to determine the age range corresponding to each face; and select the video corresponding to the age range with the largest number of people for playback.

[0076] S6, if someone is watching the display screen, it is also possible to further detect whether the person in the video is the target object based on the face recognition result, and if so, select the video preferred by the target object for playback, specifically including:

[0077] S61, deep learning technology (such as convolutional neural network (CNN)) can be used to extract features of the detected face, and extract key features of the face, such as facial contour, eye position, lip shape, etc.

[0078] S62, matching the extracted facial features with an existing face database to determine whether the recognized face is a target object recorded in the face database. The matching process can be performed using, but is not limited to, a feature vector similarity calculation method, such as Euclidean distance, cosine similarity, and the like.

[0079] S63: If the recognized face is matched to be the target object, a video preferred by the target object is selected for playback.

[0080] In this embodiment, a camera is installed on the display screen to shoot a predetermined area, and the video is identified through the captured video. The video playback processing strategy is determined according to the identification result. For example, when a face is identified, that is, it is proved that the person is looking at the screen, the video can be played normally. If it cannot be proved that the person is looking at the screen, the screen can be frozen. The above method reduces resource waste while improving the flexibility of video playback, thereby improving the user experience.

[0081] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0082] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0083] The embodiment of the present application also provides a video playback processing device. It should be noted that the video playback processing device of the embodiment of the present application can be used to execute the video playback processing method provided by the embodiment of the present application. The video playback processing device provided by the embodiment of the present application is introduced below.

[0084] Figure 3 is a schematic diagram of a video playback processing device according to an embodiment of the present application. Figure 3 As shown, the device includes: an acquisition module 700, a detection module 702 and a play strategy determination module 704, wherein:

[0085] An acquisition module 700 is used to acquire a face image in a predetermined area of ​​a display screen;

[0086] A detection module 702, connected to the acquisition module 700, is used to detect the visual direction of the object included in the face image and obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen and the cumulative time length of the visual direction not toward the display screen;

[0087] The play strategy determination module 704 is connected to the detection module 702 and is used to determine the video play strategy according to the visual detection result.

[0088] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0089] It should be noted that the acquisition module 700, the detection module 702 and the playback strategy determination module 704 correspond to steps S101 to S103 in the embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the embodiment. It should be noted that the modules as part of the device can be run in a computer terminal.

[0090] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0091] The video playback processing device includes a processor and a memory. The above units are stored in the memory as program units, and the processor executes the above program units stored in the memory to implement corresponding functions.

[0092] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the video playback process is performed by adjusting the kernel parameters.

[0093] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0094] An embodiment of the present application provides a non-volatile storage medium on which a program is stored, and when the program is executed by a processor, the above-mentioned video playback processing method is implemented.

[0095] An embodiment of the present application provides a processor, which is used to run a program, wherein the video playback processing method is executed when the program is running.

[0096] like Figure 4 As shown, an embodiment of the present application provides an electronic device, the electronic device 10 includes a processor, a memory, and a program stored in the memory and can be run on the processor, and the processor implements the following steps when executing the program: obtaining a face image in a predetermined area of ​​the display screen; detecting the visual direction of the object included in the face image, and obtaining a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen, and the cumulative time that the visual direction is not toward the display screen; determining a video playback strategy according to the visual detection result. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0097] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: acquiring a face image within a predetermined area of ​​a display screen; detecting the visual direction of an object included in the face image to obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen and the cumulative length of time that the visual direction is not toward the display screen; and determining a video playback strategy based on the visual detection result.

[0098] Optionally, the above-mentioned computer program product is also suitable for executing an initialized program having the following method steps: when the visual detection result indicates that the visual direction is toward the display screen, determining the first video playback strategy as: controlling the display screen to play the video content; or when the visual detection result indicates that the visual direction is not toward the display screen, determining the second video playback strategy as: controlling the display screen to stop playing the video, and based on the accumulated duration, determining the stopping method of controlling the display screen to stop playing the video.

[0099] Optionally, the above-mentioned computer program product is also suitable for executing an initialization program having the following method steps: when the accumulated time is less than or equal to a preset second time, determining the first stop mode as: controlling the display screen to display a still picture, wherein the preset second time is greater than or equal to the preset first time; or when the accumulated time is greater than the preset second time and less than or equal to a preset third time, determining the second stop mode as: controlling the display screen to enter a sleep state, wherein the preset third time is greater than the preset second time; or when the accumulated time is greater than the preset third time, determining the third stop mode as: controlling the display screen to shut down.

[0100] Optionally, the above-mentioned computer program product is also suitable for executing an initialized program having the following method steps: determining the object type corresponding to the object included in the face image; detecting whether the current video content played on the display screen matches the object type; when the current video content does not match the object type, determining the target video content that matches the object type; and controlling the current video content played on the display screen to be switched to the target video content.

[0101] Optionally, the above-mentioned computer program product is also suitable for executing an initialized program having the following method steps: based on the object types corresponding to N objects, M object types and the number of objects corresponding to the M object types are obtained, wherein M is an integer greater than or equal to 1; and the video content corresponding to the object type with the largest number of objects among the M object types is determined as the target video content.

[0102] Optionally, the above-mentioned computer program product is also suitable for executing an initialized program having the following method steps: when the visual detection result indicates that the visual direction is toward the display screen, detecting whether the object whose visual direction is toward the display screen is the target object; when the object whose visual direction is toward the display screen is the target object, detecting whether the current video content played on the display screen is the preferred video content of the target object; when the current video content is not the preferred video content, controlling the current video content played on the display screen to be switched to the preferred video content.

[0103] Optionally, the above-mentioned computer program product is also suitable for executing an initialized program having the following method steps: matching the facial image information of the object whose visual direction is toward the display screen with the facial image information pre-stored in the target database, and detecting whether the facial image information of the object whose visual direction is toward the display screen is in the target database; if the facial image information of the object whose visual direction is toward the display screen is in the target database, determining that the object whose visual direction is toward the display screen is the target object.

[0104] Optionally, the above-mentioned computer program product is also suitable for executing an initialized program having the following method steps: acquiring image information within a predetermined area through K image acquisition devices, wherein K is an integer greater than or equal to 2, and the K image acquisition devices correspond to different image acquisition areas; identifying facial image information in the image information to obtain a face image within the predetermined area.

[0105] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0106] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0110] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0111] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0114] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A video playback processing method, characterized in that: include: Acquire a face image in a predetermined area of ​​the display screen; Detecting the visual direction of the object included in the face image to obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is a direction toward the display screen, and a cumulative duration during which the visual direction is not a direction toward the display screen; A video playback strategy is determined according to the visual detection result.

2. The method according to claim 1, characterized in that The determining of the video playback strategy according to the visual detection result includes: When the visual detection result indicates that the visual direction is a direction toward the display screen, determining the first video playback strategy is: controlling the display screen to play video content; or When the visual detection result indicates that the visual direction is not towards the display screen, the second video playback strategy is determined as: controlling the display screen to stop playing the video, and based on the accumulated duration, determining a stop method for controlling the display screen to stop playing the video.

3. The method according to claim 2, characterized in that Based on the accumulated time, determining a stop method for controlling the display screen to stop playing the video includes: When the accumulated duration is less than or equal to the preset second duration, determining the first stop mode is: controlling the display screen to display a still picture, wherein the preset second duration is greater than or equal to the preset first duration; or When the accumulated time is greater than the preset second time and less than or equal to the preset third time, determining the second stop mode is: controlling the display screen to enter a sleep state, wherein the preset third time is greater than the preset second time; or When the accumulated time is greater than a preset third time, the third stopping mode is determined as: controlling the display screen to stop.

4. The method according to claim 2, characterized in that: When the visual detection result indicates that the visual direction is toward the display screen, after determining that the first video playback strategy is: controlling the display screen to play video content, the method further includes: Determining an object type corresponding to an object included in the face image; Detecting whether the current video content played on the display screen matches the object type; In a case where the current video content does not match the object type, determining target video content that matches the object type; Control the current video content played on the display screen to switch to the target video content.

5. The method according to claim 4, characterized in that When the visual directions of N objects among the objects included in the face image are in the direction toward the display screen, and N is greater than or equal to 2, determining the target video content matching the object type includes: Based on the object types corresponding to the N objects, M object types and the number of objects corresponding to the M object types are obtained, where M is an integer greater than or equal to 1; The video content corresponding to the object type with the largest number of objects among the M object types is determined as the target video content.

6. The method according to claim 1, characterized in that The method further comprises: In a case where the visual detection result indicates that the visual direction is a direction toward the display screen, detecting whether the object whose visual direction is a direction toward the display screen is a target object; In a case where the object whose visual direction is toward the display screen is the target object, detecting whether the current video content played on the display screen is the preferred video content of the target object; In a case where the current video content is not the preferred video content, the current video content played on the display screen is controlled to be switched to the preferred video content.

7. The method according to claim 6, characterized in that The detecting whether the object whose visual direction is toward the display screen is a target object includes: Matching the facial image information of the object whose visual direction is toward the display screen with the facial image information pre-stored in a target database, and detecting whether the facial image information of the object whose visual direction is toward the display screen is in the target database; In a case where facial image information of an object whose visual direction is toward the display screen is in the target database, the object whose visual direction is toward the display screen is determined as the target object.

8. The method according to any one of claims 1 to 7, characterized in that The step of acquiring a face image within a predetermined area of ​​the display screen comprises: Collecting image information within the predetermined area by using K image acquisition devices, wherein K is an integer greater than or equal to 2, and the K image acquisition devices correspond to different image acquisition areas; The facial image information in the image information is identified to obtain the facial image in the predetermined area.

9. A video playback processing device, characterized in that: include: An acquisition module, used to acquire a face image in a predetermined area of ​​the display screen; a detection module, configured to detect a visual direction of an object included in the face image, and obtain a visual detection result, wherein the visual detection result is used to indicate whether the visual direction is toward the display screen, and a cumulative duration during which the visual direction is not toward the display screen; The playback strategy determination module is used to determine the video playback strategy according to the visual detection result.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the video playback processing method described in any one of claims 1 to 8.

Images