Control method and control device of advertisement delivery device, and advertisement delivery device

Abstract

The application provides an advertisement putting device control method, an advertisement putting device control device and an advertisement putting device, relates to the technical field of advertisement putting, and the advertisement putting device control method comprises the following steps: acquiring first image data shot by a camera device; identifying face information in the first image data, determining an expression and a head posture of a user, and generating corresponding push information; and displaying the push information in a corresponding display screen. The application acquires the first image data shot by the camera device, identifies the face information in the first image data, determines the expression and the head posture of the user, generates the push information corresponding to the expression and the head posture of the user, and finally displays the push information in the corresponding display screen, so that the function of dynamically adjusting advertisement putting content according to real-time reactions of users is realized, user experience is better optimized, and the conversion efficiency of advertisements is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of advertisement delivery technology, and particularly relates to a control method and device of an advertisement delivery device and the advertisement delivery device. BACKGROUND

[0002] In the prior art, an advertisement delivery device can usually only deliver advertisements according to a preset advertisement play list, which leads to an unsatisfactory advertisement delivery effect and cannot maximize the attention of the audience and improve the advertisement conversion rate. SUMMARY

[0003] The main purpose of the present application is to provide a control method of an advertisement delivery device, which aims to adjust the advertisement delivery strategy in real time, thereby improving the pertinence and effect of advertisement delivery.

[0004] To achieve the above purpose, the present application provides a control method of an advertisement delivery device, the advertisement delivery device comprising a camera device and a display device, and the control method comprising:

[0005] acquiring first image data shot by the camera device;

[0006] recognizing face information in the first image data, determining the expression and head posture of a user, and generating corresponding push information;

[0007] displaying the push information in a corresponding display screen.

[0008] Optionally, the recognizing face information in the first image data, determining the expression and head posture of a user, and generating corresponding push information comprises:

[0009] reading each frame of picture in the first image data, and scaling each frame of picture according to a preset resolution to obtain first picture data;

[0010] detecting face shape information in the first picture data to obtain first recognition data;

[0011] adding a circumscribed graph to the face in the first recognition data, and assigning an identity to each added circumscribed graph;

[0012] cropping the first recognition data based on the circumscribed graph to obtain at least one first face image;

[0013] recognizing key feature points of the first face image, and determining expression information of the face based on the key feature points;

[0014] obtaining head posture and body action information of the user based on the key feature points;

[0015] Based on the facial expression information, the user's head posture, and body movement information, an audience interest score is generated to determine the push notification information.

[0016] Optionally, identifying key feature points of the first facial image and determining facial expression information based on the key feature points includes:

[0017] The key feature points of the first face image are identified based on the key point detection algorithm. The key feature points include at least one of the user's eye position, nose position, ear position, and mouth corner position.

[0018] The image parameters of the first face image are adjusted based on the key feature points to obtain the second face image;

[0019] The second facial image is processed based on a preset model algorithm to generate facial expression information.

[0020] Optionally, generating an audience interest score based on the facial expression information, the user's head posture, and body movement information specifically involves:

[0021] Based on the user's head posture, determine the angle difference between the user's head and the display screen to determine the user's attention level to the display screen;

[0022] When the level of attention is high, the user's current level of interest is determined based on the facial expression information and the body movement information to generate the audience interest score.

[0023] Optionally, the control method for the advertising delivery device further includes:

[0024] When the push information is displayed on the screen, acquire the second image data;

[0025] Identify changes in facial expressions and the duration of the user's gaze in the second image;

[0026] Based on the changes in facial expressions and the duration of gaze, the user's level of interest is determined, and feedback information corresponding to the level of interest is generated.

[0027] The push notifications are adjusted based on the feedback information.

[0028] Optionally, displaying the push information on the corresponding display screen includes:

[0029] The system renders or plays corresponding advertising videos / images in real time and displays corresponding interactive elements on the display screen, including buttons, QR codes, or links.

[0030] Optionally, generating the corresponding push information specifically involves:

[0031] obtaining a plurality of advertising materials;

[0032] tagging according to the category to which the advertising material belongs to generate an advertising material set;

[0033] According to the face information, the age and gender corresponding to the user are identified to determine the advertising interest degree of the user to the advertising material set;

[0034] According to the advertising interest degree, each advertising material set is scored based on a deep recommendation algorithm, and at least one advertising material set with the highest score is selected for delivery.

[0035] Optionally, in the case where the face information identified by the first image data is at least two, the control method of the advertising delivery device further comprises:

[0036] Obtaining the historical stay time and the historical stay times of each face information in front of the display screen, and determining the interest weight corresponding to the historical stay time and the historical stay times;

[0037] The interest weights of each face information are sorted, and the screen of the display screen is divided according to the sorting of the interest weights to generate a plurality of display screens, and the size of the plurality of display screens is determined according to the sorting of the interest weights;

[0038] Continuously monitor the face information identified by the first image data, and after re-determining the interest weight of each face information, adjust the push information to generate a new plurality of display screens.

[0039] In addition, in order to achieve the above-mentioned purpose, the present application also provides a control device, which comprises a memory, a processor and an advertising delivery device control program stored in the memory and executable on the processor, and the advertising delivery device control program is configured to implement the advertising delivery device control method as described above.

[0040] In addition, in order to achieve the above-mentioned purpose, the present application also provides an advertising delivery device, which comprises the control device as described above.

[0041] The embodiment of the present application obtains the first image data shot by the camera device, identifies the face information in the first image data, determines the expression and head posture of the user, generates the push information corresponding to the expression and head posture of the user, and finally displays the push information to the corresponding display screen, thereby realizing the function of dynamically adjusting the advertising delivery content according to the real-time reaction of the user, which not only can better optimize the user experience, but also can effectively improve the conversion efficiency of the advertising. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, further serve to explain the principles of the application.

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0044] Figure 1 is a control method flowchart of an advertising delivery device according to an embodiment of the present application;

[0045] Figure 2 is a control method flowchart of an advertising delivery device according to another embodiment of the present application;

[0046] Figure 3 is a control method flowchart of an advertising delivery device according to yet another embodiment of the present application;

[0047] Figure 4 is a control method flowchart of an advertising delivery device according to still another embodiment of the present application;

[0048] Figure 5 is a control method flowchart of an advertising delivery device according to yet another embodiment of the present application;

[0049] Figure 6 is a control method flowchart of an advertising delivery device according to another embodiment of the present application;

[0050] Figure 7 is a control method flowchart of an advertising delivery device according to yet another embodiment of the present application;

[0051] Figure 8 is a control method flowchart of an advertising delivery device according to still another embodiment of the present application.

[0052] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0053] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. The well-known modules, units, connections, links, communications or operations between them are not shown or described in detail. Furthermore, the described features, architectures or functions can be combined in any way in one or more embodiments. Those skilled in the art should understand that the following various embodiments are only used for illustration, rather than limiting the protection scope of the present application. It can also be easily understood that the modules or units or processing manners in the embodiments described herein and shown in the drawings can be combined and designed in various different configurations. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0054] In the following embodiments, the definition of various nouns or methods is generally based on the broad concept that can be implemented on the premise of the disclosed content in the embodiments, except for the cases that are logically impossible. Under such understanding, various specific sub-limits of the nouns or methods should be regarded as the invention content of the present application, and should not be regarded as the specific limit not disclosed in the specification, or be interpreted in a narrow sense or biased. Similarly, the order of steps in the method is flexible under the premise of logical implementation. The specific sub-limits of the broad concept of various nouns or methods are within the protection scope of the present application.

[0055] Since the advertisement delivery device in the prior art can usually only deliver advertisements according to a preset advertisement play list, the advertisement delivery effect is often not ideal, and cannot maximize the attention of the audience and improve the advertisement conversion rate.

[0056] The main solution of the embodiment of the present application is that the embodiment of the present application acquires first image data shot by a camera device, identifies the face information in the first image data, determines the expression and head posture of the user, generates push information corresponding to the expression and head posture of the user, and finally displays the push information to the corresponding display screen.

[0057] In the embodiment, the control device is taken as the execution subject for description.

[0058] The present application provides a solution, thereby realizing the function of dynamically adjusting the advertisement delivery content according to the real-time reaction of the user, which not only can better optimize the user experience, but also can effectively improve the conversion efficiency of the advertisement.

[0059] To this end, the application provides a control method of an advertisement delivery device; it can be understood that the advertisement delivery device is provided with a control device for storing and executing the following method, and the control device can be realized by a main controller such as MCU (Micro controller Unit), DSP (Digital Signal Process), FPGA (Field Programmable Gate Array), SOC (System On Chip) and the like.

[0060] It should be understood that the advertisement delivery device comprises a camera device and a display device, the camera device can be mounted on the display device or independently mounted with the display device, the camera device can adopt a high-definition camera with a wide-angle or fisheye lens to realize a large range coverage of a crowded area, the camera can be an infrared camera, a visible light camera or a depth camera and the like, which can capture image data in the surrounding environment in real time. The processing of the image data can use an edge device with GPU or NPU (such as Jetson series, Coral TPU) to realize real-time image processing and inference. In the scene where inference is time-consuming (especially multi-person detection), the optimization adjustment can be performed by constructing a cache queue, and each frame of data to be processed is managed by a FIFO queue to ensure real-time or near real-time (such as 10-20 fps). In the processing of the image data, the control device of the advertisement delivery device can dynamically adjust the detection frame rate or detection interval according to the load of the current computing resource, maintain a high fluency on the premise of ensuring the detection accuracy. The display device can be an LCD display screen, an LED display screen or other types of display screens, which is used to display the advertisement delivery content.

[0061] Based on the above hardware, referring to Figure 1 In an embodiment of the application, the control method of the advertisement delivery device comprises steps S100-S300, wherein:

[0062] S100, acquiring first image data photographed by the camera device;

[0063] S200, identifying face information in the first image data, determining the expression and head posture of the user to generate corresponding push information;

[0064] S300, displaying the push information to the corresponding display screen.

[0065] The first image data can include basic attribute information of the user such as gender, age range, facial features and the like, and environment information such as light conditions, background content and the like.

[0066] The facial information in the first image data can be extracted and analyzed through advanced facial recognition algorithms based on neural networks, key point extraction, and facial recognition technology. This algorithm can efficiently identify facial features from complex image data and accurately classify user expressions and head poses based on deep learning technology. For example, the algorithm can identify user expressions such as smiling, surprise, anger, and head poses such as head orientation and tilt angle.

[0067] The push information can include advertising content, coupons, product recommendations, etc., which are personalized according to the user's expressions and head poses. For example, when the user shows a smile or a positive expression, positive and encouraging advertising content can be pushed; when the user shows a surprised or curious expression, novel and interesting product recommendations can be pushed. At the same time, the push information can also be combined with basic attribute information of the user, such as gender, age range, etc., and environmental information, such as light conditions, background content, etc., for more accurate push. Such personalized push strategy not only improves the effectiveness of advertising, but also enhances the user experience and enhances the interaction between the user and the advertising device.

[0068] In actual application, assuming that the advertising device can be set in a supermarket, there can be integrated advertising screens in multiple locations in the supermarket, and the advertising screens have a camera and a display device, the display device includes a display screen, when a user approaches the advertising screen, the advertising screen will play the initial advertising screen, the camera can take pictures to obtain the first image data of the user, i.e. the image screen in the range where the user is located. Subsequently, the control device of the advertising device will quickly process the first image data, extract and analyze the facial information through the facial recognition algorithm, and determine the user's expression and head pose. Based on this information, the control device will generate push information that matches the current emotional state of the user and display it to the user in real time through the display screen. For example, if the control device identifies that the user is smiling, it may display some light and happy advertising content, such as travel, food recommendations, etc.; if the user shows a curious or exploratory pose, the control device may push some novel and innovative product information to stimulate the user's interest. This personalized push strategy based on the user's real-time emotional state not only improves the accuracy and effectiveness of advertising, but also makes the user's shopping experience in the supermarket more enjoyable and interactive.

[0069] In addition, in order to improve privacy security protection, the image data captured by the camera is desensitized, and the image data is only stored in the memory / short-term cache using the temporary storage function, the face features (such as 68 key point coordinates, expression vectors) detected are converted into code vectors, and a one-time hash (salted hash) is performed, so that the user's privacy data will not be leaked. In addition, the original frame buffer and temporary data can be automatically deleted within a preset time period (such as 30 seconds to 1 minute, which can be adjusted according to actual needs) after the advertisement delivery decision is completed, to prevent secondary use. Only anonymized statistical information (such as "how many audience show happy emotions in a certain period") is retained in the control device for subsequent data analysis.

[0070] In addition, in order to facilitate linkage between multiple advertising screens in the same business circle, or for users to more accurately promote when shopping in related cooperative business circles, the same or comparable classification standard can be used for expression recognition or interest classification in each mall / each screen, and the feature vector of the first image data of the user is hashed to obtain an anonymous ID such as UserHashA123, which prohibits reverse inference to the real identity. Data flow between multiple advertising screens, or between two business circles, when a new feature vector is recognized by the camera in business circle B, the anonymous ID feature in the local cache or cloud shared library can be compared to achieve approximate similarity matching (such as cosine similarity ≥ threshold). If the matching is successful and the label can be normally corresponded, the corresponding label advertisement is delivered, for example, "sports" is the main interest of the anonymous ID, B can immediately deliver sports equipment advertisements. In addition, an access control and security audit mechanism needs to be established, wherein only authorized users / malls are allowed to use the anonymous ID for remarketing, and transmission layer encryption (such as TLS / SSL) is used, and the feature vector is encrypted twice to avoid packet capture or data leakage.

[0071] In addition, the advertisement delivery device can also include a microphone array, which focuses on the speaker close to the screen through beamforming technology, reduces environmental noise interference, and uses WebRTC VAD or deep learning-based VAD model to extract effective voice segments from continuous audio.

[0072] The VAD or deep learning-based VAD model using WebRTC extracts the valid speech segment from the continuous audio, including: pre-processing the audio data through the VAD model, removing the silent segment and background noise, and only retaining the valid speech segment containing the user's voice. In this way, the control device can more accurately capture the user's voice instructions or feedback, improving the interaction experience between the advertising device and the user. For example, the user can request to display a specific type of advertisement through voice instructions, or like, comment, and other operations on the displayed advertisement. After receiving these voice instructions, the advertising device can quickly respond and adjust the push strategy to meet the user's personalized needs. At the same time, this voice-based interaction method also makes the advertising device more intelligent and convenient, improving the user's shopping experience in the supermarket and other scenarios.

[0073] Among them, the speech can be converted into text using deep end-to-end speech recognition (CTC, RNN-T or Transformer ASR). If only the sentiment is concerned, a CNN+RNN can be used to analyze Mel spectrum and other features to output sentiment categories such as "positive / negative / neutral".

[0074] In addition, text keyword extraction can be performed by using an NLP framework, such as jieba segmentation (Chinese), and then combined with named entity recognition (NER) or TF-IDF / TextRank to extract high-frequency keywords related to the product. The semantics expressed by the user, such as "I want to buy a phone case", can be identified as the core intent "looking for a phone case". Combined with multi-modal fusion logic, the user's expression and head pose (vision) and voice sentiment (auditory) are combined. If both are "positive", the current interest score of the user is increased. If the language expression is "like" but the expression recognition is "hate", the past historical data needs to be used to determine which signal is more reliable, or a neutral evaluation is given.

[0075] Among them, the collected anonymized voice text and expression feedback can also be continuously trained in the cloud to enable the system to more accurately identify the intent, and the effect of multi-modal fusion is measured according to the actual interaction or purchase conversion result (such as average stay time, bounce rate, etc.), to realize closed-loop evaluation.

[0076] The embodiment acquires the first image data shot by the camera device, identifies the face information in the first image data, determines the expression and head pose of the user, generates the push information corresponding to the expression and head pose of the user, and finally displays the push information to the corresponding display screen, thereby realizing the function of dynamically adjusting the advertising content according to the real-time reaction of the user, which not only optimizes the user experience, but also effectively improves the conversion efficiency of the advertisement.

[0077] Optionally, with reference to Figure 2 Another embodiment of the present application provides a control method of an advertisement delivery device, based on the above Figure 1 As shown in the embodiment, the face information in the first image data is identified, the expression and head posture of the user are determined, and the corresponding push information is generated, including steps S2010-S2070.

[0078] S2010, read each frame of picture in the first image data, and scale each frame of picture according to a preset resolution to obtain first picture data;

[0079] S2020, detect the face shape information in the first picture data to obtain first identification data;

[0080] S2030, add a circumscribed graph to the face in the first identification data, and assign an identity to each added circumscribed graph;

[0081] S2040, crop the first identification data based on the circumscribed graph to obtain at least one first face image;

[0082] S2050, identify the key feature points of the first face image, and determine the expression information of the face based on the key feature points;

[0083] S2060, obtain the head posture and body action information of the user based on the key feature points;

[0084] S2070, generate an audience interest score according to the expression information, the head posture and the body action information of the user to determine the push information.

[0085] The first image data can be a video picture, in which case each frame of picture represents a short time point. By processing these frames of picture, the dynamic expression and action of the user can be captured. Scaling the picture according to a preset resolution helps to reduce the data processing amount, improve the processing speed, and at the same time maintain sufficient details for accurate face recognition.

[0086] Detecting face shape information such as the contour of the face and the position of the facial features is a very critical step in the identification process. A “single-stage detection” model such as YOLOv5, YOLOv8, SSD or CenterNet can be used to detect all visible faces or bodies in the current frame to determine whether the face in the picture meets the expected features, thereby obtaining the first identification data.

[0087] To distinguish different faces in the picture, the (x, y, w, h) and confidence information are obtained through the above steps, and multiple target boxes are output. The target box is used to add an external graphic to the recognized face and assign a unique identity. In this way, even if multiple faces appear in the same picture, each individual can be accurately tracked and identified.

[0088] Again, using the Tracking algorithm, on the basis of detection, using Deep SORT or ByteTrack multi-target tracking algorithm, a "Tracking ID" is assigned to each target. In subsequent frames, the consistency of the target ID is maintained through appearance features (ReID feature vectors) or motion trajectories (Kalman filter).

[0089] For the detected face target box, the first face image obtained by cropping is a pure face picture without background and other interference factors, which helps to more accurately identify the key feature points of the face. These feature points include eye corners, mouth corners, nose tips, etc., and their positions and shapes can reflect the facial expression information.

[0090] In addition to expression information, key feature points can also be used to obtain user head posture and body movement information. For example, by calculating the relative position changes of feature points, it can be inferred whether the user's head is tilted, turned, and whether there are hand waving, nodding, and other body movements.

[0091] Finally, according to the expression information, head posture and body movement information, a viewer interest score can be generated. This viewer interest score can reflect the user's attention and interest in the current advertising content, serving as an important basis for generating push information. By considering these factors comprehensively, more accurate and personalized push information can be generated, improving the effectiveness of advertising and user experience.

[0092] Among them, according to the expression information, such as a smile, the head posture is directly facing the screen, and there is no obvious intention to leave the body movement, a higher interest score is given; on the contrary, if the expression is frowning or disgusted, the head posture deviates from the screen, or the body movement has a tendency to leave, a lower interest score is given. In this way, the control device can evaluate the user's reaction to the advertising content in real time and adjust the push strategy in a timely manner to ensure that the advertising content can attract and maintain the user's attention.

[0093] When generating push information, in addition to considering the viewer interest score, historical behavior data, consumption preferences, and other information of the user can also be considered to build a more detailed user portrait. In this way, push information can not only be dynamically adjusted based on the user's real-time reaction, but also be personalized according to the user's long-term habits and interests, improving the accuracy and effectiveness of advertising.

[0094] Optionally, referring to Figure 3 Another embodiment of the present application provides a control method of an advertisement delivery device, based on the above Figure 2 In the embodiment shown, the step of identifying the key feature points of the first facial image based on the key point detection algorithm, and determining the expression information of the face based on the key feature points includes steps S251-S253, wherein:

[0095] S251, identifying the key feature points of the first facial image based on a key point detection algorithm, the key feature points including at least one of the user's eye position, nose position, ear position, and mouth corner position;

[0096] S252, and adjusting the picture parameters of the first facial image based on the key feature points to obtain a second facial image;

[0097] S253, processing the second facial image based on a preset model algorithm to generate the expression information of the face.

[0098] The key point detection algorithm can be an algorithm such as Dlib, MTCNN, MediaPipe Face Mesh, or Face++, which can accurately locate the key feature points of the face, such as the eye corner, mouth corner, and nose tip, in a complex background, and perform geometric alignment on the cropped face to reduce the influence of posture. The position information of these feature points is crucial for subsequent expression recognition.

[0099] Adjusting the picture parameters, such as contrast and brightness, or Gamma correction, is to optimize the quality of the first facial image, reduce the interference of light changes on expression recognition, and ensure that more clear and accurate image information can be obtained when performing expression recognition. Through appropriate parameter adjustment, the influence of uneven lighting, shadows, and other problems on recognition effect can be reduced.

[0100] The preset model algorithm can be a convolutional neural network (CNN) model based on deep learning technology, such as ResNet, MobileNet, etc., or a structure that integrates RNN (for the time sequence changes of expressions) or Transformer (stronger feature expression), which can automatically extract features related to expressions by learning a large number of labeled expression samples, and achieve accurate classification of different expressions. In the model training process, cross-validation, data enhancement, and other techniques are used to improve the generalization ability and robustness of the model.

[0101] The preset model algorithm is pre-trained on a public expression dataset (FER2013, AffectNet), and then fine-tuned according to actual scene data (such as small-scale data collected and labeled by oneself). The process of the preset model algorithm processing the second face image can be that the second face image is input into the preset model algorithm, features are extracted through multi-layer convolution operation and pooling operation, the features are mapped to an expression category space through a full connection layer, and finally a Softmax function is used to output the probability distribution of each expression category, and the expression category with the maximum probability is selected as the final expression recognition result. By continuously optimizing the structure and parameters of the preset model algorithm and expanding the diversity and scale of the training samples, the accuracy of expression recognition can be improved. The facial expression information can include 'joy','surprise', 'neutral', 'anger', 'disgust', 'fear', etc., and can also be self-defined to be more simplified or more subdivided.

[0102] In actual application, when the camera of the advertisement delivery device captures the face picture of the user, the key feature points of the face are first extracted through the key point detection algorithm, and the picture parameters are adjusted. Then, the processed second face image is input into the preset model algorithm to generate expression information. This process is fast and efficient, and can complete the recognition and classification of the user's expression in a very short time, improve the accuracy and effect of advertisement delivery, and also bring a more pleasant and interactive shopping experience to the user.

[0103] Optionally, referring to Figure 4 The present application further provides a control method of an advertisement delivery device, based on the above-mentioned Figure 2 According to the expression information, the head posture of the user and the body action information, the audience interest score is generated, specifically steps S271-S272.

[0104] S271, according to the head posture of the user, the angle difference between the head of the user and the display screen is determined to determine the attention degree of the user to the display screen;

[0105] S272, in the case that the attention degree is high, according to the expression information and the body action information, the current interest level of the user is determined to generate the audience interest score.

[0106] The user's head posture can include various actions such as tilting, turning, nodding, or shaking. For example, when the user's head is oriented towards the display screen and slightly tilted, it may indicate that the user is interested in the current content being displayed; while when the user's head is significantly deviated from the display screen or rapidly turned, it may mean that the user's attention has shifted. Nodding action is usually interpreted as agreement or interest, while shaking may indicate dissatisfaction or rejection. By accurately capturing and analyzing these head posture information, the advertising device can more accurately judge the user's attention and interest level, thereby adjusting the push strategy and providing more user demand-oriented advertising content.

[0107] wherein whether the screen is being looked at is inferred using facial key point detection or 3D head pose estimation; if the angle deviation is too large, it is determined that the attention is low, and in the case of low attention, the user does not need to be judged further; while if the user's attention is high, it is necessary to further judge whether the audience has interest actions such as staying or pointing, which can be detected by OpenPose or MediaPipe Body.

[0108] wherein in the case of high attention, that is, when the angle difference between the user's head and the display screen is small, for example, within the range of 0-30 degrees, the user's line of sight is likely to be directly opposite the display screen, indicating that the user has a high attention to the current content being displayed. In this case, the advertising device will determine the user's current interest level according to the user's expression information and body movement information. For example, when the user exhibits positive expressions such as smiling, bright eyes, and relaxed body movements without obvious intention to leave, it can be considered that the user has a high interest level in the current advertising content, and a high audience interest score will be given. On the contrary, if the user shows negative expressions such as frowning, wandering eyes, or body movements with a tendency to leave, such as standing up and leaving, turning around, etc., it may mean that the user is not interested in the current advertising content, and a low audience interest score will be given.

[0109] wherein the identified expression information, such as expression emotion, and body movement information, i.e. posture results, are combined and scored, such as "pleasure" plus, "neutral" medium, "disgust" minus, combined with "whether the screen is being looked at" to determine the current interest level of the audience. Its calculation can be realized according to the following formula:

[0110] InterestScore i =α⋅(FacialEmotionScore)+β⋅(FocusIndicator)+γ⋅(HistoricalWatchTime)\text{InterestScore}_i

[0111] = a • (Facial Emotion Score) + β • (Focus Indicator) + γ • (Historical Watch Time) Interest Scorei

[0112] = a • (Facial Emotion Score) + β • (Focus Indicator) + γ • (Historical Watch Time)

[0113] wherein a, β, γ are adjustable weights, Facial Emotion Score is the facial emotion score, Focus Indicator is the attention indicator, and Historical Watch Time is the historical watch time.

[0114] In this way, the advertisement delivery device can capture and analyze the user's head posture, facial expression information and body movement information in real time, thereby generating an accurate audience interest score. This score not only reflects the user's attention and interest level for the current advertising content, but also serves as an important basis for adjusting the push strategy. For example, when the audience interest score is high, the advertisement delivery device can continuously display the current advertising content or push more advertising related to the user's interests; when the audience interest score is low, the advertising content can be switched in time or some novel and interesting advertisements that can attract the user's attention can be displayed to improve the effectiveness of advertising and user experience.

[0115] Optionally, with reference to Figure 5 , the present application also provides a control method of an advertisement delivery device, based on the above Figure 1 embodiments, the control method of the advertisement delivery device further includes steps S400-S700, wherein:

[0116] S400, in the case where the push information is displayed on the display screen, acquiring second image data;

[0117] S500, recognizing the facial expression change and the user's gaze duration in the second image;

[0118] S600, determining the user's interest degree according to the facial expression change and the gaze duration, to generate feedback information corresponding to the interest degree;

[0119] S700, adjusting the push information according to the feedback information.

[0120] The second image data can include subsequent behavior data of the user, such as the user's dwell time after watching the advertisement, whether a click or interactive operation is performed, and the change in physical distance between the user and the advertisement content. These second image data can refine the user's reaction to the advertisement content, thereby generating more accurate feedback information. For example, if the user stays for a long time after watching the advertisement and performs multiple interactive operations such as zooming in, zooming out, or rotating the advertisement screen, it may indicate that the user has a strong interest in the advertisement content. Conversely, if the user leaves immediately after the advertisement is displayed or shows obvious avoidance behavior such as quickly turning around or moving away from the advertisement screen, it may mean that the advertisement content does not interest the user. In addition, by analyzing the change in physical distance between the user and the advertisement content, the user's proximity or avoidance to the advertisement can also be inferred, thereby serving as a reference basis for adjusting the push strategy.

[0121] Further, the expression change in the second image and the user's gaze duration are identified, which can refine the user's emotional reaction to the advertisement content. For example, the user's smile may gradually deepen, indicating love and recognition of the advertisement content; or the user's gaze may change from bright to free, indicating that the user's interest in the advertisement content gradually weakens. In addition, the user's gaze duration is also an important indicator, which reflects the user's attention to the advertisement content and the time invested. A longer gaze duration may mean that the user has a strong interest in the advertisement content, while a shorter gaze duration may indicate that the user is not interested in the advertisement content or has lost patience.

[0122] By integrating these information, during the playing of the advertisement, the change in the audience's expression is continuously detected to determine whether to maintain happiness or generate fatigue / boredom emotions, and whether the audience continues to gaze at the screen, the leaving time point, and the user's interaction, such as scanning the code, gesture clicking, etc. are recorded. These can also be used as positive feedback, and the feedback information is labeled as positive or negative (such as watch duration > threshold = "positive", significant "disgust" expression = "negative") and fed back to the control device of the advertisement delivery device, to continuously correct the adaptation of various advertisement materials to different feature vectors, so that the advertisement delivery device can generate more accurate feedback information, thereby adjusting the push strategy and providing more user demand-oriented advertisement content. For example, when the user shows strong interest, the advertisement delivery device can continuously display the current advertisement content or push more advertisements related to the user's interest to attract the user's attention and improve the advertisement delivery effect. When the user's interest weakens or shows avoidance behavior, the advertisement delivery device should switch the advertisement content in time or display some novel and interesting advertisements to rekindle the user's interest.

[0123] In this way, the advertising device can capture and analyze user emotional responses and behavior data in real time, generate accurate feedback information, and dynamically adjust the push strategy based on this information. This not only improves the accuracy and effectiveness of advertising, but also provides users with a more enjoyable and interactive shopping experience.

[0124] Optionally, with reference to Figure 6 Another embodiment of the present application provides a control method for an advertising device, based on the above Figure 1 The embodiment shown in the figure includes step S310, wherein:

[0125] S310, real-time rendering or playing the corresponding advertising video / text, and displaying the corresponding interactive elements on the display screen, including buttons, QR codes or links.

[0126] The advertising video / text is generated in real time based on the push information to ensure that the content displayed is highly relevant to the user's current interests and needs. In addition to displaying the advertising video or text itself, some interactive elements are embedded on the display screen to enhance user interaction with the advertising device. These interactive elements can include buttons, QR codes or links, etc.

[0127] Buttons can be set as controls that trigger specific actions or responses, such as "Learn More", "Buy Now" or "Share with Friends", etc. When users click on these buttons, the advertising device will receive the corresponding instructions and perform the corresponding operations according to the pre-set logic, such as jumping to the detailed product page, starting the purchase process or generating a sharing link, etc. Such design not only improves user engagement and interactivity, but also provides advertisers with more conversion opportunities.

[0128] The QR code can serve as a quick identification and connection means, allowing users to quickly obtain more information about the advertising content or jump to the corresponding web page, application or social media platform by scanning the QR code. This is an effective way for advertisers to drive traffic and conversion, while also providing users with a convenient way to access information.

[0129] The link can directly point to the official website, e-commerce platform or social media page of the advertiser, allowing users to enter the corresponding page for browsing, purchasing or interacting after clicking the link. Such design not only increases the exposure and spread of the advertisement, but also provides advertisers with more opportunities for display and sales.

[0130] Through the design of these interactive elements, the advertising device can provide users with a more rich and diverse interactive experience, and also provide advertisers with more conversion and marketing opportunities. Such an advertising method not only improves the advertising effect and user experience, but also brings more business value to advertisers. In actual application, advertisers can flexibly select and combine these interactive elements according to their own needs and the characteristics of target audiences to achieve the best advertising effect.

[0131] Optionally, referring to Figure 7 , the present application further provides a control method of an advertising device, based on the above Figure 1 embodiments, generating corresponding push information is specifically steps S2080-S2110, wherein:

[0132] S2080, obtaining a plurality of advertising materials;

[0133] S2090, labeling according to the category of the advertising material to generate an advertising material set;

[0134] S2100, according to the face information, identifying the age and gender of the user to determine the advertising interest degree of the user to the advertising material set;

[0135] S2110, according to the advertising interest degree, scoring each advertising material set based on a deep recommendation algorithm, and selecting at least one advertising material set with the highest score for delivery.

[0136] Advertising materials can include pictures, videos, text descriptions and other forms, and these advertising materials constitute the entire content that the advertising device can show to users. In order to more effectively push advertising, it is necessary to first classify and organize these advertising materials, and the advertising can be labeled by category, such as "sports", "digital", "makeup", "catering", etc., and the corresponding age, gender, interest degree and other recommendation indexes are set, so as to accurately push according to the needs and interests of users. Therefore, labeling according to the category of the advertising material is an important step, which helps to classify the advertising material into the corresponding advertising material set. In addition, the advertising material can be supported for online update according to actual needs, and the merchant can real-time on-shelf or off-shelf some advertising materials.

[0137] After obtaining the set of advertising materials, the next step is to identify the age and gender of the user based on their facial information. This is because users of different ages and genders often have different interests and preferences for different types of advertising content. For example, young women may be more interested in fashion and beauty advertising, while middle-aged men may be more interested in car and electronic product advertising. By identifying the age and gender of the user, the advertising delivery device can more accurately determine the user's advertising interest in the set of advertising materials, ensuring that the advertising content pushed matches the user's needs and interests.

[0138] After determining the user's advertising interest in the set of advertising materials, it is necessary to score each set of advertising materials based on a deep recommendation algorithm. The deep recommendation algorithm is an advanced machine learning algorithm that can learn user interests and behavior patterns by analyzing large amounts of user data, and recommend advertising content that best meets the user's needs based on these patterns. In the process of making recommendations, first, the audience's interest tags (such as "sports" and "young") are matched according to the rules, and obviously unsuitable advertisements are filtered out. Then use the deep recommendation model (such as Wide&Deep, DeepFM) to score the remaining advertising materials, select the top 1~N to deliver. And combine the actual playing effect (click / stay time) to feedback, to update the recommendation parameter information of the deep recommendation algorithm. In the control method of the advertising delivery device, the deep recommendation algorithm is used to score the set of advertising materials, and the most suitable set of advertising materials for delivery is selected according to the score. In this way, the advertising delivery device can ensure that the advertising content pushed not only matches the user's needs and interests, but also maximizes the user's attention, improving the effectiveness and conversion rate of advertising delivery.

[0139] It should be understood that when the advertising screen is placed in a supermarket, multiple people will often stop and watch at the same time, and different people will have different preferences, so the advertising needs to be delivered according to the preferences of different users. Therefore, before obtaining the user's expression information, all audience Tracking IDs and their InterestScore (interest weight) need to be summarized, wherein if the user has historical data (such as someone has stayed on this screen multiple times before), it can be added to this score to form a "short-term + long-term" mixed weight.

[0140] To achieve the above purpose, with reference to Figure 8 A further embodiment of the present application provides a control method of an advertising delivery device, based on the embodiment shown in Figure 1 In the case where the facial information identified by the first image data is at least two, the control method of the advertising delivery device further includes steps S800-S1000.

[0141] S800, obtain the historical stay time and the historical stay times of each of the face information in front of the display screen, and determine the interest weight corresponding to the historical stay time and the historical stay times;

[0142] S900, sort the interest weight of each of the face information, and divide the picture of the display screen according to the sorting of the interest weight to generate a plurality of display pictures, the size of the plurality of display pictures being determined according to the sorting of the interest weight;

[0143] S1000, continuously monitor the face information identified by the first image data, and after re-determining the interest weight of each of the face information, adjust the push information to generate a new plurality of display pictures.

[0144] The historical stay time and the historical stay times are important indicators reflecting the attention and interest degree of users to the advertising device. By analyzing the historical behavior data of users in front of the display screen, the advertising device can understand the preferences and habits of each user to the advertisement, so as to more accurately push the advertisement content meeting the needs of the user.

[0145] Among them, the historical stay time refers to the length of time that the user stays in front of the display screen, and the historical stay times reflects the frequency of the user watching the advertisement multiple times. These two indicators jointly constitute the comprehensive consideration of the user's interest in the advertisement. For example, a user stays in front of the display screen for a long time and watches the advertisement multiple times, which may indicate that he has a strong interest in the advertisement content; on the contrary, if the user only stays for a short time or rarely watches the advertisement, it may mean that he is not interested in the advertisement content.

[0146] Based on these historical behavior data, the advertising device will assign an interest weight to each user, which reflects the attention and interest degree of the user to the advertisement content. By sorting the interest weight, the advertising device can determine which users are more interested in the advertisement, so as to push them with more advertisement content meeting their needs.

[0147] After determining the interest weight, the person with the highest InterestScore (interest weight) is determined as the "leading object", and the highest matching advertisement is preferentially pushed. If the InterestScore (interest weight) of multiple viewers exceeds a certain threshold, the advertisement pushing device divides the screen according to the weight order to generate multiple display screens; among them, the main screen places the highest-weighted advertisement, and the sub-screen places other advertisements that the audience may be interested in, or a split-screen / carousel pushing mode is enabled. The size of these display screens is determined according to the interest weight order to ensure that more interested users can see larger and clearer advertisement screens. In this way, not only the accuracy and effect of advertisement pushing are improved, but also a more personalized viewing experience is brought to the user.

[0148] In addition, the advertisement pushing device also continuously monitors the face information identified by the first image data and re-determines the interest weight of each user. As the user's behavior changes and interests change, the interest weight will also be adjusted accordingly. Or when the leading object leaves, another suitable advertisement material is automatically switched according to the scores of the remaining audience. In this way, the advertisement pushing device can capture and analyze user behavior data in real time, dynamically adjust the pushing strategy, and ensure that the pushed advertisement content always matches the user's needs and interests. In addition, the advertisement playing time can also be dynamically lengthened or shortened according to the interest degree of the current audience, such as the leading object being obviously "happy" with the advertisement, the display time is extended.

[0149] In this way, the advertisement pushing device can provide more accurate and personalized advertisement services for users, and also provide more conversion and marketing opportunities for advertisers. Such an advertisement pushing method not only improves the effect of advertisement pushing and user experience, but also brings more business value to advertisers.

[0150] The application also provides a control device, which comprises a memory, a processor, and a control program of the advertisement pushing device stored in the memory and executable on the processor, and the control program is configured to implement the control method of the advertisement pushing device.

[0151] It is worth noting that, since the control device of the application is based on the control method of the advertisement pushing device described above, the embodiments of the control device of the application include all the technical solutions of all the embodiments of the control method of the advertisement pushing device, and the technical effects achieved are also exactly the same, which will not be repeated here.

[0152] The application also provides an advertisement pushing device comprising the control device described in the above embodiments.

[0153] It is worth noting that, since the advertisement delivery device of the present application is based on the above-mentioned control device, the embodiments of the advertisement delivery device of the present application include all the technical solutions of all the embodiments of the above-mentioned control device, and the technical effects achieved are also completely the same, which will not be described here.

[0154] It should be noted that in this document, the terms "comprise", "comprising", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or systems that include a series of elements not only include those elements, but also include other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or systems. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0155] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0156] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a number of instructions to make a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0157] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields based on the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A control method of an advertisement distribution apparatus including an imaging device and a display device, characterized by, The control method comprises: acquiring first image data captured by the camera device; identifying facial information in the first image data, determining the expression and head posture of the user, and generating corresponding push information; displaying the push information on the corresponding display screen; wherein, in the case that the facial information identified by the first image data is at least two, the control method of the advertising device further comprises: acquiring the historical stay time and the historical stay times of each facial information in front of the display screen, and determining the interest weight corresponding to the historical stay time and the historical stay times; sorting the interest weight of each facial information, and dividing the screen of the display screen according to the sorting of the interest weight to generate multiple display screens, the size of multiple display screens being determined according to the sorting of the interest weight; continuously monitoring the facial information identified by the first image data, and re-determining the interest weight of each facial information, and adjusting the push information to generate new multiple display screens.

2. The control method of the advertisement distribution device according to claim 1, wherein The identification of facial information in the first image data, the determination of the expression and head posture of the user, and the generation of corresponding push information comprise: reading each frame of picture in the first image data, and scaling each frame of picture according to a preset resolution to obtain first picture data; detecting the facial shape information in the first picture data to obtain first identification data; adding an external graph to the face in the first identification data, and assigning an identity to each added external graph; cropping the first identification data based on the external graph to obtain at least one first facial image; identifying the key feature points of the first facial image, and determining the expression information of the face based on the key feature points; obtaining the head posture and body action information of the user based on the key feature points; generating an audience interest score according to the expression information, the head posture and body action information of the user, to determine the push information.

3. The control method of the advertisement distribution device according to claim 2, wherein The identification of the key feature points of the first facial image and the determination of the expression information of the face based on the key feature points comprise: identifying the key feature points of the first facial image based on a key point detection algorithm, the key feature points including at least one of the eye position, nose position, ear position and mouth corner position of the user; adjusting the picture parameters of the first facial image based on the key feature points to obtain a second facial image; processing the second facial image based on a preset model algorithm to generate the expression information of the face.

4. The control method of the advertisement distribution device according to Claim 2, wherein The generation of an audience interest score according to the expression information, the head posture and body action information of the user specifically comprises: determining the angle difference between the head of the user and the display screen according to the head posture of the user, to determine the attention degree of the user to the display screen; in the case that the attention degree is high, determining the current interest level of the user according to the expression information and the body action information, to generate the audience interest score.

5. The control method of an advertisement distribution device according to Claim 1, wherein The control method of the advertising device further comprises: In a case where the push information is displayed on the display screen, second image data is acquired; facial expression changes in the second image and a gaze duration of the user are identified; an interest degree of the user is determined according to the facial expression changes and the gaze duration, to generate feedback information corresponding to the interest degree; the push information is adjusted according to the feedback information.

6. The control method of an advertisement distribution device according to Claim 1, wherein The display of the push information on the corresponding display screen comprises: real-time rendering or playing of a corresponding advertisement video / text and display of a corresponding interactive element on the display screen, the interactive element comprising a button, a two-dimensional code or a link.

7. The control method of the advertisement distribution device according to Claim 1, wherein The generation of the corresponding push information specifically comprises: acquisition of a plurality of advertisement materials; tagging according to categories to which the advertisement materials belong, to generate an advertisement material set; identification of an age and a gender of the user according to the face information, to determine an advertisement interest degree of the user for the advertisement material set; scoring of each of the advertisement material sets based on a deep recommendation algorithm according to the advertisement interest degree, and selection of at least one advertisement material set with the highest score for delivery.

8. A control device characterized by comprising: The control device comprises a memory, a processor and an advertisement delivery device control program stored on the memory and executable on the processor, and the advertisement delivery device control program is configured to implement the advertisement delivery device control method according to any one of claims 1 to 7.

9. An advertisement delivery apparatus characterized by comprising: The control device according to claim 8. The control device according to claim 8.

Images