A method for evaluating user recognition of advertisements based on feature recognition

Abstract

The present invention belongs to the field of big data processing, and specifically relates to a method for evaluating the user's recognition of advertisements based on feature recognition. The method comprises the following steps: S1: obtaining the feature data of the currently playing advertisement: including obtaining the playing time T of each advertisement played, and the keyword data set associated with each advertisement. S2: obtaining the feedback data of each user on the advertisement playing, including: obtaining the voice stream data generated by all users in the advertisement delivery area during the advertisement playing, monitoring the video stream data of all users in the advertisement delivery area, and the instruction issued by one or more users in the advertisement delivery area to switch the currently playing advertisement; judging whether the instruction to switch the currently playing advertisement is received. S3: calculating the recognition evaluation value of each user for the current advertisement based on different feedback information. The present invention solves the problem that the existing advertisement delivery equipment cannot obtain the feedback and evaluation of offline users on the advertisement delivery.

Description

[0001] This application is a divisional application with application number: CN 202110687605.2, application date: 2021 / 06 / 21, and invention name: An intelligent media management system based on VOC car owner big data platform. Technical Field

[0002] The present invention belongs to the field of big data processing, and in particular relates to a method for evaluating user recognition of advertisements based on feature recognition. Background Art

[0003] There are a large number of advertising devices for playing advertisements in almost all scenes such as elevators, shopping malls, garages, and subway stations. These devices will play built-in advertising videos in a loop. Existing advertising devices can usually only play advertisements in the server in a fixed playback order, and cannot adjust the playback order or content of advertisements for different users. If the content needs to be changed, the relevant equipment managers need to manually switch or update the content played by the device remotely or on-site.

[0004] When placing advertisements online, advertisers can analyze and judge the needs of users based on the captured web browsing behavior of users, or the usage or search records in various apps, predict the preferences or needs of users, and accurately push targeted advertisements to users based on this. When placing advertisements offline, it is difficult to judge the evaluation of users on advertisements, and advertisers can only make judgments based on experience and place advertisements in specific areas. In fact, most advertisers do not even care about user feedback when placing offline advertisements. They only hope to push the same advertisement content intensively and fully in different places through "carpet bombing" in order to achieve better publicity effects. This advertising method is inefficient and wasteful. The reason for this dilemma is that there is no method in the prior art that can effectively evaluate the user's recognition of advertisements. If this problem can be solved, the effect of placing different advertisements can be effectively analyzed offline. Summary of the invention

[0005] In order to solve the problem that existing advertising delivery devices have low efficiency in advertising delivery and cannot obtain feedback and evaluation of offline users on advertising delivery, the present invention provides a method for evaluating user recognition of advertisements based on feature recognition.

[0006] The present invention is implemented by the following technical solutions:

[0007] A method for evaluating user recognition of advertisements based on feature recognition comprises the following steps:

[0008] S1: Get the characteristic data of the currently playing advertisement:

[0009] The playing time T of each advertisement played and the keyword data set associated with each advertisement are obtained.

[0010] S2: Obtaining feedback data from each user on the advertisement playback, including:

[0011] Acquire the voice stream data generated by all users in the advertising delivery area during the advertising playback, monitor the video stream data of all users in the advertising delivery area, and the instructions issued by one or more users in the advertising delivery area to switch the currently playing advertisement; determine whether an instruction to switch the currently playing advertisement is received: if yes, assign 1 to the feature quantity SW reflecting the instruction; otherwise, assign 0 to SW.

[0012] S3: Calculate the recognition evaluation value of each user for the current advertisement, including the following process:

[0013] S31: Perform speech recognition on the speech stream data, extract keywords that match the feature data in the keyword data set, and count their number N1.

[0014] S32: Perform video action recognition on the video stream data; extract gestures and actions representing the user's feedback on the currently playing advertisement, and count their number N2.

[0015] S33: Perform video action recognition on the video stream data; extract the characteristic actions that reflect the changes in the eye focus position of each user, and calculate the attention time t of each user on the currently playing advertisement based on the characteristic actions. n ; Where n represents the user ID of the current user.

[0016] S34: Sampling the framed images of the video stream data at alternate frames according to the sampling frequency; performing image recognition on the sampled images at alternate frames; extracting the facial expressions of each user, and classifying the facial expressions into like, ignore or disgust; counting the number of the three types of expression classification results of each user respectively, and calculating the proportion of the number of the three types of expression classification results of each user in the total sample size of the user.

[0017] S35: Get the value of SW.

[0018] S36: Calculate the recognition evaluation value E of each user for the current advertisement by the following formula: n :

[0019]

[0020] In the above formula, n represents the user ID of the current user, E n represents the evaluation value of the currently playing advertisement by the user numbered n, E n ≥0, and En The larger the value of reflects the higher the user's recognition of the currently played multimedia; represents the attention concentration of the user numbered n on the currently playing advertisement; k1 represents the influence factor of voice information feedback on the overall recognition evaluation result; k2 represents the influence factor of gesture feedback on the overall recognition evaluation result; k3 represents the influence factor of expression feedback on the overall recognition evaluation result; k4 represents the influence factor of attention concentration on the overall recognition evaluation result; m1 represents the score of a single keyword in voice information feedback; m2 represents the score of a single gesture in gesture feedback; m3 represents the score of attention concentration; a represents the score of liking expression, p 1,n is the ratio of the number of images sampled every other frame that the user n classifies as liking the expression; b is the score of ignoring the expression, p 2,n represents the proportion of expressions classified as ignored by user number n in the total number of images sampled every other frame; c represents the score of disgust expression, p 3,n It represents the proportion of expressions classified as disgust by user number n in the total number of images sampled every other frame.

[0021] As a further improvement of the present invention, in step S1, the associated words are set before each advertisement is released, and the characteristic data in the keyword data set associated with each advertisement at least includes:

[0022] (1) Keywords that reflect the product being advertised;

[0023] (2) Keywords that reflect the target customer group of the advertisement;

[0024] (3) Keywords that reflect the spokesperson or character image of the advertisement;

[0025] (4) High-frequency or special keywords in advertising words;

[0026] (5) Classification of advertisement duration;

[0027] (6) Classification of advertising styles.

[0028] As a further improvement of the present invention, in step S2, the method of the user's instruction to switch the currently playing advertisement includes key input, voice interaction and gesture interaction; the voice interaction is realized by identifying the voice keywords issued by the user to request switching the currently playing advertisement; the gesture interaction is realized by identifying the characteristic gestures issued by the user to request switching the currently playing advertisement; the key input represents the key input instruction requested to switch the currently playing advertisement input directly by the user through keys.

[0029] As a further improvement of the present invention, the voice keywords are obtained by the voice recognition algorithm based on the real-time voice stream data. The characteristic gestures are obtained by the video action recognition algorithm based on the real-time video stream data. The key input instructions are obtained by the physical switch key module installed at the advertisement playing site.

[0030] As a further improvement of the present invention, in step S2, the user's feedback data on the advertisement playback includes the following contents:

[0031] (1) Changes in users’ facial expressions when watching an ad, including liking, ignoring, or disliking it;

[0032] (2) Whether the user's direct discussion of the advertisement involves the content in the keyword dataset corresponding to the advertisement;

[0033] (3) The gestures made by users when watching the advertisement;

[0034] (4) the amount of time a user focuses on an advertisement;

[0035] (5) Whether the user requests to switch the currently playing advertisement.

[0036] As a further improvement of the present invention, the time that a user focuses on a certain advertisement is calculated by the following formula:

[0037]

[0038] In the above formula, t n Indicates the time that the user numbered n pays attention to the currently playing advertisement; t 1n Indicates the viewing time of the user numbered n during the current advertisement playback; t 2n Indicates the closing time of the user numbered n during the current advertisement playback; t 3n Indicates the time that user number n keeps his head down during the current ad playback; t 4n Indicates the head turning duration of user number n during the current ad playback.

[0039] As a further improvement of the present invention, in step S32, the gestures and actions of the user to provide feedback on the currently playing advertisement include nodding, clapping, pointing the hand at the advertisement playing interface, and raising or turning the head from a non-direct viewing state to a direct viewing state during the advertisement playing.

[0040] As a further improvement of the present invention, the voice stream data is obtained through a microphone installed at the advertisement playing device.

[0041] As a further improvement of the present invention, the video stream data is obtained through a video monitoring device installed around the advertisement playing device.

[0042] As a further improvement of the present invention, in step S34, expression recognition is performed using a neural network algorithm trained with a large number of samples.

[0043] The technical solution provided by the present invention has the following beneficial effects:

[0044] The present invention provides a method for evaluating the user's recognition of advertisements based on feature recognition. Various types of feedback information made by users when playing advertisements are obtained, and then various types of feedback behaviors of users are identified by using cutting-edge technologies such as voice recognition, image recognition, and video action recognition. Each type of feedback behavior is quantified, and then a standard, unified, and reference-worthy recognition evaluation value is given to different advertisements.

[0045] The recognition evaluation value calculated by the present invention takes into account all types of feedback behaviors that users can check during the advertisement viewing period. Based on the identification of various types of characteristics of feedback behaviors, the accurate recognition evaluation of each user for different advertisements is obtained; which can then be used as a basis for evaluating the effectiveness of advertisement delivery. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of module connections of an intelligent media management system based on a VOC car owner big data platform provided in Example 1 of the present invention;

[0047] Figure 2 This is a flow chart of a method for accurate advertising delivery based on user portraits in Embodiment 1 of the present invention;

[0048] Figure 3 This is a logic block diagram of a process for obtaining a user tag of the current user in Embodiment 1 of the present invention;

[0049] Figure 4 This is a logic block diagram of a process for acquiring a target portrait data set of a current user group in Embodiment 1 of the present invention;

[0050] Figure 5 This is a flow chart of a method for creating an advertising analysis database according to Embodiment 2 of the present invention;

[0051] Figure 6 A diagram for distinguishing types of feature data contained in identity tags in an advertisement analysis database in Example 2 of the present invention;

[0052] Figure 7 A diagram for distinguishing the types of feature data contained in the user portrait data set in Example 2 of the present invention;

[0053] Figure 8 This is a module diagram of a system for creating an advertisement analysis database in Embodiment 3 of the present invention;

[0054] Fig. 9 This is a flow chart of a method for evaluating user recognition of advertisements based on feature recognition in Embodiment 4 of the present invention;

[0055] Fig.10 This is a flow chart of a method for timely analyzing user needs in a business district scenario in Example 5 of the present invention;

[0056] Fig.11 This is a flow chart of a method for matching user needs and advertising content in Embodiment 6 of the present invention;

[0057] Fig.12 This is a module schematic diagram of a garage giant screen MAX smart terminal with intelligent voice interaction function provided in Example 7 of the present invention;

[0058] Fig.13 This is a diagram for distinguishing the types of switching instructions used by the human-computer interaction module in a garage giant screen MAX smart terminal with an intelligent voice interaction function according to Example 7 of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] Example 1

[0061] This embodiment provides an intelligent media management system based on the VOC car owner big data platform, which is used to obtain the matching degree between the current user and the advertisement to be placed, and then adjust the advertisement play sequence table. Figure 1 As shown, the intelligent media management system includes: a keyword extraction module, a historical user information query module, a user type classification module, a user tag establishment module, an identity feature recognition module, a target portrait data set establishment module, and an advertisement play sequence table adjustment module.

[0062] The keyword extraction module is used to extract a keyword data set associated with each advertisement in the advertisement play sequence table, and the characteristic data in the keyword data set is a plurality of pre-set keywords related to the content of the advertisement.

[0063] In this embodiment, the feature data in the keyword data set of each advertisement is the same as that in Embodiment 1, and at least includes:

[0064] (1) Keywords that reflect the product being advertised.

[0065] (2) Keywords that reflect the target customer group of the advertisement.

[0066] (3) Keywords that reflect the advertising spokesperson or advertising character.

[0067] (4) High-frequency or special keywords in advertising slogans.

[0068] (5) Classification of advertising duration.

[0069] (6) Classification of advertising styles.

[0070] The historical user information query module is used to query the user portrait data set of each historical user from an advertising analysis database, and obtain various feature data about each historical user in the user portrait data set. Among them, the advertising analysis database is stored in the VOC car owner cloud big data platform. The advertising analysis database is the database created in Example 1. The advertising analysis database contains the user portrait data set of the collected historical users; the user portrait data set includes the facial feature data and user tags of each historical user, and the user tags include identity tags, preference tags and dislike tags; the identity tags store feature data reflecting the identity characteristics of the user, and the feature data in the identity tags include: gender, age group, dressing style and other features; other features represent identifiable non-gender, age group and dressing style features that are useful for distinguishing the identity characteristics of the user. The preference tags store feature data reflecting the object of the user's preference, and the dislike tags store feature data reflecting the object of the user's dislike. Among the features extracted by the identity feature recognition module, the age group is divided into 0-10 years old, 10-20 years old, 20-30 years old, 30-50 years old, 50-70 years old or over 70 years old; the dressing style includes leisure, business, sports, children or the elderly. Other features in the identity tag reflect the content including whether the person wears glasses, whether the person wears a hat, whether the person has hair loss, whether the person wears lipstick, whether the person wears high heels, whether the person has a beard, and whether the person wears a watch. For the above features, if yes, feature data reflecting the feature is added to other features, otherwise, feature data is not added to other features.

[0071] The user type classification module is used to extract the facial features of all target users in an advertising delivery area; then the extracted facial features are compared with the facial features of all historical users in the advertising analysis database to distinguish whether the current user is a historical user or a new user.

[0072] In this embodiment, the data sources of the user type classification module and the identity feature recognition module are both video stream data of multi-angle monitoring of the advertising delivery area; the user type classification module includes a facial feature extraction unit, a facial feature comparison unit and a user type classification unit. The facial feature extraction unit is used to extract the facial features of all users appearing in the video stream data; the facial feature comparison unit is used to obtain all facial features extracted by the facial feature extraction unit and the facial features of all historical users queried by the historical user information query module, and perform feature comparison on the two. The user type classification unit is used to classify all users appearing in the video stream data as historical users or new users according to the results distinguished by the facial feature comparison unit.

[0073] The user tag establishment module is used to establish an empty user tag for each newly added user. The established user tag includes an identity tag, a preference tag and a dislike tag. The user tag establishment module also adds an exclusive user number to the identity tag of each newly added user.

[0074] The identity feature recognition module is used to extract identity features of the newly added user and add the extracted identity features to the identity tag of the corresponding newly added user.

[0075] The target portrait dataset building module is used to:

[0076] (1) Set a historical user ratio critical value q0 and calculate the ratio q of current users identified as historical users in the current user group in the advertising delivery area.

[0077] (2) Determine the relationship between q and q0, and make the following decisions based on the results:

[0078] (i) When q ≥ q0, extract the feature data from the preference tags of all historical users, remove the duplicates from the feature data, and use it as the target profile data set for the current user group;

[0079] (ii) When q<q0, first extract the feature data in the preference tags of all historical users. Then calculate the overlap degree Dc1 between the content in the identity tag of each newly added user and the content in the identity tag of each historical user in the advertising analysis database. The calculation formula of Dc1 is as follows:

[0080]

[0081] Extract the feature data from the preference tags of the historical users with the largest overlap Dc1 with the identity tags of each new user. Merge the two parts of feature data and remove duplicates; use them as the target portrait data set for the current user group.

[0082] The advertisement play sequence table adjustment module is used to: (1) calculate the overlap degree Dc2 between the feature data in the keyword data set associated with each advertisement extracted by the keyword extraction module and the feature data in the target portrait data set. The calculation formula of Dc2 is as follows:

[0083]

[0084] (2) Reorder the advertisements in the advertisement play sequence table according to the calculation result of Dc2 of each advertisement from large to small, and obtain an adjusted advertisement play sequence table.

[0085] In this embodiment, the provided intelligent media management system based on the VOC car owner big data platform is applied to an advertising delivery system with a multi-angle monitoring device, the advertising delivery system is used to play the advertisements to be delivered according to the advertisement play sequence table; the multi-angle monitoring device is used to obtain the multi-angle monitored video stream data of all target users in the advertising delivery area of ​​the advertising delivery device. The identity feature recognition module uses the image recognition unit to perform image recognition on the framed images of the captured video stream data, and then extracts the features reflected in the image that are the same as the feature data category stored in the identity tag.

[0086] Of course, in other embodiments, the relevant advertising delivery system and multi-angle monitoring equipment can also be used as part of the intelligent media management system based on the VOC car owner big data platform in this embodiment, thereby integrating and coordinating the data collection, data processing, data analysis and advertising delivery processes required to be completed in this embodiment.

[0087] This embodiment also includes a method for accurate advertising delivery based on user portraits. This accurate delivery method is applied to an intelligent media management system based on the VOC car owner big data platform in this embodiment; Figure 2 As shown, the precise delivery method includes the following steps:

[0088] Step 1: Get the user tag of the current user, such as Figure 3 As shown, the specific process is as follows:

[0089] 1. Obtain the facial features of each current user in the advertising delivery area.

[0090] 2. Perform facial recognition on each current user in turn, query an advertising analysis database containing a user portrait data set of multiple historical users based on the facial recognition results, and make the following judgments:

[0091] (1) When the facial features of the current user match the feature data in one of the historical user's facial feature data, all feature data in the user tag of the historical user are obtained.

[0092] (2) When the facial features of the current user do not match the feature data in the facial feature data of all historical users, the current user is determined to be a new user, and an empty user tag is created for the new user.

[0093] Among them, the user portrait dataset contains the facial feature data and user tags of the corresponding historical users; user tags include identity tags, preference tags and dislike tags.

[0094] 3. Obtain multi-angle images of the newly added user, perform image recognition on the multi-angle images, and supplement the feature data in the identity tag of the newly added user based on the recognition result; wherein the feature data supplemented in the identity tag includes user number, gender, age group, clothing style and other features; other features represent identifiable non-gender, age group and clothing style features that are useful for distinguishing user identity features.

[0095] Step 2: Create a target portrait dataset for the current user group, such as Figure 4 As shown, the specific process is as follows:

[0096] 1. Set a historical user ratio critical value q0, and calculate the ratio q of current users identified as historical users in the advertising delivery area in the current user group.

[0097] 2. Determine the relationship between q and q0, and make the following decisions based on the results:

[0098] (1) When q≥q0, extract the feature data from the preference tags of all historical users, remove the duplicates from the feature data, and use it as the target portrait data set for the current user group.

[0099] (2) When q<q0, first extract the feature data in the preference tags of all historical users; then calculate the overlap degree Dc1 between the content in the identity tag of each new user and the content in the identity tag of each historical user in turn. The calculation formula of Dc1 is as follows:

[0100]

[0101] Extract the feature data from the preference tags of the historical users whose identity tags have the largest overlap Dc1 with each new user; merge the two parts of feature data (the identified historical users and the preference tags of the historical users whose identity tags have the largest overlap with each new user) and, after deduplication of the feature data, use them as the target portrait data set for the current user group.

[0102] Step 3: Adjust the play order of the advertisements in the advertisement play sequence table. The specific process is as follows:

[0103] 1. Obtain a keyword data set associated with each advertisement in the advertisement play sequence table. The feature data in the keyword data set is a plurality of pre-set keywords related to the content of the currently played advertisement.

[0104] 2. Obtain the feature data in the target portrait data set, and calculate the overlap degree Dc2 between the feature data in the keyword data set associated with each advertisement and the feature data in the target portrait data set. The calculation formula of Dc2 is as follows:

[0105]

[0106] 3. Sort the advertisements in the advertisement play sequence list in descending order according to the calculation result of Dc2 of each advertisement to obtain a re-adjusted advertisement play sequence list.

[0107] The principle and implementation logic of the method for adjusting the advertisement play sequence table in the advertisement delivery system provided in this implementation are mainly based on the following:

[0108] Since the present embodiment has acquired the data in the created advertisement analysis database, when advertisements are delivered, facial recognition is performed on all users in the advertisement delivery area to distinguish whether these users are historical users in the advertisement analysis database or new users not collected by the advertisement analysis database.

[0109] Considering that the advertising analysis data has realized the profiling process of historical users, that is, the rich feature data in the user tags. At this time, when the vast majority of users in the advertising delivery area are historical users, it can be considered that the needs and preferences of these historical users can represent the entire current user group. By obtaining the corresponding historical user preference tags and then extracting the feature data therein, a target portrait dataset can be obtained to characterize the preferences or needs of the current user group.

[0110] When the number of new users in the advertising delivery area reaches a certain level, it is not possible to rely solely on historical users for profiling. At this time, it is obviously too late to conduct real-time analysis on these new users, but since this implementation can query an advertising analysis data set with a large enough sample size and rich enough data, this embodiment can identify the identity characteristics of these new users (which can be achieved through image recognition technology), and then use this identity feature to compare with the user label in the advertising analysis data set, extract the most suitable historical user, and use the user label of this historical user temporarily as the user label of the new user, and then obtain the features in the preference label of the new user. Because the identity characteristics of the user (such as age, height, gender, dress, physiological characteristics) are highly correlated with the user's needs or preferences (features in the preference label). Therefore, this approximate substitution in this embodiment should be highly credible. This embodiment can obtain a target portrait data set of a user group containing a large number of new users through this technical solution.

[0111] After obtaining the target portrait data set of the user group in the advertising delivery area, the present embodiment further compares the feature data in the target portrait data set with the keyword data set of each advertisement to be played, and the overlap between the two can be found. The higher the overlap, the more it indicates that the user group is the target customer of the advertisement. At this time, these advertisements should be placed in a priority position. Based on this logic, the present embodiment realizes the reordering of the advertisement playback sequence list to ensure that the most suitable advertisements can be delivered to the target group with higher priority.

[0112] Example 2

[0113] This embodiment provides an advertising analysis database containing multiple historical user data. The advertising analysis database is the advertising analysis database mentioned in Embodiment 1. The data in the advertising analysis database realizes an accurate portrait of user interests and hobbies; thus, accurate advertising targeted marketing can be carried out to users.

[0114] The data in the advertising analysis database is mainly obtained by identifying the identity characteristics of users in scenes such as elevators, garages, and shopping malls, and analyzing the user's recognition evaluation results of video advertisements. The data in the advertising analysis database mainly includes the following aspects:

[0115] (1) Facial features of users; this feature is mainly used to distinguish the identities of different users and serves as the user's unique identity marker. At the same time, the advertising analysis database also assigns a unique user number to the user based on different identity markers.

[0116] (2) User identity characteristics: This part of the data is relatively rich, covering all the characteristics that can be obtained to distinguish the user's identity characteristics, including age, height, body shape, clothing, physiological state, etc. These characteristics are of reference value for judging the user's work type, behavioral habits, demand characteristics, hobbies, group affiliation, etc.

[0117] (3) Objects of user preference: The data in this part is obtained through user feedback on different types of advertisements. The content in this part is constantly updated and optimized. It can basically depict the objects of user concern and preference in the current state.

[0118] (4) Objects that users dislike: The data in this part is obtained through users’ feedback on different types of advertisements. The content in this part is also constantly updated and optimized. It can basically depict the objects that users do not care about or dislike in their current state.

[0119] In this embodiment, if Figure 5 As shown, the method for creating an advertising analysis database is as follows:

[0120] Step 1: Create user tags for each user

[0121] 1. During the advertisement playback process, the facial features of each user are obtained in turn and facial recognition is performed on the facial features.

[0122] 2. Query the advertising analysis database based on the facial recognition results to determine whether the current user's facial features match the facial features of a historical user in the advertising analysis database:

[0123] (1) If yes, skip the current user.

[0124] (2) Otherwise, an empty user tag is created for the current user; the user tag includes identity tag, preference tag and dislike tag.

[0125] 3. Obtain multi-angle images of each user, and supplement the feature data in the identity tag of each user based on the image recognition results of the multi-angle images.

[0126] In this step, it is possible to create a profile for each user, whether it is a new user or a historical user, as long as the user appears in the target area and can be captured, the user can be profiled and analyzed. This enables the scale of the advertising analysis database established in this embodiment to reach a relatively high level, and the samples are also rich enough. This lays a data foundation for the later application of this database for application development.

[0127] In this embodiment, if Figure 6As shown, the feature data supplemented in the identity tag include user number, gender, age group, dressing style and other features; other features represent identifiable non-gender, age group and dressing style features that are useful for distinguishing user identity features.

[0128] The age group in the identity tag is one of 0-10 years old, 10-20 years old, 20-30 years old, 30-50 years old, 50-70 years old, and over 70 years old classified according to the image recognition result; the dressing style in the identity tag includes leisure, business, sports, children or the elderly. In this embodiment, considering that age has an important impact on the needs of users, the age feature is one of the identity features that must be considered. At the same time, since conventional image information collection cannot directly obtain the user's occupational characteristics, this embodiment can roughly divide the user's occupation or social identity to a certain extent by classifying the user's dressing style.

[0129] At the same time, other features in the identity tag reflect whether the user wears glasses, a hat, hair loss, lipstick, high heels, beard, watch, etc. If the above features are true, feature data reflecting the features are added to other features, otherwise, feature data is not added to other features. Other features in the identity tag are very typical features that distinguish users, and these features are highly correlated with the consumption needs of different users. For example, women who wear lipstick and high heels may have a higher favorability for advertisements for clothing and cosmetics. People with beards are usually not very concerned about razors. People with hair loss may be more interested in hair growth products and health products, etc.

[0130] In fact, after applying more diverse feature extraction technologies, this embodiment can also obtain more different types of identity features. The richer the feature quantities obtained, the more refined the feature classification of the user.

[0131] Step 2: Get the feature data of the currently playing ad

[0132] 1. Obtain the playing time T of each advertisement played and the keyword data set associated with each advertisement.

[0133] The characteristic data in the keyword data set are a plurality of pre-set keywords related to the content of the currently played advertisement. The characteristic data in the keyword data set of each advertisement at least includes:

[0134] (1) Keywords that reflect the product being advertised.

[0135] (2) Keywords that reflect the target customer group of the advertisement.

[0136] (3) Keywords that reflect the advertising spokesperson or advertising character.

[0137] (4) High-frequency or special keywords in advertising slogans.

[0138] (5) Classification of advertising duration.

[0139] (6) Classification of advertising styles.

[0140] In this embodiment, a rich set of keywords are set for each advertisement, and these keywords include various types of information that customers can receive from an advertisement. When a user approves of the advertisement, or gives positive feedback on the content in the advertisement, it can be considered that some or all of the features in the keyword data set of the advertisement are of concern or preference to the user. On the contrary, when a user shows disgust or gives negative feedback on an advertisement, it can be considered that the user is indifferent to or disgusted with certain features in the keyword data set of the advertisement. In this way. When the sample size of the feedback data of the corresponding users on different types of advertisements is collected large enough, the preferences of the customers can be basically analyzed, and then the user's preferences can be profiled.

[0141] Step 3: Obtain feedback data from each user on the ad playback

[0142] 1. Acquire the voice stream data generated by all users in the advertising delivery area during the advertising playback, monitor the video stream data of all users in the advertising delivery area, and the instructions issued by one or more users in the advertising delivery area to switch the currently playing advertisement.

[0143] The instructions issued by the user to switch the currently playing advertisement include key input, voice interaction and gesture interaction. Voice interaction is achieved by identifying the voice keywords issued by the user to switch the currently playing advertisement; gesture interaction is achieved by identifying the characteristic gestures issued by the user to switch the currently playing advertisement; key input refers to the key input instruction directly input by the user through keys to switch the currently playing advertisement.

[0144] Voice keywords are obtained by a voice recognition algorithm based on real-time voice stream data; feature gestures are obtained by a video action recognition algorithm based on real-time video stream data; key input instructions are obtained through a physical switch key module installed at the advertising playback site.

[0145] In this embodiment, the user's feedback mainly includes the following aspects:

[0146] (1) Changes in users’ facial expressions when watching advertisements.

[0147] (2) Direct discussion of the advertisement by users. For example, discussing an actor or spokesperson in the advertisement, discussing the effects of the product, etc.

[0148] (3) Gestures made by users when watching advertisements. For example, a user's hand points directly at the advertisement playback device to prompt other users to pay attention and watch. This reflects that the user is concerned about the currently playing advertisement.

[0149] (4) The amount of time a user focuses on an advertisement.

[0150] (5) The user requests to switch the currently playing advertisement. This directly reflects that the user does not like the advertisement.

[0151] In addition, when technical conditions are mature, other types of feedback can also be extracted and applied to later data analysis, such as the user's laughter and other detailed characteristic movements.

[0152] 2. Determine whether an instruction to switch the currently playing advertisement is received. If yes, assign 1 to the feature quantity SW reflecting the instruction; otherwise, assign 0 to SW.

[0153] Step 4: Calculate each user's recognition rating of the current advertisement

[0154] 1. Perform speech recognition on the speech stream data, extract the keywords that match the feature data in the keyword data set, and count their number N1.

[0155] 2. Perform video action recognition on the video stream data; extract gestures and actions representing the user's feedback on the currently playing advertisement, and count their number N2.

[0156] Among them, the gestures and actions of the user to give feedback on the currently playing advertisement include nodding, clapping, pointing the hand to the advertisement playing interface during the advertisement playing, raising or turning the head from a non-direct viewing state to a direct viewing state, etc.

[0157] 3. Perform video action recognition on the video stream data; extract the characteristic actions that reflect the changes in the eye focus position of each user, and calculate the attention time t of each user on the currently playing advertisement based on the characteristic actions n ; Wherein, n represents the user number of the current user.

[0158] The time t that the user numbered n pays attention to the currently playing ad n The calculation method is as follows:

[0159]

[0160] In the above formula, t 1nIndicates the viewing time of the user numbered n during the current advertisement playback; t 2n Indicates the closing time of the user numbered n during the current advertisement playback; t 3n Indicates the time that user number n keeps his head down during the current ad playback; t 4n Indicates the head turning duration of user number n during the current ad playback.

[0161] In this embodiment, when counting the time that the user pays attention to the advertisement, both the time that the user looks directly at the advertisement playback interface and the time that the user does not look directly at the advertisement are taken into account. In this embodiment, the relatively accurate attention time is obtained mainly by eliminating the time that is determined to be in a non-attention state and then averaging it with the time that is determined to be in an attention state.

[0162] 4. Sample the framed images of the video stream data at alternate frames according to the sampling frequency; perform image recognition on the sampled images; extract the facial expressions of each user, and classify the facial expressions into like, ignore or disgust; count the number of the three types of expression classification results of each user respectively, and calculate the proportion of the three types of expression classification results of each user in the total sample size of the user.

[0163] 5. Get the value of the SW.

[0164] 6. Calculate the recognition evaluation value E of each user for the current advertisement using the following formula: n :

[0165]

[0166] In the above formula, n represents the user ID of the current user, E n represents the evaluation value of the currently playing advertisement by the user numbered n, E n ≥0, and E n The larger the value of reflects the higher the user's recognition of the currently played multimedia; represents the attention concentration of the user numbered n on the currently playing advertisement; k1 represents the influence factor of voice information feedback on the overall recognition evaluation result; k2 represents the influence factor of gesture feedback on the overall recognition evaluation result; k3 represents the influence factor of expression feedback on the overall recognition evaluation result; k4 represents the influence factor of attention concentration on the overall recognition evaluation result; m1 represents the score of a single keyword in voice information feedback; m2 represents the score of a single gesture in gesture feedback; m3 represents the score of attention concentration; a represents the score of liking expression, p 1,n is the ratio of the number of images sampled every other frame that the user n classifies as liking the expression; b is the score of ignoring the expression, p 2,nrepresents the proportion of expressions classified as ignored by user number n in the total number of images sampled every other frame; c represents the score of disgust expression, p 3,n It represents the proportion of expressions classified as disgust by user number n in the total number of images sampled every other frame.

[0167] In this embodiment, expression recognition can be accomplished by using a neural network algorithm trained with a large number of samples. There are also a large number of products that can be directly applied to speech recognition, video action recognition, etc., and this embodiment will not be repeated for this part.

[0168] In this embodiment, various types of feedback information of the user on the played advertisement are extracted from the user's voice stream data and video stream data through the technology of voice recognition, image recognition and video action recognition. After these feedback information are quantified by the method provided by this embodiment, an evaluation result reflecting the user's recognition of the current advertisement can be obtained. This result reflects the user's likes and dislikes of the current advertisement, and can be used to characterize the user's needs or interests.

[0169] Step 5: Establish or update the advertising analysis database

[0170] 1. Set E n A high threshold E h and a low threshold E l , where E h Indicates the critical value of the user liking the currently playing advertisement, E l Indicates the critical value of the user's disgust with the currently playing advertisement, E l >0.

[0171] 2. When E n ≥E h And p 1，n +p 2，n ≥p 3，n When the feature data in the keyword data set associated with the currently playing advertisement is added to the preference tag corresponding to the current user, the feature data of the supplemented preference tag is deduplicated; and the feature data in the dislike tag corresponding to the current user that is identical to the feature data in the keyword data set is deleted.

[0172] 3. When E≤E l And p 2，n +p 3，n ≥p 1，n When the feature data in the keyword data set associated with the currently playing advertisement is added to the disgust tag corresponding to the current user, the feature data of the supplemented disgust tag is deduplicated; then the feature data in the preference tag corresponding to the current user that matches the feature data in the keyword data set is deleted.

[0173] 4. Update the user label of each user, obtain a new user portrait data set for each user, and create an advertising analysis database.

[0174] Among them, Figure 7 As shown, the user portrait dataset includes the facial feature data and user labels of the corresponding users.

[0175] The core content of the advertisement analysis database is the content of the like and dislike tags obtained by analyzing the user's behavior. This part of the content is the direct data used for user demand analysis in the later stage. In this embodiment, the user's likes and dislikes can be directly estimated through the user's feedback when watching the advertisement, and the user's likes and dislikes should be consistent with some or all of the features in the keyword data set of the advertisement. Therefore, in this embodiment, after each advertisement is played, the user's accurate attitude towards the advertisement is determined by analyzing and counting the user's feedback information, and then the keyword data set of the advertisement is used as the feature in the like or dislike tag of the current user when specific conditions are met.

[0176] In order to avoid misclassification, it is necessary to conduct a more rigorous review of the determined user's attitude. The determination process of this embodiment introduces a special threshold determined based on expert experience as the basis for judging the user's true attitude. The threshold E in this embodiment is h and E l It is determined after repeated verification and has a high degree of credibility, thus ensuring that the final user portrait is accurate and reliable.

[0177] Example 3

[0178] This embodiment provides a system for creating an advertisement analysis database. The system adopts the method for creating an advertisement analysis database included in Embodiment 2 to implement the process of creating and updating the advertisement analysis database.

[0179] like Figure 8 As shown, the creation system includes: a historical user query module, an advertising feature data extraction module, a user feedback data extraction module, a facial recognition module, an image recognition module, a speech recognition module, a video action recognition module, a user tag establishment module, a recognition evaluation value calculation module, and a database creation module.

[0180] Among them, the historical user query module is used to query an advertising analysis database and extract the user portrait data set of historical users collected therein; the user portrait data set includes the facial feature data and user tags of each historical user, and the user tags include identity tags, preference tags and dislike tags.

[0181] The advertisement feature data extraction module is used to extract the playing time T of each advertisement played and a keyword data set associated with the advertisement when an advertisement delivery system plays the advertisement.

[0182] The user feedback data extraction module is used to: (1) obtain the voice information generated by the users who watch the advertisements in the advertisement delivery area when the advertisement delivery system plays the advertisements, and obtain the voice stream data related to each advertisement. (2) obtain the multi-angle monitoring video of all users who watch the advertisements in the advertisement delivery area when the advertisement delivery system plays the advertisements, and obtain the video stream data related to each advertisement. (3) obtain a switching instruction issued by the users who watch the advertisements when the advertisement delivery system plays the advertisements, and the switching instruction includes a keyboard input instruction, a voice interaction instruction or a gesture interaction instruction; and when the acquisition is successful, the feature quantity SW representing the switching instruction is assigned to 1, otherwise the SW is assigned to 0.

[0183] The facial recognition module is used to obtain an image data set based on the video stream data through frame processing, and extract the facial features of each user appearing in the image data set; complete the facial features of the current user with the facial features of each historical user in the advertising analysis database, and distinguish between new users and historical users.

[0184] The image recognition module is used to perform image recognition on the image data set obtained by frame processing of the video stream data, and then: (1) obtain various feature data reflecting the identity characteristics of the newly added users. (2) extract the expressions of all users during the advertisement playback and classify the expressions into one of like, ignore or hate.

[0185] The speech recognition module is used to perform speech recognition on the speech stream data, and then: (1) obtain the speech interaction instruction issued by the user during the advertisement playing, which represents the request to switch the currently playing advertisement. (2) extract all the words in the speech stream data, and find out the keywords that match the feature data in the keyword data set.

[0186] The video action recognition module is used to perform video action recognition on video stream data, and then: (1) extract the gesture interaction instructions issued by a certain user in the video stream data, which represent the request to switch the currently playing advertisement. (2) Extract the gesture actions issued by a certain user in the video stream data, which represent the feedback to the currently playing advertisement. (3) Extract the characteristic actions reflecting the change of the eye focus position of a certain user during the current advertisement playback.

[0187] The user tag establishment module is used to create an empty user tag for each newly identified user, and to add various feature data reflecting the identity features of the newly identified user acquired by the image recognition module to the identity tag of the corresponding user.

[0188] The recognition evaluation value calculation module is used to: (1) obtain keywords that match the feature data in the keyword data set identified by the voice recognition module from the voice stream data, and count their number N1. (2) obtain gestures that reflect the user's feedback on the currently playing advertisement identified by the video action recognition module, and count their number N2. (3) obtain feature actions that reflect the change of the eye focus position of a certain user during the current advertisement playback process identified by the video action recognition module, and calculate the current user's attention time t on the currently playing advertisement based on the feature actions. n ; Where n represents the user number of the current user. The recognition evaluation value calculation module calculates the attention time t of the user numbered n on the currently played advertisement n The calculation formula is as follows:

[0189]

[0190] In the above formula, t 1n Indicates the viewing time of the user numbered n during the current advertisement playback; t 2n Indicates the closing time of the user numbered n during the current advertisement playback; t 3n Indicates the time that user number n keeps his head down during the current ad playback; t 4n Indicates the head turning duration of user number n during the current ad playback.

[0191] (4) Obtain the number of three-category expression classification results of each user identified by the image recognition module, and calculate the proportion of the number of three-category expression classification results of each user in the total sample size. (5) Obtain the value of SW. (6) Calculate the recognition evaluation value E of each user for the current advertisement by the following formula: n :

[0192]

[0193] In the above formula, n represents the user ID of the current user, E n represents the evaluation value of the currently playing advertisement by the user numbered n, E n ≥0, and E n The larger the value of reflects the higher the user's recognition of the currently played multimedia; represents the attention concentration of the user numbered n on the currently playing advertisement; k1 represents the influence factor of voice information feedback on the overall recognition evaluation result; k2 represents the influence factor of gesture feedback on the overall recognition evaluation result; k3 represents the influence factor of expression feedback on the overall recognition evaluation result; k4 represents the influence factor of attention concentration on the overall recognition evaluation result; m1 represents the score of a single keyword in voice information feedback; m2 represents the score of a single gesture in gesture feedback; m3 represents the score of attention concentration; a represents the score of liking expression, p 1,n is the ratio of the number of images sampled every other frame that the user n classifies as liking the expression; b is the score of ignoring the expression, p 2,n represents the proportion of expressions classified as ignored by user number n in the total number of images sampled every other frame; c represents the score of disgust expression, p 3,n It represents the proportion of expressions classified as disgust by user number n in the total number of images sampled every other frame.

[0194] The database creation module is used to: (1) set E according to expert experience n A high threshold E h and a low threshold E l ; Among them, E h Indicates the critical value of the user liking the currently playing advertisement, E l Indicates the critical value of the user's disgust with the currently playing advertisement, E l > 0. (2) Make the following judgments and decisions for each user: (i) When E n ≥E h And p 1，n +p 2，n ≥p 3，n , the feature data in the keyword data set associated with the currently playing advertisement is added to the preference tag corresponding to the current user, and the feature data of the supplemented preference tag is deduplicated; then the feature data in the dislike tag corresponding to the current user that is identical to the feature data in the keyword data set is deleted. (ii) When E≤E l And p 2，n +p 3，n ≥p 1，n When the feature data in the keyword data set associated with the currently playing advertisement is added to the dislike tag corresponding to the current user, the feature data of the supplemented dislike tag is deduplicated; then the feature data in the preference tag corresponding to the current user that matches the feature data in the keyword data set is deleted. (3) The user tag of each user is updated in turn to obtain a new user portrait data set for each user, thereby completing the creation or update of the advertisement analysis database. The user portrait data set includes the facial feature data and user tag of the corresponding user.

[0195] The advertising analysis database in this embodiment is empty at the beginning of creation. After the user portrait dataset of the first historical user is entered into it, the creation system of the advertising analysis database determines whether the current user is a new user or a historical user by comparing the facial features of the current user with the facial features of historical users in the advertising analysis database; and enters the user portrait dataset of the distinguished new user into the advertising analysis database, or updates the user label in the user portrait dataset of historical users already in the advertising analysis database.

[0196] Example 4

[0197] Based on the above-mentioned embodiment, this embodiment provides a method for evaluating the user's recognition of advertisements based on feature recognition, such as Fig. 9 As shown, the method comprises the following steps:

[0198] Step 1: Get the feature data of the currently playing ad

[0199] The playing time T of each advertisement played and the keyword data set associated with each advertisement are obtained.

[0200] The characteristic data in the keyword data set are a plurality of pre-set keywords related to the content of the currently played advertisement. The characteristic data in the keyword data set of each advertisement at least includes:

[0201] (1) Keywords that reflect the product being advertised.

[0202] (2) Keywords that reflect the target customer group of the advertisement.

[0203] (3) Keywords that reflect the advertising spokesperson or advertising character.

[0204] (4) High-frequency or special keywords in advertising slogans.

[0205] (5) Classification of advertising duration.

[0206] (6) Classification of advertising styles.

[0207] Step 2: Obtain feedback data from each user on the ad playback

[0208] 1. Acquire the voice stream data generated by all users in the advertising delivery area during the advertising playback, monitor the video stream data of all users in the advertising delivery area, and the instructions issued by one or more users in the advertising delivery area to switch the currently playing advertisement.

[0209] The instructions issued by the user to switch the currently playing advertisement include key input, voice interaction and gesture interaction. Voice interaction is achieved by identifying the voice keywords issued by the user to switch the currently playing advertisement; gesture interaction is achieved by identifying the characteristic gestures issued by the user to switch the currently playing advertisement; key input refers to the key input instruction directly input by the user through keys to switch the currently playing advertisement.

[0210] Voice keywords are obtained by a voice recognition algorithm based on real-time voice stream data; feature gestures are obtained by a video action recognition algorithm based on real-time video stream data; key input instructions are obtained through a physical switch key module installed at the advertising playback site.

[0211] In this embodiment, the user's feedback mainly includes the following aspects:

[0212] (1) Changes in users’ facial expressions when watching advertisements.

[0213] (2) Direct discussion of the advertisement by users. For example, discussing an actor or spokesperson in the advertisement, discussing the effects of the product, etc.

[0214] (3) Gestures made by users when watching advertisements. For example, a user's hand directly points to the advertisement playback device to prompt other users to pay attention, which reflects that the user is concerned about the currently playing advertisement.

[0215] (4) The amount of time a user focuses on an advertisement.

[0216] (5) The user requests to switch the currently playing advertisement. This directly reflects that the user does not like the advertisement.

[0217] In addition, when technical conditions are mature, other types of feedback can also be extracted and applied to later data analysis, such as the user's laughter and other detailed characteristic movements.

[0218] 2. Determine whether an instruction to switch the currently playing advertisement is received. If yes, assign 1 to the feature quantity SW reflecting the instruction; otherwise, assign 0 to SW.

[0219] Step 3: Calculate each user's recognition rating of the current advertisement

[0220] 1. Perform speech recognition on the speech stream data, extract the keywords that match the feature data in the keyword data set, and count their number N1.

[0221] 2. Perform video action recognition on the video stream data; extract gestures and actions representing the user's feedback on the currently playing advertisement, and count their number N2.

[0222] Among them, the gestures and actions of the user to give feedback on the currently playing advertisement include nodding, clapping, pointing the hand to the advertisement playing interface during the advertisement playing, raising or turning the head from a non-direct viewing state to a direct viewing state, etc.

[0223] 3. Perform video action recognition on the video stream data; extract the characteristic actions that reflect the changes in the eye focus position of each user, and calculate the attention time t of each user on the currently playing advertisement based on the characteristic actions n ; Where n represents the user ID of the current user.

[0224] The time t that the user numbered n pays attention to the currently playing ad n The calculation method is as follows:

[0225]

[0226] In the above formula, t 1n Indicates the viewing time of the user numbered n during the current advertisement playback; t 2n Indicates the closing time of the user numbered n during the current advertisement playback; t 3n Indicates the time that user number n keeps his head down during the current ad playback; t 4n Indicates the head turning duration of user number n during the current ad playback.

[0227] In this embodiment, when counting the time that the user pays attention to the advertisement, both the time that the user looks directly at the advertisement playback interface and the time that the user does not look directly at the advertisement are taken into account. In this embodiment, a relatively accurate attention time can be obtained by excluding the time that is determined to be in a non-attention state and then averaging the time that is determined to be in an attention state.

[0228] 4. Sample the framed images of the video stream data at alternate frames according to the sampling frequency; perform image recognition on the sampled images; extract the facial expressions of each user, and classify the facial expressions into like, ignore or disgust; count the number of the three types of expression classification results of each user respectively, and calculate the proportion of the three types of expression classification results of each user in the total sample size of the user.

[0229] 5. Get the value of the SW.

[0230] 6. Calculate the recognition evaluation value E of each user for the current advertisement using the following formula: n :

[0231]

[0232] In the above formula, n represents the user ID of the current user, E nrepresents the evaluation value of the currently playing advertisement by the user numbered n, E n ≥0, and E n The larger the value of reflects the higher the user's recognition of the currently played multimedia; represents the attention concentration of the user numbered n on the currently playing advertisement; k1 represents the influence factor of voice information feedback on the overall recognition evaluation result; k2 represents the influence factor of gesture feedback on the overall recognition evaluation result; k3 represents the influence factor of expression feedback on the overall recognition evaluation result; k4 represents the influence factor of attention concentration on the overall recognition evaluation result; m1 represents the score of a single keyword in voice information feedback; m2 represents the score of a single gesture in gesture feedback; m3 represents the score of attention concentration; a represents the score of liking expression, p 1,n is the ratio of the number of images sampled every other frame that the user n classifies as liking the expression; b is the score of ignoring the expression, p 2,n represents the proportion of expressions classified as ignored by user number n in the total number of images sampled every other frame; c represents the score of disgust expression, p 3,n It represents the proportion of expressions classified as disgust by user number n in the total number of images sampled every other frame.

[0233] The method provided in this embodiment can identify various types of features of feedback from users when playing advertisements, and then obtain the user's recognition evaluation of the advertisement. This method can collect various types of user feedback, and the obtained user recognition evaluation result of the advertisement is more accurate, which can be used as a basis for evaluating the effect of advertising.

[0234] Example 5

[0235] This embodiment provides a method for timely analyzing user needs in a business district scenario. This method is further developed on the method in Embodiment 4 and achieves the most direct and rapid prediction or evaluation of user needs of specific users. Fig.10 As shown, the method comprises the following steps:

[0236] Step 1: Obtain the facial features of the current user in the advertising delivery area.

[0237] Step 2: Perform facial recognition on the current user in turn, and query an advertising analysis database (the advertising analysis database is the advertising analysis database in the aforementioned embodiment) containing a user portrait data set of multiple historical users based on the facial recognition results, and make the following judgment:

[0238] (1) When the facial features of the current user match the feature data in one of the historical user's facial feature data, all feature data in the user tag of the historical user are obtained.

[0239] (2) When the facial features of the current user do not match the feature data in the facial feature data of all historical users, the current user is determined to be a new user, and an empty user tag is created for the new user.

[0240] Among them, the user portrait dataset contains the facial feature data and user tags of the corresponding historical users; user tags include identity tags, preference tags and dislike tags.

[0241] Step 3: Acquire multi-angle images of the newly added user, perform image recognition on the multi-angle images, and supplement the feature data in the identity tag of the newly added user based on the recognition result; wherein the feature data supplemented in the identity tag includes user number, gender, age group, dressing style and other features; other features represent identifiable non-gender, age group and dressing style features that are useful for distinguishing user identity features.

[0242] Step 4: Compare all feature data in the identity tag with the identity tags of each historical user in the advertisement analysis database, and calculate the feature overlap degree Dc3 between the two. The calculation formula of Dc3 is as follows:

[0243]

[0244] Step 5: Extract the feature data from the like and dislike tags of the historical user with the largest feature overlap Dc3 value with the current user in the advertising analysis database, and fill the above feature data into the user portrait dataset of the newly added user to complete the timely analysis process of the user needs of the current user.

[0245] By analyzing the above process, it can be found that the method in this embodiment can analyze and identify the user as soon as he or she appears, and then establish an estimated feature and behavior portrait data set to predict the user's likes and dislikes; and based on this prediction, timely analysis of user needs can be achieved. This analysis method is more timely and effective, and there is no need to "track" and evaluate users for a long time. Therefore, it has a high practical value. At the same time, it should be noted that the accuracy of this timely analysis result is highly correlated with the sample size in the advertising analysis database containing the user portrait data set of multiple historical users. The larger the sample size of the advertising analysis database, the more accurate the result of this timely analysis.

[0246] The logic of the method of this embodiment is to first obtain the facial features of the user appearing in a specific scene, determine whether the data sample of the user has been included in the advertising analysis database, and directly extract the content of the user's like and dislike tags included in the advertising analysis database as the user portrait data set of the user, and then analyze and predict the user needs of the user. When the data sample of the user is not included in the advertising analysis database, first extract the identity features of the user, and then extract the like and dislike tags from the identity tags of the historical users whose identity features are most similar to the user (determined by Dc3) among the historical users included in the advertising analysis database, and use them as the user portrait data set of the current user, and then analyze the user needs of the user.

[0247] Example 6

[0248] This embodiment provides a method for matching user needs with advertisement content. The method is developed on the basis of the above-mentioned embodiment and is used to select the advertisement that best matches the current user from the advertisements to be delivered. Fig.11 As shown, the matching method includes the following steps:

[0249] Step 1: Obtain a keyword data set of all advertisements to be delivered; the keyword data set is the keyword data set established in any of the aforementioned embodiments; the keyword data set includes keywords reflecting various characteristic data of the advertisement content.

[0250] Step 2: Obtain a user portrait data set of the current user. The user portrait data set is the final result obtained by the method for timely analyzing user needs in a business district scenario provided in the aforementioned embodiment 5;

[0251] Step 3: Calculate the matching degree Dc4 between the feature data in the keyword data set of each advertisement and the data in the current user portrait data set. The calculation formula of Dc4 is as follows:

[0252]

[0253] Step 4: The advertisement with the largest value of Dc4 is taken as the advertisement that best matches the current user, completing the matching process between user demand and advertisement content.

[0254] The matched advertisements are most compatible with the actual needs of the users, and thus the best promotion effect can be achieved. In practical applications, the most compatible advertisements should be given priority for the identified current users.

[0255] The method for matching user needs and advertising content used in this embodiment is feature matching. In this feature matching process, the features that characterize user needs (features in preference characterization) are derived based on user feedback during the playback of historical advertisements, and the feature data is the keyword of the corresponding advertisement. Therefore, when matching features with actual advertisements to be placed, it is usually very easy to match successfully, and considering that user preferences are usually consistent and long-term, the result of this feature matching will also be more accurate.

[0256] Example 7

[0257] This embodiment provides a garage giant screen MAX smart terminal with intelligent voice interaction function, which is used to update the advertisements to be placed in the advertisement playback sequence list according to the process of the user interacting with itself when playing advertisements. The solution of this embodiment belongs to the in-depth development and application of the technical solutions and achievements in the above-mentioned embodiments. The garage giant screen MAX smart terminal with intelligent voice interaction function in this embodiment adopts some of the processing methods and device modules in the above-mentioned embodiments.

[0258] Specifically, Fig.12 As shown, the garage giant screen MAX intelligent terminal provided in this embodiment includes: an advertising playback module, a voice acquisition module, a video monitoring module, an advertising feature data extraction module, a user feedback data extraction module, an image recognition module, a voice recognition module, a video action recognition module, a human-computer interaction module, a recognition evaluation value calculation module and an advertising playback sequence update module.

[0259] The advertisement playing module is used to play each advertisement to be put in sequence according to the advertisement playing sequence table, and switch the advertisement being played after receiving a switching instruction issued by the human-computer interaction module. The advertisement playing module is the Garage Giant Screen MAX display screen.

[0260] The voice collection module is used to collect voice information generated by the user group watching the advertisement around the advertisement playing module when the advertisement playing module plays each advertisement. The voice collection module is a plurality of microphones installed around the Garage Giant Screen MAX display screen; the microphones are distributed on the side facing the display surface of the Garage Giant Screen MAX display screen.

[0261] The video monitoring module is used to monitor the user group watching the advertisements around the advertisement playing module from multiple angles when the advertisement playing module plays each advertisement. The viewing range of the video monitoring module is the side facing the display surface of the garage giant screen MAX display screen. The video monitoring module includes multiple monitoring cameras, and each monitoring camera shoots the viewing range from different angles.

[0262] The advertisement feature data extraction module is used to extract the playing time T of each advertisement played by the advertisement playing module, and a keyword data set associated with the advertisement.

[0263] The user feedback data extraction module is used to: (1) receive the voice information collected by the voice collection module to obtain the voice stream data related to each advertisement. (2) receive the multi-angle monitoring video collected by the video monitoring module to obtain the video stream data related to each advertisement. (3) obtain the switching instruction issued by the human-computer interaction module to switch the currently playing advertisement, and when the switching instruction is received, assign the feature quantity SW representing the switching instruction to 1, otherwise assign SW to 0.

[0264] The image recognition module is used to perform image recognition on the image data set obtained by frame processing of the video stream data, and then extract the expressions of all users during the advertisement playing, and classify the expressions into one of like, ignore or hate. The image recognition module includes an expression recognition unit, which uses a neural network recognition algorithm trained with a large number of training sets to complete the classification process of the user's expression in the image.

[0265] The speech recognition module is used to perform speech recognition on the speech stream data, and then: (1) obtain the voice interaction command issued by the user during the advertisement playback, which represents the request to switch the currently playing advertisement. (2) Extract all the words in the speech stream data and find out the keywords that match the feature data in the keyword data set.

[0266] Among them, the speech recognition module includes a speech interaction instruction extraction unit and a keyword extraction unit. The speech interaction instruction extraction unit sends the extracted speech interaction instructions to the speech interaction unit in the human-computer interaction module; the keyword extraction unit sends the extracted keywords that match the feature data in the keyword data set to the recognition evaluation value calculation module.

[0267] The video action recognition module is used to perform video action recognition on video stream data, and then: (1) extract the gesture interaction instructions issued by a certain user in the video stream data, which represent the request to switch the currently playing advertisement; (2) extract the gesture actions issued by a certain user in the video stream data, which represent the feedback to the currently playing advertisement; (3) extract the characteristic actions reflecting the change of the eye focus position of a certain user during the current advertisement playing process.

[0268] Among them, the video action extraction module includes a gesture interaction instruction extraction unit, a posture action feedback extraction unit, and an eye feature action extraction unit; the gesture interaction instruction extraction unit sends the extracted gesture interaction instruction to the gesture interaction unit in the human-computer interaction module; the posture action feedback extraction unit and the eye feature action extraction unit send the extracted feature data to the recognition evaluation value calculation module.

[0269] The human-computer interaction module is used to obtain an instruction from a user requesting to switch the currently playing advertisement, and issue a switching instruction; wherein, Fig.13 As shown, the user's request to switch the currently playing advertisement includes key input, voice interaction and gesture interaction. Among them, the human-computer interaction module includes a physical key module, which is used to receive key input instructions directly issued by the user to switch the currently playing advertisement; the human-computer interaction module also includes a voice interaction unit and a gesture interaction unit; the voice interaction unit is used to obtain the voice interaction instruction issued by the user to switch the currently playing advertisement, and the voice interaction instruction is obtained by the voice recognition module according to the real-time voice stream data through voice recognition; the gesture interaction unit is used to obtain the gesture interaction instruction issued by the user to switch the currently playing advertisement, and the gesture interaction instruction is obtained by the video action recognition module according to the real-time video stream data through video action recognition.

[0270] The recognition evaluation value calculation module is used to: (1) obtain the keywords identified by the speech recognition module that match the feature data in the keyword data set, and count their number N1. (2) obtain the gestures and actions that represent the user's feedback on the currently playing advertisement identified by the video action recognition module, and count their number N2. (3) obtain the feature actions that reflect the changes in the eye focus position of a certain user during the current advertisement playback process identified by the video action recognition module, and calculate the current user's attention time t on the currently playing advertisement based on the feature actions. n ; Attention duration t n The calculation formula is as follows:

[0271]

[0272] In the above formula, t 1n Indicates the viewing time of the user numbered n during the current advertisement playback; t 2n Indicates the closing time of the user numbered n during the current advertisement playback; t 3n Indicates the time that user number n keeps his head down during the current ad playback; t 4n Indicates the head turning time of user number n during the current advertisement playback. (4) Obtain the number of three-category expression classification results of each user identified by the image recognition module, and calculate the proportion of the number of three-category expression classification results of each user in the total sample size. (5) Obtain the value of SW. (6) Calculate the recognition evaluation value E of each user for the current advertisement by the following formula n :

[0273]

[0274] In the above formula, n represents the user ID of the current user, E n represents the evaluation value of the currently playing advertisement by the user numbered n, E n ≥0, and E n The larger the value of reflects the higher the user's recognition of the currently played multimedia; represents the attention concentration of the user numbered n on the currently playing advertisement; k1 represents the influence factor of voice information feedback on the overall recognition evaluation result; k2 represents the influence factor of gesture feedback on the overall recognition evaluation result; k3 represents the influence factor of expression feedback on the overall recognition evaluation result; k4 represents the influence factor of attention concentration on the overall recognition evaluation result; m1 represents the score of a single keyword in voice information feedback; m2 represents the score of a single gesture in gesture feedback; m3 represents the score of attention concentration; a represents the score of liking expression, p 1,n is the proportion of expressions classified as liked by user n in the total number of images sampled every other frame; b represents the score of ignoring expressions, p 2,n is the proportion of expressions classified as ignored by user number n in the total number of images sampled every other frame; c represents the score of disgust expression, p 3,n The percentage of expressions classified as disgust by user number n in the total number of images sampled every other frame.

[0275] The advertisement play sequence update module is used to: (1) obtain the average recognition evaluation results of each advertisement in the advertisement sequence table of all advertisements played within an update cycle; The calculation formula is as follows:

[0276]

[0277] In the above formula, i represents the number of each advertisement in the advertisement play sequence table. (2) According to the (3) Obtain the advertisements that need to be added and their quantity, delete the corresponding number of advertisements that are ranked at the bottom of the score ranking table from the advertisement play sequence table, and add the advertisements that need to be added to the advertisement play sequence table to complete the update process of the advertisement play sequence table.

[0278] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating user recognition of advertisements based on feature recognition, characterized in that: The steps include: S1: Get the characteristic data of the currently playing advertisement: Obtain the playing time T of each advertisement played, and the keyword data set associated with each advertisement; S2: Obtaining feedback data from each user on the advertisement playback, including: Acquire the voice stream data generated by all users in the advertising delivery area during the advertisement playback, monitor the video stream data of all users in the advertising delivery area, and the instructions issued by one or more users in the advertising delivery area to switch the currently playing advertisement; determine whether the instruction to switch the currently playing advertisement is received: if yes, assign 1 to the feature quantity SW reflecting the instruction; otherwise, assign 0 to SW; S3: Calculate the recognition evaluation value of each user for the current advertisement, including the following process: S31: Perform speech recognition on the speech stream data, extract keywords that match the feature data in the keyword data set, and count their number N1; S32: Perform video action recognition on the video stream data; extract gesture actions representing the user's feedback on the currently playing advertisement, and count the number N2 of gesture actions; S33: Perform video action recognition on the video stream data; extract the characteristic actions that reflect the changes in the eye focus position of each user, and calculate the attention time t of each user on the currently playing advertisement based on the characteristic actions. n ; Where n represents the user ID of the current user; S34: sampling the framed images of the video stream data at alternate frames according to the sampling frequency; performing image recognition on the sampled images at alternate frames; extracting the facial expressions of each user, and classifying the facial expressions into like, ignore or hate; counting the number of the three types of expression classification results of each user respectively, and calculating the proportion of the number of the three types of expression classification results of each user in the total sample size of the user; S35: Get the value of SW; S36: Calculate the recognition evaluation value E of each user for the current advertisement by the following formula: n : In the above formula, n represents the user ID of the current user, E n represents the evaluation value of the currently playing advertisement by the user numbered n, E n ≥0, and E n The larger the value of reflects the higher the user's recognition of the currently played multimedia; represents the attention concentration of the user numbered n on the currently playing advertisement; k1 represents the influence factor of voice information feedback on the overall recognition evaluation result; k2 represents the influence factor of gesture feedback on the overall recognition evaluation result; k3 represents the influence factor of expression feedback on the overall recognition evaluation result; k4 represents the influence factor of attention concentration on the overall recognition evaluation result; m1 represents the score of a single keyword in voice information feedback; m2 represents the score of a single gesture in gesture feedback; m3 represents the score of attention concentration; a represents the score of liking expression, p 1,n is the ratio of the number of images sampled every other frame that the user n classifies as liking the expression; b is the score of ignoring the expression, p 2,n represents the proportion of expressions classified as ignored by user number n in the total number of images sampled every other frame; c represents the score of disgust expression, p 3,n It represents the proportion of expressions classified as disgust by user number n in the total number of images sampled every other frame.

2. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: In step S1, the associated words are set before each advertisement is released, and the characteristic data in the keyword data set associated with each advertisement includes at least: (1) Keywords that reflect the product being advertised; (2) Keywords that reflect the target customer group of the advertisement; (3) Keywords that reflect the spokesperson or character image of the advertisement; (4) High-frequency or special keywords in advertising words; (5) Classification of advertisement duration; (6) Classification of advertising styles.

3. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: In step S2, the methods of the user's instruction to switch the currently playing advertisement include key input, voice interaction and gesture interaction; the voice interaction is achieved by identifying the voice keywords issued by the user to request switching the currently playing advertisement; the gesture interaction is achieved by identifying the characteristic gestures issued by the user to request switching the currently playing advertisement; the key input represents the key input instruction requested to switch the currently playing advertisement input directly by the user through keys.

4. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 3, characterized in that: In step S2, voice keywords are obtained by a voice recognition algorithm based on real-time voice stream data recognition; the characteristic gestures are obtained by a video action recognition algorithm based on real-time video stream data; and key input instructions are obtained through a physical switch key module installed at the advertisement broadcasting site.

5. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: In step S2, the user's feedback data on the advertisement playback includes the following contents: (1) Changes in users’ facial expressions when watching an ad, including liking, ignoring, or disliking it; (2) whether the user's direct discussion of the advertisement involves the content in the keyword data set corresponding to the advertisement; (3) The gestures made by users when watching the advertisement; (4) the amount of time a user focuses on an advertisement; (5) Whether the user requests to switch the currently playing advertisement.

6. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 5, characterized in that: The time a user's attention span when watching an advertisement is calculated by the following formula: In the above formula, t n Indicates the time that the user numbered n pays attention to the currently playing advertisement; t 1n Indicates the viewing time of user number n during the current ad playback; t 2n Indicates the closing time of the user numbered n during the current advertisement playback; t 3n Indicates the time that user number n keeps his head down during the current ad playback; t 4n Indicates the head turning duration of user number n during the current ad playback.

7. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: In step S32, the gestures of the user giving feedback to the currently playing advertisement include nodding, clapping, pointing the hand to the advertisement playing interface, and raising or turning the head from a non-direct viewing state to a direct viewing state during the advertisement playing.

8. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: The voice stream data is obtained through a pickup installed at the advertisement playing device.

9. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: The video stream data is obtained through video monitoring equipment installed around the advertisement playing device.

10. The method for evaluating the user's acceptance of advertisements based on feature recognition according to claim 1, characterized in that: In step S34, expression recognition is performed using a neural network algorithm trained with a large number of samples.

Images