A method and system for intelligent interactive media display

Abstract

This invention provides an intelligent interactive media display method and system. The method includes: offline construction of a visitor image library; capturing faces of people standing in front of the media wall using a camera installed on the media wall, and extracting qualified facial images using face detection technology; then, using face analysis technology to perform attribute analysis on the extracted qualified facial images to determine the gender, age, and identity attributes of the faces in front of the media wall; combining the constructed visitor image library with the results of face detection and analysis to intelligently display the media wall; and using gesture recognition technology to allow users to interact with the media wall without contact, including pausing, starting, scrolling forward, scrolling backward, and deleting. This invention innovatively applies face recognition and gesture recognition technologies to building walls, intelligently displaying media wall content and providing a contactless interactive method, giving users not only a sense of presence and surprise, but also a sense of interaction and friendliness.

Description

Technical Field

[0001] This invention relates to an intelligent interactive media display method and system based on face recognition and motion recognition, belonging to the field of intelligent recognition and multimedia display. Background Technology

[0002] Exterior media walls, as an emerging architectural form and media display method, are gaining increasing popularity. The integration of new media technologies with physical architecture transforms them from mere buildings into multimedia spaces for viewing and listening. In recent years, artificial intelligence technology has experienced unprecedented development; therefore, integrating AI into exterior media walls to enhance their intelligence and interactivity holds profound significance.

[0003] Furthermore, with the improvement of people's material and spiritual living standards, theme parks and outdoor recreational activities are becoming increasingly popular. After enjoying the fun, people often want to record the wonderful moments along the way as souvenirs, share them on social media, and reminisce. However, during rides such as rafting, log flumes, mine trains, pirate ships, space shuttle rides, roller coasters, and cable cars in theme parks or outdoor recreational activities, visitors cannot use their own cameras to take pictures. Therefore, entertainment venues need to provide a service where photographers or cameras at different attractions capture and monitor visitors, recording their various states during the ride and compiling the data into a computer. After the ride, visitors can select the photos they want from the computer at the attraction. This is a great suggestion and business opportunity, but it is difficult to implement. The variable subjects and the sheer volume of photos make manual photo selection very difficult and inefficient. Therefore, a software system that intelligently sorts, displays, and prints massive amounts of photos is needed to replace manual selection, display, and printing, thereby improving efficiency and saving manpower.

[0004] If artificial intelligence can be combined with building exterior media wall technology to intelligently display and self-print tourists' photos, it would not only serve a promotional purpose but also reduce labor costs and improve user experience. Therefore, this invention proposes an intelligent interactive media display method and system based on facial recognition. Summary of the Invention

[0005] This invention utilizes artificial intelligence, building facade display, and facial analysis and recognition technologies to propose an intelligent interactive media display method, comprising the following steps:

[0006] Step 1: Build an offline tourist image library, collect tourist photos, use the corresponding standard facial images as the identity tags of each tourist, and create an image folder for each tourist, and store tourist images by gender and age.

[0007] Step 2: Use the camera installed on the media wall to capture the faces of people standing in front of the building's media wall, and use face detection technology to extract qualified face images. Then, use face analysis technology to perform attribute analysis on the extracted qualified face images to determine the gender, age, and identity attributes of the faces in front of the building's media wall.

[0008] Step 3: Based on the constructed tourist image library, intelligently display the media wall according to the results of face detection and analysis, and assign each displayed image a memorable number.

[0009] Step 4: Use gesture recognition technology to enable users to interact with the media wall without contact, including pausing, starting, scrolling forward, scrolling backward, and deleting.

[0010] Furthermore, the specific implementation method of step 1 is as follows:

[0011] Step 11: Install cameras at the entrance of the theme park or amusement park, use a face detection model to collect a standard frontal face image as the identity label of each visitor, and create an image folder for each visitor.

[0012] Step 12: Use photographers and the venue's surveillance cameras to capture images of each visitor's experience;

[0013] Step 13: The massive amount of collected travel photos are categorized and stored in corresponding folders according to the identity tags established in Step 11 to create a tourist travel image library. In addition, the massive amount of collected travel photos are subjected to facial gender and age recognition, and then stored in categories according to gender and age.

[0014] Furthermore, the specific implementation method of the face detection model in step 11 is as follows:

[0015] The face detection model described is implemented using a lightweight detection model. Its network structure consists of three main parts: a backbone network, a multi-scale network (FPN), and a detection head. First, the backbone network extracts features from the input video image. Then, the multi-scale network fuses the features extracted by the backbone network at multiple scales. Finally, the detection head performs face detection and localization on the fused features. The backbone network uses MobileNetv3 as the backbone, the multi-scale network adopts a top-down structure, and the detection head uses a four-conv+bn structure, which outputs the location confidence, the scale of the detection box, the offset, and the facial key points, respectively.

[0016] The loss function of the face detection model is the same as that of CenterNet, consisting of a FocalLoss-type location classification loss and an L1 regression loss. The location classification loss function is as follows:

[0017]

[0018] In the formula Y xyc The true center point category is either 1 or 0. The confidence level of the prediction, where α and β are constants;

[0019] The scale loss function is:

[0020]

[0021] Among them, s k The actual dimensions of the target bounding box. The dimensions of the predicted target bounding box are given by N, and the number of training samples is given by N.

[0022] The offset loss function is:

[0023]

[0024] in, (x k ,y k ) represents the coordinates of the center point, and smoothL1(·) represents smooth L1 regression;

[0025] The Points loss function is:

[0026]

[0027] in, The difference in distance between the key point and the center point. The distance difference between the predicted keypoint and the center point; (c x ,c y (p) represents the coordinates of the center point. x ,p y (box) represents the coordinates of facial landmarks. w ,box h ) represents the length and width of the face detection bounding box; the final loss function is:

[0028] L = L location +λ1L scale +λ2L offset +λ3L points .

[0029] Furthermore, in step 13, the multi-task neural network MDCNN is used to simultaneously identify face gender and age. The network model design is as follows:

[0030] Suppose we use N training samples to learn T=2 tasks, using Let represent the i-th sample in the t-th task ({1,…,T}) and its corresponding task label. Then, the learning objective of MDCNN is:

[0031]

[0032] Where, f(x) i (,θ) represents the deep network function, where θ is the network parameter to be solved, l t (·) represents the loss function for different tasks, and Φ(θ) is the regularization penalty term;

[0033] For the gender identification problem, since it is a binary classification problem, the softmax loss function is directly adopted: l G = -(1-g)·log(1-p0)-g·log(p1), where g = 0 or 1 indicates male or female, and (p0,p1) represents the probability of each gender;

[0034] For the face age recognition problem, since it is a multi-class classification problem, the loss function is center loss:

[0035]

[0036] Where, x k Let y represent the deep learning feature of the k-th sample in a mini-bach. k Indicates its corresponding category; c yk Indicates y k The deep learning feature centers of the class, W and b represent the weights and bias parameters in the last fully connected layer, respectively; m and n represent the mini-batch and class sizes, respectively; and λ is the balance factor.

[0037] Therefore, the final multi-task joint loss function is:

[0038] loss full =β1l G +β2l A

[0039] Among them, β1 and β2 are balance factors.

[0040] Furthermore, the specific implementation method of step 2 is as follows:

[0041] Face analysis technology employs a multi-task learning network based on lifelong learning. To facilitate knowledge transfer and avoid catastrophic forgetting, for each task, after training for that task, the importance Ω of each parameter in the network to that task is calculated. ij Ω represents the proportion of the parameter value in the i-th row and j-th column to the total parameter values, and this proportion is used in subsequent training tasks. ij It is added to the loss function as a regularization term whenever a new task is trained: for Ω ij For larger parameters, minimize their changes during gradient descent, as these parameters are important for past tasks and their values ​​need to be preserved to avoid catastrophic forgetting; while for Ω... ij Smaller parameters allow for gradient updates with larger magnitudes, resulting in better performance on new tasks. Therefore, the loss function for the nth task is:

[0042]

[0043] Among them, L n (θ) is the loss function for the current task, λ is the balance factor, and θ ij These are the parameters in the i-th row and j-th column of the current model parameter matrix. Ω consists of model parameters obtained after training the first n-1 tasks. After training on each task, Ω... ij All will be updated, where n=3, and the training order of the three tasks is gender recognition, age recognition, and identity recognition. When performing the first task, Set to 0;

[0044] The backbone network of the multi-task network adopts RestNet or EfficientNet.

[0045] Furthermore, the specific implementation method of step 3 is as follows:

[0046] 31) Based on face detection technology, if no tourists are found in front of the media wall, the media wall will randomly play images of tourists playing, or play promotional videos, or stop playing to save electricity.

[0047] 32) If there is a face in front of the media wall, but the person is a new visitor, then intelligently play pictures of the visitor with a similar gender and age based on the face's gender and age.

[0048] 33) If there is a face in front of the media wall and it is a face in the visitor database, then the visitor's photos will be played intelligently based on their facial identity tags.

[0049] Furthermore, the specific implementation method of step 4 is as follows:

[0050] A gesture recognition method based on deep collaborative dictionary learning is used to achieve contactless interaction. The specific implementation process is as follows:

[0051] Feature extraction is performed on the input gesture image to be recognized. A lightweight feature extraction network is constructed using MobileNetV3 to propose feature vectors. m is the dimension of the feature vector;

[0052] The feature vector y is represented by a linear combination using a gesture-complete dictionary D, a deep sparse representation model is established, and the coefficient vector is solved:

[0053]

[0054] Where w represents the sparse representation coefficients of the demand solution. This is an approximation of w, where γ is a constant equilibrium factor, ||·|| 1 / 2 Represents the 1 / 2 norm;

[0055] Gesture classification and recognition are performed based on the coefficient vector of the linear combination obtained from the above formula. The gestures are 5 categories, defined as pause, start, forward flip, backward flip, and delete.

[0056] Furthermore, before representing the feature vector y using a linear combination of gesture-complete dictionaries, a training step for the gesture-complete dictionary is included. To improve the accuracy and robustness of the model, a deep collaborative learning dictionary training method is proposed.

[0057] Based on the principles of deep learning, the gesture training sample X is subjected to multi-level deep decomposition learning to obtain the gesture sub-dictionary D. c :

[0058]

[0059] Where, D = {D 1 ,…,D c ,…,D 5}, where X is the gesture training sample matrix; For different levels of the c-th type of gesture, D is the corresponding coefficient matrix; s W s These are the shared dictionary and its corresponding coefficient matrix, i.e., the parameters to be solved; according to the principle of nonnegative matrix factorization, we have:

[0060]

[0061] The above formula can be interpreted as performing a multi-level decomposition on the training sample X, which includes gesture information such as pause, start, forward, backward, and delete, to find a new representation. W s and new representation space

[0062] To enhance the non-linear representation capability of the dictionary, the coefficient matrix is ​​modified using a deep neural network approach:

[0063]

[0064] Where i represents the i-th layer, ranging from [1 to m], and g(·) is a non-linear activation function. Therefore, the objective of the deep collaborative dictionary learning model is:

[0065]

[0066] Furthermore, the deep collaborative dictionary learning model employs a training method similar to that of a stacked autoencoder network, consisting of two stages: layer-by-layer pre-training and overall fine-tuning.

[0067] (1) Layer-by-layer pre-training stage

[0068] A. Order Solving the minimization problem: Complete the first level of decomposition;

[0069] B. Regarding W1 c Continue to break down Complete the second level of decomposition;

[0070] This process continues until all layers are pre-trained. By using a greedy decomposition method layer by layer, the training of each layer becomes a traditional dictionary learning problem. Depending on the actual needs, sparse constraints or supervised learning can be added to the dictionary learning of each layer.

[0071] (2) Overall fine-tuning stage

[0072] This is achieved by minimizing the loss function and using stochastic gradient descent.

[0073] Furthermore, step 5, self-service printing, is included. If visitors see an image that interests them on the media wall and wish to print it as a souvenir, they can proceed with the following steps:

[0074] 51) Note down the numbers of the pictures that interest you;

[0075] 52) Enter the image number at the self-service printer;

[0076] 53) Choose a payment method, including online payment, WeChat payment, Alipay payment, bank transfer or cash payment;

[0077] 54) Select the printing method, including on-site printing, downloading to USB drive, or forwarding to email;

[0078] 55) Confirmation complete.

[0079] This invention also provides an intelligent interactive media display wall system, comprising the following modules:

[0080] The image organization and classification module is used to build an offline tourist image library, collect tourist photos, use the corresponding standard facial images as the identity tags of each tourist, and create an image folder for each tourist, and classify and store tourist images by gender and age.

[0081] The face analysis and recognition module is used to capture the faces of people standing in front of the building's media wall using cameras installed on the media wall, and to extract qualified face images using face detection technology. Then, face analysis technology is used to analyze the attributes of the extracted qualified face images to determine the gender, age, and identity attributes of the faces in front of the building's media wall.

[0082] The media wall smart display module is used to combine the constructed visitor image library to intelligently display the results of face detection and analysis on the media wall, and assign each displayed image a memorable number.

[0083] The intelligent interaction module is used to enable users to interact with the media wall without contact using gesture recognition technology, including pausing, starting, scrolling forward, scrolling backward, and deleting.

[0084] The present invention has the following advantages and effects compared with the prior art:

[0085] 1) Ingenious Media Display Wall: This invention innovatively applies facial recognition and gesture recognition technology to building walls to intelligently display media wall content and provide a contactless interactive method, so that users not only feel a sense of presence and surprise, but also a sense of interaction and friendliness.

[0086] 2) A novel face analysis method is proposed: In order to solve the problems of too many and too large models and long computation time in the deployment of face analysis models, this invention proposes to use a single model to perform face gender, age and identity recognition based on a lifelong learning method, so as to improve model efficiency and running time.

[0087] 3) A novel gesture recognition method is proposed: In order to improve the accuracy of gesture recognition, this invention proposes a gesture recognition method based on deep collaborative dictionary learning. Attached Figure Description

[0088] Figure 1 This is a flowchart of an intelligent interactive media display method.

[0089] Figure 2 This is a network structure diagram of a face detection model. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0091] The flowchart of the intelligent interactive media display method disclosed in this invention is as follows: Figure 1 As shown, it mainly includes five steps:

[0092] Step 1: Build an offline tourist image library, collect tourist photos, use the corresponding standard facial images as the identity tags of each tourist, and create an image folder for each tourist, and store tourist images by gender and age.

[0093] This step can be done offline in the background, mainly involving the collection and organization of visitor facial images. First, cameras are installed at the entrance of the theme park or amusement park. Facial detection technology is used to collect a standard (frontal, neutral expression, well-lit) facial image of each visitor as their identification tag, and an image folder is created for each visitor.

[0094] Then, photographers and surveillance cameras within the venue were used to capture images of each visitor's experience.

[0095] Finally, the massive amount of collected travel photos were categorized and stored in corresponding folders according to the identity tags established in the first step, creating a tourist image library. Additionally, the collected photos were further categorized and stored by gender and age group of the faces in the images.

[0096] Preferably, the massive amount of collected travel photos can be filtered to select high-resolution, good-viewing-angle, attractive, and promotional photos containing human faces for storage.

[0097] Preferably, in order to improve the speed of building a tourist sightseeing image database, the massive amount of collected sightseeing images are first clustered using image clustering and facial attribute analysis techniques. Based on the clustering, the images are then manually filtered and categorized to build the database.

[0098] The face detection technology is characterized by its lightweight detection model, which facilitates terminal deployment. Its network structure is as follows: Figure 2As shown, the model mainly consists of three parts: a backbone network, a multi-scale network (FPN), and a detection head. The backbone network first extracts features from the input video image, then the multi-scale network fuses these extracted features at multiple scales. Finally, the detection head uses the fused features for face detection and localization. Specifically, the backbone network uses MobileNetv3; the multi-scale network employs a top-down structure without using a panning module (PAN); and the detection head uses a four-conv+bn structure, outputting the location confidence score, bounding box scale, offset, and facial landmarks. The face detection model's loss function is the same as CenterNet, consisting of a FocalLoss-type location classification loss and an L1 regression loss. The location classification loss function is as follows:

[0099]

[0100] In the formula Y xyc The true center point category is either 1 or 0. The confidence level of the prediction, where α and β are constants;

[0101] The scale loss function is:

[0102]

[0103] Among them, s k The actual dimensions of the target bounding box. The dimensions of the predicted target bounding box are given by N, and the number of training samples is given by N.

[0104] The offset loss function is:

[0105]

[0106] in, (x k ,y k ) represents the coordinates of the center point, and smoothL1(·) represents smooth L1 regression;

[0107] The Points loss function is:

[0108]

[0109] in, The difference in distance between the key point and the center point. The distance difference between the predicted keypoint and the center point; (c x ,c y (p) represents the coordinates of the center point. x,p y (box) represents the coordinates of facial landmarks. w ,box h ) represents the length and width of the face detection bounding box; the final loss function is:

[0110] L = L location +λ1L scale +λ2L offset +λ3L points .

[0111] This invention employs a multi-task neural network (MDCNN) method for simultaneous facial gender and age recognition. The network model design is as follows:

[0112] Suppose we use N training samples to learn T=2 tasks, using Let represent the i-th sample in the t-th task ({1,…,T}) and its corresponding task label. Then, the learning objective of MDCNN is:

[0113]

[0114] Where, f(x) i (,θ) represents the deep network function, where θ is the network parameter to be solved, l t (·) represents the loss function for different tasks, and Φ(θ) is the regularization penalty term;

[0115] For the gender identification problem, since it is a binary classification problem, the softmax loss function is directly adopted: l G = -(1-g)·log(1-p0)-g·log(p1), where g = 0 or 1 indicates male or female, and (p0,p1) represents the probability of each gender;

[0116] For the face age recognition problem, since it is a multi-class classification problem, the loss function is center loss:

[0117]

[0118] Where, x k Let y represent the deep learning feature of the k-th sample in a mini-bach. k Indicates its corresponding category; c yk Indicates y kThe deep learning feature centers of the class are defined by W and b, which represent the weights and bias parameters in the final fully connected layer, respectively; m and n represent the mini-batch and class sizes, respectively; and λ is a balancing factor. The first term in the above equation is the traditional softmax loss function, and the second term is the center loss function. Introducing the center loss function enhances the cohesion of the extracted features and maintains good performance even with limited training samples.

[0119] Therefore, the final multi-task joint loss function is:

[0120] loss full =β1l G +β2l A

[0121] Among them, β1 and β2 are balance factors.

[0122] Since different modality recognition tasks have varying degrees of difficulty, β is used. k This represents the weight allocation for different loss functions; furthermore, since the complexity of different modality recognition tasks varies, the convergence speed of the network also differs. To avoid overfitting or local minima problems caused by multi-task learning, this invention will study a task-driven early stopping strategy for training and optimizing deep networks.

[0123] Step 2: Use the camera installed on the media wall to capture the faces of people standing in front of the building's media wall, and use face detection technology to extract qualified face images. Then, use face analysis technology to perform attribute analysis on the extracted qualified face images to determine the gender, age, and identity attributes of the faces in front of the building's media wall.

[0124] This step is completed online on-site, mainly to detect and analyze the users of the multimedia building wall for subsequent intelligent display.

[0125] Cameras installed on the media wall capture images of people standing in front of it, and face detection technology is used to extract qualified facial images. Then, face analysis technology is used to analyze the attributes of the extracted qualified facial images to determine the gender, age, and identity attributes of the people in front of the media wall.

[0126] The facial analysis technology mentioned is a lifelong learning-based facial analysis technology, as detailed below:

[0127] This invention integrates three tasks—facial gender, age, and identity recognition—into a single model for multi-task deep learning. Based on lifelong learning, it utilizes the same network for continuous training and learning across all tasks, resulting in a multi-task learning network. This network employs a lifelong learning strategy to learn each task, minimizing forgetting knowledge from previous tasks while learning new ones, and transferring knowledge from old tasks to new tasks to improve performance. It represents a forward knowledge transfer strategy—that is, how to leverage knowledge from old tasks to enhance learning of new tasks.

[0128] Face analysis technology employs a multi-task learning network based on lifelong learning. To facilitate knowledge transfer and avoid catastrophic forgetting, for each task, after training for that task, the importance Ω of each parameter in the network to that task is calculated. ij Ω represents the proportion of the parameter value in the i-th row and j-th column to the total parameter values, and this proportion is used in subsequent training tasks. ij It is added to the loss function as a regularization term whenever a new task is trained: for Ω ij For larger parameters, minimize their changes during gradient descent, as these parameters are important for past tasks and their values ​​need to be preserved to avoid catastrophic forgetting; while for Ω... ij Smaller parameters allow for gradient updates with larger magnitudes, resulting in better performance on new tasks. Therefore, the loss function for the nth task is:

[0129]

[0130] Among them, L n (θ) is the loss function for the current task, λ is the balance factor, and θ ij These are the parameters in the i-th row and j-th column of the current model parameter matrix. Ω consists of model parameters obtained after training the first n-1 tasks. After training on each task, Ω... ij All will be updated, where n=3, and the training order of the three tasks is gender recognition, age recognition, and identity recognition. When performing the first task, Set to 0;

[0131] The backbone network of the multi-task network can be RestNet or EfficientNet. When using a lifelong learning model for multi-task learning of face attributes, the learning order of different attribute tasks has a great impact on the final performance of the primary network. This invention will use a cross-validation strategy to find the difficulty of each task and the learning order.

[0132] Step 3: Based on the constructed tourist image library, intelligently display the media wall according to the results of face detection and analysis, and assign each displayed image a memorable number.

[0133] This step involves intelligent media wall display based on the results of facial analysis of the two individuals.

[0134] 1) Based on face detection technology (the same method used in step 1), if no tourists are found in front of the media wall, the media wall will randomly play images of tourists, or play promotional videos, or stop playing to save electricity.

[0135] 2) If there is a face in front of the media wall, but the person is a new visitor, then intelligently play pictures of the visitor with a similar gender and age based on the face's gender and age.

[0136] 3) If there is a face in front of the media wall and it is a face in the visitor database, then the visitor's photos will be played intelligently based on their facial identity tag.

[0137] Preferably, when displaying photos of the visit, each photo should be given an easy-to-remember number so that visitors can print them out as souvenirs later.

[0138] Step 4: Use gesture recognition technology to enable users to interact with the media wall without contact, including pausing, starting, scrolling forward, scrolling backward, and deleting.

[0139] This step utilizes gesture recognition technology to allow users to interact with the media wall without physical contact. Users can perform intelligent interactions with the media wall such as pausing, starting, scrolling forward, scrolling backward, and deleting.

[0140] This invention employs a gesture recognition method based on deep collaborative dictionary learning, specifically as follows:

[0141] This invention extracts features from the input gesture image to be recognized. It utilizes MobileNetV3 to construct a lightweight feature extraction network and proposes feature vectors. m is the dimension of the feature vector;

[0142] The feature vector y is represented by a linear combination using a gesture-complete dictionary D, a deep sparse representation model is established, and the coefficient vector is solved:

[0143]

[0144] Where w represents the sparse representation coefficients of the demand solution. This is an approximation of w, where γ is a constant equilibrium factor, ||·|| 1 / 2 This represents the 1 / 2 norm.

[0145] Gesture classification and recognition are performed based on the coefficient vector of the linear combination obtained from the above formula.

[0146] The gestures described in this invention are categorized into 5 types, namely pause, start, flip forward, flip backward, and delete.

[0147] Furthermore, before representing the feature vector y using a linear combination of gesture-complete dictionaries, a training step for the gesture-complete dictionary is included. To improve the accuracy and robustness of the model, this invention proposes a deep collaborative learning dictionary training method:

[0148] Based on the principles of deep learning, the gesture training sample X is subjected to multi-level deep decomposition learning to obtain the gesture sub-dictionary D. c :

[0149]

[0150] in, X is the gesture training sample matrix; For each layer of the c-th type of gesture, there is a dictionary (basis matrix). D is the corresponding coefficient matrix (encoding matrix); s W s These are the shared dictionary and its corresponding coefficient matrix, respectively. According to the principle of nonnegative matrix factorization, we have:

[0151]

[0152] The above formula can be interpreted as performing a multi-level decomposition on the training sample X, which includes gesture information such as pause, start, forward scrolling, backward scrolling, and delete, to find a new representation. W s and new representation space

[0153] To enhance the non-linear representation capability of the dictionary, the coefficient matrix is ​​modified using a deep neural network approach:

[0154]

[0155] Where i represents the i-th layer, ranging from [1 to m], and g(·) is a non-linear activation function. Therefore, the objective function of the deep collaborative dictionary learning model is:

[0156]

[0157] The deep collaborative dictionary learning model employs a training method similar to that of a stacked autoencoder network, consisting of two stages: layer-by-layer pre-training and overall fine-tuning.

[0158] • Layer-by-layer pre-training phase

[0159] A. Order Solving the minimization problem: Complete the first level of decomposition;

[0160] B. Continue to break down Complete the second level of decomposition;

[0161] This process is repeated until all layers are pre-trained. By using a greedy decomposition method layer by layer, the training of each layer becomes a traditional dictionary learning problem (except that a shared dictionary is added here). Depending on the actual needs, sparse constraints or supervised learning can be added to the dictionary learning of each layer. For ease of expression and clarity, the above formula does not include a non-linear activation function.

[0162] • Overall fine-tuning phase

[0163] This is achieved by minimizing the loss function and using stochastic gradient descent.

[0164] Step 5, Self-service printing: If visitors see an image that interests them on the media wall and wish to print it as a souvenir:

[0165] 1) Note down the image numbers that interest you;

[0166] 2) Enter the image number at the self-service printer;

[0167] 3) Choose a payment method (online payment, WeChat payment, Alipay payment, bank transfer, or cash payment, etc.);

[0168] 4) Select the printing method (on-site printing, download to USB drive, forward to email, etc.);

[0169] 5) Confirmation complete.

[0170] This invention also provides an intelligent interactive media display wall system, comprising the following modules:

[0171] The image organization and classification module is used to build an offline tourist image library, collect tourist photos, use the corresponding standard facial images as the identity tags of each tourist, and create an image folder for each tourist, and classify and store tourist images by gender and age.

[0172] The face analysis and recognition module is used to capture the faces of people standing in front of the building's media wall using cameras installed on the media wall, and to extract qualified face images using face detection technology. Then, face analysis technology is used to analyze the attributes of the extracted qualified face images to determine the gender, age, and identity attributes of the faces in front of the building's media wall.

[0173] The media wall smart display module is used to combine the constructed visitor image library to intelligently display the results of face detection and analysis on the media wall, and assign each displayed image a memorable number.

[0174] The intelligent interaction module is used to enable users to interact with the media wall without contact using gesture recognition technology, including pausing, starting, scrolling forward, scrolling backward, and deleting.

[0175] The specific implementation methods of each module and the corresponding steps are not described in this invention.

[0176] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for intelligent interactive media display, characterized in that, Includes the following steps: Step 1: Build an offline tourist image library, collect tourist photos, use the corresponding standard facial images as the identity tags of each tourist, and create an image folder for each tourist, and store tourist images by gender and age. Step 2: Use the camera installed on the media wall to capture the faces of people standing in front of the building's media wall, and use face detection technology to extract qualified face images. Then, use face analysis technology to perform attribute analysis on the extracted qualified face images to determine the gender, age, and identity attributes of the faces in front of the building's media wall. Step 3: Based on the constructed tourist image library, intelligently display the media wall according to the results of face detection and analysis, and assign each displayed image a memorable number. Step 4: Use gesture recognition technology to enable users to interact with the media wall without contact, including pausing, starting, scrolling forward, scrolling backward, and deleting. The specific implementation method of step 4 is as follows: A gesture recognition method based on deep collaborative dictionary learning is used to achieve contactless interaction. The specific implementation process is as follows: Feature extraction is performed on the input gesture image to be recognized. A lightweight feature extraction network is constructed using MobileNetV3 to propose feature vectors. m is the dimension of the feature vector; The feature vector y is represented by a linear combination using a gesture-complete dictionary D, a deep sparse representation model is established, and the coefficient vector is solved: Where w represents the sparse representation coefficients of the demand solution. This is an approximation of w, where γ is a constant equilibrium factor, ||·|| 1 / 2 Represents the 1 / 2 norm; Gesture classification and recognition are performed based on the coefficient vector of the linear combination obtained from the above formula. The gestures are 5 categories, defined as pause, start, forward flip, backward flip, and delete. Furthermore, before representing the feature vector y using a linear combination of gesture-complete dictionaries, a training step for the gesture-complete dictionary is included. To improve the accuracy and robustness of the model, a deep collaborative learning dictionary training method is proposed. Based on the principles of deep learning, the gesture training sample X is subjected to multi-level deep decomposition learning to obtain the gesture sub-dictionary D. c ∈D: Where, D = {D 1 ,…,D c ,…,D 5 }, where X is the gesture training sample matrix; For different levels of the c-th type of gesture, D is the corresponding coefficient matrix; s W s These are the shared dictionary and its corresponding coefficient matrix, i.e., the parameters to be solved; according to the principle of nonnegative matrix factorization, we have: The above formula can be interpreted as performing a multi-level decomposition on the training sample X, which includes gesture information such as pause, start, forward, backward, and delete, to find a new representation. W s and new representation space To enhance the non-linear representation capability of the dictionary, the coefficient matrix is ​​modified using a deep neural network approach: Where i represents the i-th layer, ranging from [1 to m], and g(·) is a non-linear activation function, therefore, the objective function of the deep collaborative dictionary learning model is:

2. The intelligent interactive media display method as described in claim 1, characterized in that: The specific implementation method of step 1 is as follows: Step 11: Install cameras at the entrance of the theme park or amusement park, use a face detection model to collect a standard frontal face image as the identity label of each visitor, and create an image folder for each visitor. Step 12: Use photographers and the venue's surveillance cameras to capture images of each visitor's experience; Step 13: The massive amount of collected travel photos are categorized and stored in corresponding folders according to the identity tags established in Step 11 to create a tourist travel image library. In addition, the massive amount of collected travel photos are subjected to facial gender and age recognition, and then stored in categories according to gender and age.

3. The intelligent interactive media display method as described in claim 2, characterized in that: The specific implementation method of the face detection model in step 11 is as follows: The face detection model described is implemented using a lightweight detection model. Its network structure consists of three main parts: a backbone network, a multi-scale network (FPN), and a detection head. First, the backbone network extracts features from the input video image. Then, the multi-scale network fuses the features extracted by the backbone network at multiple scales. Finally, the detection head performs face detection and localization on the fused features. The backbone network uses MobileNetv3 as the backbone, the multi-scale network adopts a top-down structure, and the detection head uses a four-conv+bn structure, which outputs the location confidence, the scale of the detection box, the offset, and the facial key points, respectively. The loss function of the face detection model is the same as that of CenterNet, consisting of a FocalLoss-type location classification loss and an L1 regression loss. The location classification loss function is as follows: In the formula Y xyc The true center point category is either 1 or 0. The confidence level of the prediction is given by α and β, which are constants. The scale loss function is: Among them, s k The actual dimensions of the target bounding box. The dimensions of the predicted target bounding box are given by N, and the number of training samples is given by N. The offset loss function is: in, For the predicted offset value, (x) k ,y k ) represents the coordinates of the center point, and smoothL1(·) represents smooth L1 regression; The Points loss function is: in, The difference in distance between the key point and the center point. The distance difference between the predicted keypoint and the center point, (c x ,c y (p) represents the coordinates of the center point. x ,p y (box) represents the coordinates of facial landmarks. w ,box h ) represents the length and width of the face detection bounding box; the final loss function is: L=L location +λ1L scale +λ2L offset +λ3L points 。 4. The intelligent interactive media display method as described in claim 2, characterized in that: In step 13, the multi-task neural network MDCNN is used to simultaneously recognize face gender and age. The network model design is as follows: Suppose we use N training samples to learn T=2 tasks, using Let represent the i-th sample in the t-th task ({1,…,T}) and its corresponding task label. Then, the learning objective of MDCNN is: Where, f(x) i (,θ) represents the deep network function, where θ is the network parameter to be solved. Let Φ(θ) represent the loss function for different tasks, where Φ(θ) is the regularization penalty term. For the gender identification problem, since it is a binary classification problem, the softmax loss function is directly adopted: Where g = 0 or 1 represents male or female, and (p0, p1) represents the probability of each gender; For the face age recognition problem, since it is a multi-class classification problem, the loss function is center loss: Where, x k Let y represent the deep learning feature of the k-th sample in a mini-bach. k Indicates its corresponding category; Indicates y k The deep learning feature centers of the class, W and b represent the weights and bias parameters in the last fully connected layer, respectively; m and n represent the mini-batch and class sizes, respectively; and λ is the balance factor. Therefore, the final multi-task joint loss function is: Among them, β1 and β2 are balance factors.

5. The intelligent interactive media display method as described in claim 1, characterized in that: The specific implementation method of step 2 is as follows: Face analysis technology employs a multi-task learning network based on lifelong learning. To facilitate knowledge transfer and avoid catastrophic forgetting, for each task, after training for that task, the importance Ω of each parameter in the network to that task is calculated. ij Ω represents the proportion of the parameter value in the i-th row and j-th column to the total parameter values, and this proportion is used in subsequent training tasks. ij It is added to the loss function as a regularization term; therefore, the loss function for the nth task is: Among them, L n (θ) is the loss function for the current task, λ is the balance factor, and θ ij These are the parameters in the i-th row and j-th column of the current model parameter matrix. Ω consists of model parameters obtained after training the first n-1 tasks. After training on each task, Ω... ij All will be updated, where n=3, and the training order of the three tasks is gender recognition, age recognition, and identity recognition. When performing the first task, Set to 0; The backbone network of the multi-task network adopts RestNet or EfficientNet.

6. The intelligent interactive media display method as described in claim 2, characterized in that: The specific implementation method of step 3 is as follows: 31) Based on face detection technology, if no tourists are found in front of the media wall, the media wall will randomly play images of tourists playing, or play promotional videos, or stop playing to save electricity. 32) If there is a face in front of the media wall, but the person is a new visitor, then intelligently play pictures of the visitor with a similar gender and age based on the face's gender and age. 33) If there is a face in front of the media wall and it is a face in the visitor database, then the visitor's photos will be played intelligently based on their facial identity tags.

7. The intelligent interactive media display method as described in claim 1, characterized in that: The deep collaborative dictionary learning model employs a training method similar to that of a stacked autoencoder network, consisting of two stages: layer-by-layer pre-training and overall fine-tuning. (1) Layer-by-layer pre-training stage A. Order Solving the minimization problem: Complete the first level of decomposition; B. Regarding W1 c Continue to break down Complete the second level of decomposition; This process continues until all layers are pre-trained. By using a greedy decomposition method layer by layer, the training of each layer becomes a traditional dictionary learning problem. Sparse constraints or supervised learning are then added to the dictionary learning of each layer as needed. (2) Overall fine-tuning stage This is achieved by minimizing the loss function and using stochastic gradient descent.

8. The intelligent interactive media display method as described in claim 1, characterized in that: It also includes step 5, self-service printing. If a visitor sees an image that interests them on the media wall and wants to print it as a souvenir, they should follow these steps: 51) Note down the numbers of the pictures that interest you; 52) Enter the image number at the self-service printer; 53) Choose a payment method, including online payment, WeChat payment, Alipay payment, bank transfer or cash payment; 54) Select the printing method, including on-site printing, downloading to USB drive, or forwarding to email; 55) Confirmation complete.

9. An intelligent interactive media display wall system, characterized in that, Includes the following modules: The image organization and classification module is used to build an offline tourist image library, collect tourist photos, use the corresponding standard facial images as the identity tags of each tourist, and create an image folder for each tourist, and classify and store tourist images by gender and age. The face analysis and recognition module is used to capture the faces of people standing in front of the building's media wall using cameras installed on the media wall, and to extract qualified face images using face detection technology. Then, face analysis technology is used to analyze the attributes of the extracted qualified face images to determine the gender, age, and identity attributes of the faces in front of the building's media wall. The media wall smart display module is used to combine the constructed visitor image library to intelligently display the results of face detection and analysis on the media wall, and assign each displayed image a memorable number. The intelligent interaction module uses gesture recognition technology to allow users to interact with the media wall without physical contact, including pausing, starting, scrolling forward, scrolling backward, and deleting. The specific implementation is as follows: A gesture recognition method based on deep collaborative dictionary learning is used to achieve contactless interaction. The specific implementation process is as follows: Feature extraction is performed on the input gesture image to be recognized. A lightweight feature extraction network is constructed using MobileNetV3 to propose feature vectors. m is the dimension of the feature vector; The feature vector y is represented by a linear combination using a gesture-complete dictionary D, a deep sparse representation model is established, and the coefficient vector is solved: Where w represents the sparse representation coefficients of the demand solution. This is an approximation of w, where γ is a constant equilibrium factor, ||·|| 1 / 2 Represents the 1 / 2 norm; Gesture classification and recognition are performed based on the coefficient vector of the linear combination obtained from the above formula. The gestures are 5 categories, defined as pause, start, forward flip, backward flip, and delete. Furthermore, before representing the feature vector y using a linear combination of gesture-complete dictionaries, a training step for the gesture-complete dictionary is included. To improve the accuracy and robustness of the model, a deep collaborative learning dictionary training method is proposed. Based on the principles of deep learning, the gesture training sample X is subjected to multi-level deep decomposition learning to obtain the gesture sub-dictionary D. c ∈D: Where, D = {D 1 ,…,D c ,…,D 5 }, where X is the gesture training sample matrix; For different levels of the c-th type of gesture, D is the corresponding coefficient matrix; s W s These are the shared dictionary and its corresponding coefficient matrix, i.e., the parameters to be solved; according to the principle of nonnegative matrix factorization, we have: The above formula can be interpreted as performing a multi-level decomposition on the training sample X, which includes gesture information such as pause, start, forward, backward, and delete, to find a new representation. W s and new representation space To enhance the non-linear representation capability of the dictionary, the coefficient matrix is ​​modified using a deep neural network approach: Where i represents the i-th layer, ranging from [1 to m], and g(·) is a non-linear activation function, therefore, the objective function of the deep collaborative dictionary learning model is:

Images