Multi-modal VR advertisement pushing method and system based on face recognition
Through multimodal data feature extraction and prediction models, combined with user interaction feedback optimization algorithms, the problem of poor user experience in existing technologies in VR advertising push is solved, accurate advertising delivery effects are achieved, and a more efficient user interaction experience and advertising delivery platform is provided.
Patent Information
- Application Number
- CN202510663345.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2025-09-19
AI Technical Summary
Existing VR advertising push methods lack personalization and cannot be adjusted dynamically in a timely manner, resulting in a low match between advertisements and user needs and interests and a poor user experience.
By collecting user data, using deep learning models to extract and weightedly fuse multimodal data features, combining a hybrid model of support vector machines and deep neural networks to predict user interests and behaviors, and dynamically adjusting the weights of the advertising matching algorithm based on user interaction data, personalized advertising push is achieved.
It achieves more accurate and personalized advertising push, improves advertising conversion rate and user experience, and provides advertisers with a more efficient delivery platform.
Smart Images
Figure CN120672401A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of VR advertising technology, and in particular to a multimodal VR advertising push method and system based on face recognition. Background Art
[0002] With the rapid development of virtual reality (VR) technology, its application in many fields, such as entertainment, education, and commerce, is becoming increasingly widespread, bringing users a new immersive experience. In the commercial field, advertising, as a key marketing tool, is also gradually being integrated into the VR environment.
[0003] Most existing VR advertising push methods are relatively simple, usually based on simple user attributes (such as age, gender, etc.) or preset scenarios. The lack of personalization leads to a low degree of fit between advertisements and users' actual needs and interests. Users may feel bored by a large number of irrelevant or uninteresting advertisements, thereby reducing the user experience. At the same time, the updating and adjustment of advertising content is not flexible enough. When user behavior and preferences change, the system cannot dynamically adjust the advertising push strategy in a timely manner, resulting in a decrease in the match between advertisements and the user's current status.
[0004] Therefore, it is necessary to provide a new multimodal VR advertising push method and system based on face recognition. Summary of the Invention
[0005] Based on the above-mentioned problems existing in the prior art, the purpose of the embodiments of the present invention is to provide a multimodal VR advertising push method and system based on face recognition. The present invention obtains multimodal data and uses a model for accurate prediction to achieve accurate identification of user behavior and preferences, thereby providing personalized advertising push.
[0006] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a multimodal VR advertising push method based on face recognition, comprising:
[0007] S1, collects multimodal data of users in a virtual reality environment and preprocesses the multimodal data;
[0008] S2, uses a deep learning network model to extract features from preprocessed multimodal data;
[0009] S3, dynamically weighted fusion of the extracted image features, speech features, and tactile data to generate a unified multimodal feature vector;
[0010] S4, building and training a multimodal VR advertising prediction model, inputting the fused multimodal feature vector into the multimodal VR advertising prediction model, and outputting user interest tags and behavior prediction results;
[0011] S5: Build an advertising database and select corresponding advertising content based on the user's current interest tags and historical behavior data, embed it into the VR scene, and push it;
[0012] S6, obtaining the user's interaction data on the pushed advertisements, and dynamically adjusting the advertisement matching algorithm weights based on the user's interaction data feedback.
[0013] Furthermore, the multimodal data includes image data, voice data and tactile data. The collection of the image data includes using at least two high-resolution RGB cameras, which are installed on both sides of the front of the VR helmet, for capturing the user's facial expressions, eye movements and head posture in real time; the collection of the tactile data includes configuring a piezoelectric tactile sensor on the VR handle to detect the user's grip strength and operation gestures.
[0014] Furthermore, the preprocessing of the multimodal data includes compressing the image data, performing noise reduction on the voice data, and normalizing the tactile data.
[0015] Furthermore, the network model using deep learning extracts features from the preprocessed multimodal data, including:
[0016] Step S21, using a deep learning convolutional neural network to extract features from the image data, and the convolutional neural network uses ResNet-50;
[0017] Step S22, using a deep learning recurrent neural network to extract features from the speech data, where the recurrent neural network uses a long short-term memory network;
[0018] Step S23: extract features from the tactile data using a one-dimensional convolutional neural network.
[0019] Furthermore, the image features, speech features and tactile data after feature extraction are weighted and fused to generate a unified multimodal feature vector, including:
[0020] Step S31, judging based on the user's current scenario, and dynamically updating the weight coefficient using exponential smoothing method;
[0021] The calculation formula for dynamically updating the weight coefficient of the exponential smoothing method is:
[0022]
[0023] in, is the weight coefficient of the mth modality (such as vision, hearing, touch) at time t, is the weight coefficient of the mth mode at time t-1, a is the smoothing coefficient, Preset weights for the scene of the mth mode;
[0024] In step S32, the visual, auditory, and tactile feature vectors are mapped to a unified dimension through a fully connected layer of deep learning, and then key features are filtered through an attention mechanism to generate a fused multimodal feature vector.
[0025] Furthermore, the multimodal VR advertising prediction model is constructed and trained, the fused multimodal feature vector is input into the multimodal VR advertising prediction model, and the user interest tags and behavior prediction results are output, including:
[0026] Step S41: The multimodal VR advertising prediction model adopts a hybrid model architecture combining support vector machine and deep neural network;
[0027] Step S42: training a multimodal VR advertising prediction model using a large amount of labeled multimodal data;
[0028] Step S43: The fused multimodal feature vector is used as the input of the model, including visual features, auditory features, and information. The multimodal feature vector is integrated through a fully connected layer to ensure that each feature can affect the model's prediction results, and the user interest tag and behavior prediction results are output;
[0029] In step S44, an online learning mechanism is adopted, and the multimodal VR advertising prediction model can receive new user data in real time and dynamically update model parameters to adapt to changes in user preferences.
[0030] Furthermore, the acquisition of user interaction data for pushed ads and the dynamic adjustment of ad matching algorithm weights based on user interaction data feedback include:
[0031] Step S61, obtaining user interaction data on the pushed advertisement, including click-through rate, dwell time, tactile feedback intensity, and voice evaluation;
[0032] Step S62: Analyze the collected interaction data in depth to extract key features;
[0033] Step S63: Using a reinforcement learning framework, dynamically adjust the weight of the ad matching algorithm based on user feedback.
[0034] A multimodal VR advertising push system based on face recognition, applied to the above-mentioned multimodal VR advertising push method based on face recognition, comprises:
[0035] The data acquisition module is used to collect multimodal data of users in the virtual reality environment and pre-process the multimodal data;
[0036] Feature extraction module, used to extract features from pre-processed multimodal data using a deep learning network model;
[0037] The weighted fusion module is used to dynamically weight the image features, speech features, and tactile data after feature extraction to generate a unified multimodal feature vector;
[0038] Modeling and training module, used to build and train multimodal VR advertising prediction models;
[0039] The behavior prediction module is used to input the fused multimodal feature vector into the multimodal VR advertising prediction model and output user interest tags and behavior prediction results;
[0040] The advertising push module is used to build an advertising database and select corresponding advertising content based on the user's current interest tags and historical behavior data, embedding it into the VR scene for push;
[0041] The algorithm optimization module is used to obtain user interaction data on pushed ads and dynamically adjust the ad matching algorithm weights based on feedback from user interaction data.
[0042] The embodiment of the present invention further provides a network-side server, comprising:
[0043] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned multimodal VR advertising push method based on face recognition.
[0044] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned multimodal VR advertising push method based on face recognition.
[0045] The beneficial effects of the present invention are as follows: a multimodal VR advertising push method based on face recognition of the present invention includes: collecting multimodal data of users in a virtual reality environment and preprocessing the multimodal data; using a deep learning network model to extract features from the preprocessed multimodal data; dynamically weighting and fusing the image features, voice features and tactile data after feature extraction to generate a unified multimodal feature vector; constructing and training a multimodal VR advertising prediction model, inputting the fused multimodal feature vector into the multimodal VR advertising prediction model, and outputting user interest tags and behavior prediction results; constructing an advertising database, and based on the user's current interest tags and historical behavior data, selecting corresponding advertising content to embed into the VR scene for push; obtaining the user's interaction data on the pushed advertisement, and dynamically adjusting the advertisement matching algorithm weight based on the user's interaction data. The multimodal VR advertising push method based on face recognition of the present invention collects and pre-processes multimodal data, uses a deep learning model to accurately extract features and then dynamically weighted fuses them, combines the prediction model with the online learning mechanism to output user interest and behavior predictions in real time, and then uses a multi-dimensional label advertising library and a dynamic matching algorithm to push personalized advertisements. Finally, closed-loop optimization is performed based on user interaction feedback, thereby achieving more accurate and personalized advertising push, improving advertising conversion rates, enhancing user experience, and providing advertisers with a more efficient advertising delivery platform. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The present invention will be further described below with reference to the accompanying drawings and examples.
[0047] In the picture:
[0048] Figure 1 This is a flowchart of a multimodal VR advertising push method based on face recognition provided by the first embodiment of the present invention;
[0049] Figure 2 A schematic diagram of the modules of a multimodal VR advertising push system based on face recognition provided by the second embodiment of the present invention;
[0050] Figure 3 It is a structural diagram of a network-side server provided according to a third embodiment of the present invention. DETAILED DESCRIPTION
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0052] First embodiment:
[0053] The first embodiment of the present invention provides a multimodal VR advertising push method based on face recognition, including: collecting multimodal data of users in a virtual reality environment and preprocessing the multimodal data; using a deep learning network model to extract features from the preprocessed multimodal data; dynamically weighted fusion of the extracted image features, voice features and tactile data to generate a unified multimodal feature vector; constructing and training a multimodal VR advertising prediction model, inputting the fused multimodal feature vector into the multimodal VR advertising prediction model, and outputting user interest tags and behavior prediction results; constructing an advertising database, and based on the user's current interest tags and historical behavior data, selecting corresponding advertising content to embed into the VR scene for push; obtaining the user's interaction data on the pushed advertisement, and dynamically adjusting the advertisement matching algorithm weight based on feedback from the user's interaction data. The multimodal VR advertising push method based on face recognition of the present invention collects and pre-processes multimodal data, uses a deep learning model to accurately extract features and then dynamically weighted fuses them, combines the prediction model with the online learning mechanism to output user interest and behavior predictions in real time, and then uses a multi-dimensional label advertising library and a dynamic matching algorithm to push personalized advertisements. Finally, closed-loop optimization is performed based on user interaction feedback, thereby achieving more accurate and personalized advertising push, improving advertising conversion rates, enhancing user experience, and providing advertisers with a more efficient advertising delivery platform.
[0054] The following is a detailed description of the implementation details of the multimodal VR advertising push method based on face recognition in this embodiment. The following content is only for the convenience of understanding the implementation details, and is not necessary for the implementation of this solution. The specific process of this embodiment is as follows: Figure 1 shown.
[0055] Step S1: Collect multimodal data of the user in the virtual reality environment and preprocess the multimodal data.
[0056] Specifically, multimodal data includes image data, voice data, and tactile data.
[0057] The collection of image data includes the use of at least two high-resolution RGB cameras, installed on both sides of the front of the VR helmet, to capture the user's facial expressions, eye movements and head posture in real time.
[0058] The collection of voice data includes configuring four high-sensitivity microphones distributed on the edges of the VR helmet to capture the user's voice commands and environmental sounds.
[0059] The collection of tactile data includes configuring a piezoelectric tactile sensor on the VR handle to detect the user's grip strength and operation gestures.
[0060] Preprocessing of multimodal data includes compressing image data, denoising speech data, and normalizing tactile data.
[0061] Specifically, since the collected image data has a large amount of data, directly processing the image data will result in high computing costs and transmission delays. The H.264 encoding standard is used to perform inter-frame compression on the video stream, which can significantly reduce the data transmission bandwidth requirements while ensuring video quality. In addition, an image denoising algorithm is used in the compression process. As an example, the non-local mean filtering algorithm is used to effectively eliminate motion blur and noise by analyzing similar areas in the image, thereby maintaining the details and edge information of the image and providing clearer image data for subsequent feature extraction. The local mean filtering algorithm can effectively eliminate motion blur, ensuring that the system can accurately capture the user's facial expressions and head posture.
[0062] Spectral subtraction is used to eliminate environmental noise. By analyzing the spectral characteristics of the speech signal, spectral subtraction can effectively remove background noise, improve the accuracy of speech recognition, and provide high-quality data for subsequent speech feature extraction.
[0063] Since the sensitivity of tactile sensors in different VR controllers may vary, standardizing tactile data can eliminate the sensitivity differences between different sensors, normalize the tactile signals, and generate a standardized tactile intensity vector. Normalization ensures the comparability of data collected by different sensors by mapping the values of tactile signals into a unified range, thereby providing standardized data support for subsequent tactile feature extraction and behavior analysis.
[0064] Step S2: Use a deep learning network model to extract features from the preprocessed multimodal data.
[0065] Specifically, the steps for extracting features from preprocessed multimodal data using a deep learning network model include:
[0066] In step S21, a deep learning convolutional neural network is used to extract features from the image data, and the convolutional neural network uses ResNet-50.
[0067] Specifically, by extracting key facial expression points, such as the curvature of the mouth corners and eye contours, the user's facial expression key points can reflect the user's emotional state and focus of attention. For example, a raised corner of the mouth may indicate a state of happiness, while changes in the eye contours may indicate the user's level of concentration. Furthermore, a convolutional neural network is used to extract features of the user's head movement trajectory, which can reflect the user's visual focus and behavioral habits. For example, a user frequently moving their head left and right may indicate curiosity about the surrounding environment, while a prolonged gaze in a certain direction may indicate interest in that direction.
[0068] Step S22: extract features from the speech data using a deep learning recurrent neural network, where the recurrent neural network uses a long short-term memory network.
[0069] Specifically, a long short-term memory network is used to extract temporal emotional features from speech signals, such as intonation and speaking rate. These temporal emotional features can reflect the user's emotional state. For example, a rising intonation may indicate excitement, while a faster speaking rate may indicate nervousness.
[0070] Step S23: extract features from the tactile data using a one-dimensional convolutional neural network.
[0071] Through a one-dimensional convolutional neural network, the time series features in tactile data, such as click frequency, sliding direction, etc., are automatically learned, thereby reflecting the user's operating habits and preferences.
[0072] Step S3: Dynamically weight and fuse the extracted image features, speech features, and tactile data to generate a unified multimodal feature vector.
[0073] Specifically, the steps of weighted fusion of the extracted image features, speech features, and tactile data to generate a unified multimodal feature vector include:
[0074] Step S31 : make a judgment based on the user's current scenario and dynamically update the weight coefficient using exponential smoothing method.
[0075] Specifically, the current interaction scene is judged by detecting the head posture and gaze point in the image data. As an example, when the user issues a voice command, it is judged as a voice interaction scene, at which time the auditory modal weight is 50%, the visual weight is 40%, and the tactile weight is 10%. When the gaze point in the image data is fixed on a virtual object (staying for > 1s), it is judged as a visual focus scene, at which time the visual weight is 60%, the auditory weight is 25%, and the tactile weight is 15%. When a sliding / pinch gesture is detected in the tactile data, it is judged as a gesture operation scene, at which time the tactile weight is 40%, the visual weight is 35%, and the auditory weight is 25%.
[0076] The calculation formula for dynamically updating the weight coefficient of the exponential smoothing method is:
[0077]
[0078] in, is the weight coefficient of the mth modality (such as vision, hearing, touch) at time t, is the weight coefficient of the mth mode at time t-1, a is the smoothing coefficient, Preset weights for the scene of the mth mode.
[0079] Through exponential smoothing method, combined with historical weight and scene preset weights Dynamically update the current modal weight Make the weight distribution more in line with real-time scenarios and historical laws.
[0080] In step S32, the visual, auditory, and tactile feature vectors are mapped to a unified dimension through a fully connected layer of deep learning, and then key features are filtered through an attention mechanism to generate a fused multimodal feature vector.
[0081] Step S4: construct and train a multimodal VR advertising prediction model, input the fused multimodal feature vector into the multimodal VR advertising prediction model, and output user interest tags and behavior prediction results.
[0082] Specifically, the steps of constructing and training a multimodal VR advertising prediction model, inputting the fused multimodal feature vector into the multimodal VR advertising prediction model, and outputting user interest tags and behavior prediction results include:
[0083] In step S41, the multimodal VR advertising prediction model adopts a hybrid model architecture that combines support vector machine (SVM) and deep neural network (DNN).
[0084] Among them, the SVM part is mainly used to process linearly separable features and can effectively classify user interests; the DNN part is used to capture complex nonlinear relationships and predict user behavior.
[0085] Step S42: Use a large amount of labeled multimodal data to train the multimodal VR advertising prediction model.
[0086] Specifically, the training data includes user behavior characteristics in different scenarios, as well as corresponding interest tags and behavior results. The model parameters are adjusted using optimization algorithms (such as the Adam optimizer) to enable the model to accurately predict user interests and behaviors.
[0087] In step S43, the fused multimodal feature vector is used as the input of the model, including visual features, auditory features and information. The multimodal feature vector is integrated through a fully connected layer to ensure that each feature can affect the prediction results of the model, and the user interest tags and behavior prediction results are output.
[0088] In step S44, an online learning mechanism is adopted, and the multimodal VR advertising prediction model can receive new user data in real time and dynamically update model parameters to adapt to changes in user preferences.
[0089] Specifically, when users operate or interact in a VR environment, multimodal data is collected in real time. After preprocessing and feature extraction, it is input into the multimodal VR ad prediction model. The multimodal VR ad prediction model quickly outputs the user's current interest tags and behavior prediction results. Furthermore, the multimodal VR ad prediction model utilizes an online learning mechanism, which allows it to receive new user data in real time and dynamically update model parameters, ensuring that the model always accurately reflects the user's latest behavior and preferences, thereby improving prediction accuracy and the relevance of ad push.
[0090] Step S5: construct an advertising database, and based on the user's current interest tags and historical behavior data, select corresponding advertising content to be embedded in the VR scene for push.
[0091] Specifically, the ad database uses a multi-dimensional tagging approach to store ad content. Each ad entry corresponds to a user's behavior and interest tags, enabling the model to quickly match appropriate ad content. Based on the user's current interest tags and historical behavior data, a dynamic matching algorithm is used to select the most appropriate ad content. This algorithm comprehensively considers the user's interests, behavioral patterns, and the relevance and priority of the ad content to calculate a match score between each ad and the user. Based on this match score, an ad push queue is generated, and the top few ads with the highest match scores are selected for push, ensuring that the user receives ads that best match their interests and preferences.
[0092] Furthermore, seamlessly embedding advertising content into VR scenes not only enhances the user's sense of immersion, but also allows the advertising content to naturally blend with the virtual environment, improving the acceptability of advertising and the user's willingness to interact.
[0093] Step S6: Obtain the user's interaction data on the pushed advertisement, and dynamically adjust the advertisement matching algorithm weight based on the user's interaction data.
[0094] Specifically, the steps of obtaining user interaction data on pushed ads and dynamically adjusting the ad matching algorithm weights based on user interaction data include:
[0095] Step S61: Obtain user interaction data on the pushed advertisement, including click-through rate, dwell time, tactile feedback intensity, and voice evaluation.
[0096] Specifically, the click-through rate is the ratio of the number of times users click on an ad to the number of times the ad is displayed, reflecting the user's direct interest in the ad; the dwell time is the time the user stays on the ad, reflecting the user's attention and potential interest in the ad; the tactile feedback intensity is the strength and frequency of the user's tactile operation on the ad content through the VR controller, indicating the user's participation and preference for the ad; the voice evaluation is the user's voice evaluation of the ad, and the sentiment analysis model is used to analyze the user's emotional response and judge the user's preference for the ad.
[0097] Step S62: Analyze the collected interaction data in depth to extract key features.
[0098] Specifically, by analyzing users' click behavior patterns, we can identify their preferences for specific types of ads; by analyzing the distribution of dwell time, we can understand the degree of user interest in different advertising content; the analysis results provide data support for the optimization of ad matching algorithms.
[0099] Step S63: Using a reinforcement learning framework, dynamically adjust the weight of the advertisement matching algorithm based on user feedback.
[0100] Specifically, a reinforcement learning framework is used to dynamically adjust the ad matching algorithm. During the reinforcement learning process, user interaction data is treated as environmental feedback signals, and the weight of the ad matching algorithm is adjusted based on the positive or negative feedback signals.
[0101] As an example, the weight of the ad matching algorithm is dynamically adjusted based on the user's click-through rate and stay time to increase the priority of high-click-through-rate ads, thereby ensuring that the most appropriate ad content can be selected for push in different scenarios, thereby improving the ad conversion rate and user satisfaction.
[0102] The present invention provides a multimodal VR advertising push method based on face recognition, comprising: collecting multimodal data of users in a virtual reality environment and preprocessing the multimodal data; using a deep learning network model to extract features from the preprocessed multimodal data; dynamically weighting and fusing the extracted image features, voice features, and tactile data to generate a unified multimodal feature vector; constructing and training a multimodal VR advertising prediction model, inputting the fused multimodal feature vector into the multimodal VR advertising prediction model, and outputting user interest tags and behavior prediction results; constructing an advertising database, and based on the user's current interest tags and historical behavior data, selecting corresponding advertising content to embed into a VR scene for push; obtaining user interaction data on the pushed advertisements, and dynamically adjusting the weight of an advertisement matching algorithm based on feedback from the user's interaction data. The multimodal VR advertising push method based on face recognition of the present invention collects and pre-processes multimodal data, uses a deep learning model to accurately extract features and then dynamically weighted fuses them, combines the prediction model with the online learning mechanism to output user interest and behavior predictions in real time, and then uses a multi-dimensional label advertising library and a dynamic matching algorithm to push personalized advertisements. Finally, closed-loop optimization is performed based on user interaction feedback, thereby achieving more accurate and personalized advertising push, improving advertising conversion rates, enhancing user experience, and providing advertisers with a more efficient advertising delivery platform.
[0103] Second embodiment:
[0104] like Figure 2 As shown, the second embodiment of the present invention provides a multimodal VR advertising push system based on face recognition, which includes: a data acquisition module 201, a feature extraction module 202, a weighted fusion module 203, a modeling training module 204, a behavior prediction module 205, an advertising push module 206, and an algorithm optimization module 207.
[0105] Specifically, the data acquisition module 201 is used to collect multimodal data of users in a virtual reality environment and preprocess the multimodal data; the feature extraction module 202 is used to extract features from the preprocessed multimodal data using a deep learning network model; the weighted fusion module 203 is used to dynamically weighted fuse the image features, voice features and tactile data after feature extraction to generate a unified multimodal feature vector; the modeling training module 204 is used to build and train a multimodal VR advertising prediction model; the behavior prediction module 205 is used to input the fused multimodal feature vector into the multimodal VR advertising prediction model, and output user interest tags and behavior prediction results; the advertising push module 206 is used to build an advertising database, and based on the user's current interest tags and historical behavior data, select the corresponding advertising content to be embedded in the VR scene for push; the algorithm optimization module 207 is used to obtain the user's interaction data on the pushed advertisements, and dynamically adjust the weight of the advertisement matching algorithm based on the user's interaction data.
[0106] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.
[0107] It is worth noting that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovations of the present invention, this embodiment does not include units that are not closely related to solving the technical problems proposed by the present invention. However, this does not mean that other units do not exist in this embodiment.
[0108] The third embodiment of the present invention relates to a network side server, such as Figure 3 As shown, it includes at least one processor 302; and a memory 301 that is communicatively connected to the at least one processor 302; wherein the memory 301 stores instructions that can be executed by the at least one processor 302, and the instructions are executed by the at least one processor 302 to enable the at least one processor 302 to execute the above-mentioned data processing method.
[0109] Memory 301 and processor 302 are connected using a bus. The bus can include any number of interconnected buses and bridges, connecting various circuits of one or more processors 302 and memory 301. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 302 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits it to processor 302.
[0110] The processor 302 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 301 can be used to store data used by the processor 302 when performing operations.
[0111] A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the multimodal VR advertising push method based on face recognition in the first embodiment.
[0112] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.
[0113] The above is only an embodiment of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the field are aware of all common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the field can improve and implement this scheme in combination with their own abilities under the inspiration given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.
[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A multimodal VR advertising push method based on face recognition, characterized in that: include: S1, collects multimodal data of users in a virtual reality environment and preprocesses the multimodal data; S2, uses a deep learning network model to extract features from preprocessed multimodal data; S3, dynamically weighted fusion of the extracted image features, speech features, and tactile data to generate a unified multimodal feature vector; S4, building and training a multimodal VR advertising prediction model, inputting the fused multimodal feature vector into the multimodal VR advertising prediction model, and outputting user interest tags and behavior prediction results; S5: Build an advertising database and select corresponding advertising content based on the user's current interest tags and historical behavior data, embed it into the VR scene, and push it; S6, obtaining the user's interaction data on the pushed advertisements, and dynamically adjusting the advertisement matching algorithm weights based on the user's interaction data feedback.
2. The multimodal VR advertising push method based on face recognition according to claim 1, characterized in that: The multimodal data includes image data, voice data and tactile data. The collection of the image data includes using at least two high-resolution RGB cameras, which are installed on both sides of the front of the VR helmet, respectively, to capture the user's facial expressions, eye movements and head posture in real time; the collection of the tactile data includes configuring a piezoelectric tactile sensor on the VR handle to detect the user's grip strength and operation gestures.
3. The multimodal VR advertising push method based on face recognition according to claim 1, characterized in that: The preprocessing of the multimodal data includes compressing the image data, performing noise reduction on the voice data, and normalizing the tactile data.
4. The multimodal VR advertising push method based on face recognition according to claim 3 is characterized in that: The network model using deep learning extracts features from the preprocessed multimodal data, including: Step S21, using a deep learning convolutional neural network to extract features from the image data, and the convolutional neural network uses ResNet-50; Step S22, using a deep learning recurrent neural network to extract features from the speech data, where the recurrent neural network uses a long short-term memory network; Step S23: extract features from the tactile data using a one-dimensional convolutional neural network.
5. The multimodal VR advertising push method based on face recognition according to claim 1, characterized in that: The weighted fusion of the extracted image features, speech features and tactile data to generate a unified multimodal feature vector includes: Step S31, judging based on the user's current scenario, and dynamically updating the weight coefficient using exponential smoothing method; The calculation formula for dynamically updating the weight coefficient of the exponential smoothing method is: in, is the weight coefficient of the mth modality (such as vision, hearing, touch) at time t, is the weight coefficient of the mth mode at time t-1, a is the smoothing coefficient, Preset weights for the scene of the mth mode; In step S32, the visual, auditory, and tactile feature vectors are mapped to a unified dimension through a fully connected layer of deep learning, and then key features are filtered through an attention mechanism to generate a fused multimodal feature vector.
6. The multimodal VR advertising push method based on face recognition according to claim 1, characterized in that: The multimodal VR advertising prediction model is constructed and trained, the fused multimodal feature vector is input into the multimodal VR advertising prediction model, and the user interest tags and behavior prediction results are output, including: Step S41: The multimodal VR advertising prediction model adopts a hybrid model architecture combining support vector machine and deep neural network; Step S42: training a multimodal VR advertising prediction model using a large amount of labeled multimodal data; Step S43: The fused multimodal feature vector is used as the input of the model, including visual features, auditory features, and information. The multimodal feature vector is integrated through a fully connected layer to ensure that each feature can affect the model's prediction results, and the user interest tag and behavior prediction results are output; In step S44, an online learning mechanism is adopted, and the multimodal VR advertising prediction model can receive new user data in real time and dynamically update model parameters to adapt to changes in user preferences.
7. The multimodal VR advertising push method based on face recognition according to claim 1, characterized in that: The step of obtaining user interaction data on pushed advertisements and dynamically adjusting advertisement matching algorithm weights based on user interaction data feedback includes: Step S61, obtaining user interaction data on the pushed advertisement, including click-through rate, dwell time, tactile feedback intensity, and voice evaluation; Step S62: Analyze the collected interaction data in depth to extract key features; Step S63: Using a reinforcement learning framework, dynamically adjust the weight of the advertisement matching algorithm based on user feedback.
8. A multimodal VR advertising push system based on face recognition, characterized in that: The multimodal VR advertising push method based on face recognition as described in claims 1-7 is applied to the system comprising: The data acquisition module is used to collect multimodal data of users in the virtual reality environment and pre-process the multimodal data; Feature extraction module, used to extract features from pre-processed multimodal data using a deep learning network model; The weighted fusion module is used to dynamically weight the image features, speech features, and tactile data after feature extraction to generate a unified multimodal feature vector; Modeling and training module, used to build and train multimodal VR advertising prediction models; The behavior prediction module is used to input the fused multimodal feature vector into the multimodal VR advertising prediction model and output user interest tags and behavior prediction results; The advertising push module is used to build an advertising database and select corresponding advertising content based on the user's current interest tags and historical behavior data, embedding it into the VR scene for push; The algorithm optimization module is used to obtain user interaction data on pushed ads and dynamically adjust the ad matching algorithm weights based on feedback from user interaction data.
9. A network side server, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal VR advertising push method based on face recognition as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal VR advertising push method based on face recognition according to any one of claims 1 to 7 is implemented.