Intelligent control system and method for a jukebox

Abstract

The application discloses an intelligent control system and method of a song ordering machine, comprising: a data acquisition module, which is used for acquiring facial image data and environmental audio data of a user; an emotion analysis module, which is used for calculating a current emotion state vector of the user through a pre-trained emotion recognition model based on the facial image data; an environment analysis module, which is used for identifying and extracting an environmental feature parameter set of a current environment based on the environmental audio data, wherein the environmental feature parameter set at least comprises an environmental sound pressure level, an environmental rhythm mode and a spectrum feature of existing music in the environment; a multi-modal fusion and decision module, which is used for performing multi-modal fusion on the current emotion state vector and the environmental feature parameter set, generating a comprehensive environmental-emotion index, and dynamically generating and recommending a matched candidate song list from a preset song library according to the comprehensive environmental-emotion index. The application can perform more intelligent control of the song ordering machine.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of control systems, in particular to an intelligent control system and method of a song selection machine. BACKGROUND

[0002] The song selection machine is a core audio playback control device for entertainment scenarios. Its core function is to provide search, selection and playback services for a preset song library, thereby meeting the needs of users to select songs independently and create atmosphere, and improving the entertainment experience of groups or individuals. With the upgrading of consumption, the song selection machine has gradually developed towards intelligence and scene from the early stage of supporting only manual search, and needs to adapt to diversified user needs, becoming a key carrier for atmosphere creation in entertainment scenarios.

[0003] The song selection machine control system is the core logic module for realizing the functions of the song selection machine, responsible for data acquisition, song selection decision and playback control. However, the existing control system has obvious technical defects and cannot meet the needs of intelligent entertainment. Specifically, there are problems such as single-dimensional perception, lack of multi-modal fusion capability, rigid recommendation logic, no dynamic optimization mechanism, insufficient adaptation to multi-user scenarios, and neglect of group needs. Therefore, an intelligent control system and method of a song selection machine are proposed. SUMMARY

[0004] The present application solves the above technical problems by the following technical solutions. The present application comprises:

[0005] A data acquisition module for acquiring facial image data and environmental audio data of a user;

[0006] An emotion analysis module connected to the data acquisition module, configured to calculate a current emotion state vector of the user based on the facial image data through a pre-trained emotion recognition model , wherein the emotion state vector , wherein represents the intensity probability value of the i-th basic emotion state, i = 1 … n;

[0007] An environment analysis module connected to the data acquisition module, configured to identify and extract an environmental feature parameter set of the current environment based on the environmental audio data, wherein the environmental feature parameter set at least includes environmental sound pressure level, environmental rhythm mode and spectral features of existing music in the environment;

[0008] A multi-modal fusion and decision module connected to the emotion analysis module and the environment analysis module, respectively, configured to perform multi-modal fusion on the current emotion state vector and the environmental feature parameter set to generate a comprehensive environment-emotion index and according to the comprehensive environment-emotion index dynamically generating and recommending a matched candidate song list from a preset song library;

[0009] The control execution module is connected with the multi-modal fusion and decision module, and is configured to receive a selection instruction made by the user from the candidate song list and control the song ordering machine to play a corresponding song.

[0010] Further, the emotion analysis module calculates an emotion state vector The process includes:

[0011] The collected facial image is preprocessed, including face detection, alignment and feature point extraction;

[0012] The processed image region is input into a deep convolutional neural network model optimized through transfer learning;

[0013] The output layer of the neural network model adopts a Softmax function, and the intensity probability distribution of the user corresponding to multiple basic emotion states is calculated, and the calculation process is represented as:

[0014] ;

[0015] wherein, is a high-level feature vector extracted from the fully connected layer of the neural network model, and W and b are the weight matrix and bias vector of the output layer, respectively.

[0016] Further, the multi-modal fusion and decision module generates the comprehensive environment-emotion index The calculation formula of the comprehensive environment-emotion index is:

[0017] ;

[0018] wherein, and are adaptive adjustment fusion weight coefficients, and + =1, and the initial value is preset by the system;

[0019] is a preset weight of the i-th emotion state;

[0020] is an environment quantification function, which is used to map the environment feature parameter set to a scalar value.

[0021] Further, the calculation process of the environment quantification function includes:

[0022] Fast Fourier transform is performed on the existing music signal in the environment to extract the energy distribution of the main melody frequency band thereof;

[0023] According to the predefined spectrum-emotion mapping table, the energy distribution is mapped to a basic environment emotion value ;

[0024] The ambient sound pressure level is introduced The final The calculation formula is:

[0025] ;

[0026] Wherein, is a correction coefficient, is a system preset comfortable reference sound pressure level.

[0027] Further, the multi-modal fusion and decision module dynamically generates a candidate song list, and the step includes:

[0028] The comprehensive environment-emotion index is converted into a target recommendation vector ;

[0029] The emotion label vector of each song in the song library is calculated The cosine similarity between the target recommendation vector is taken as the matching degree score , and the calculation formula is:

[0030] ;

[0031] According to the matching degree score The songs are sorted from high to low, and the top ranked songs are selected to form the candidate song list.

[0032] Further, the system further includes a feedback learning module;

[0033] The feedback learning module is used to record the final song selection behavior of the user;

[0034] The user's selection is compared with the system's initial recommendation, the recommendation deviation is calculated, and the fusion weight coefficient , of the deviation data is used for iterative optimization and update.

[0035] Further, the data acquisition module includes a rotatable high-definition camera array and a microphone array;

[0036] The control system realizes multi-user tracking through the camera array, and when multiple users are identified, the emotion state vector of each user The weighted average is performed to calculate the comprehensive sentiment vector of the group , and the calculation formula is:

[0037] ;

[0038] Wherein, K is the number of users, is the weight of the kth user, which is determined by the activity of the user, the relative distance from the jukebox and the historical song ordering frequency;

[0039] The multi-modal fusion and decision module uses to replace the single user's for subsequent fusion and decision.

[0040] Further, the multi-modal fusion and decision module also introduces the historical song ordering record data of the user when generating the candidate song list;

[0041] Based on the collaborative filtering algorithm, the similarity between the songs in the song library and the historical preference songs of the user is calculated to generate a user preference correction factor ;

[0042] The matching degree score and the user preference correction factor are combined by weighting to obtain the final recommendation score , and the calculation formula is:

[0043] ;

[0044] Wherein, is the balance weight, used to adjust the proportion between the environmental sentiment matching and the user's personal preference;

[0045] According to the final recommendation score , the songs are sorted to generate a candidate song list.

[0046] An intelligent control method of a jukebox, comprising the following steps:

[0047] Collecting at least the facial image data of the user and the environmental audio data;

[0048] Based on the facial image data, the current sentiment state vector of the user is calculated by a pre-trained emotion recognition model ;

[0049] Based on the environmental audio data, the environmental feature parameter set of the current environment is identified and extracted , and the environmental feature parameter set at least includes the environmental sound pressure level, the environmental rhythm mode and the frequency spectrum of the existing music in the environment.

[0050] characteristics;

[0051] the current emotional state vector and the environmental characteristic parameter set multi-modal fusion to generate a comprehensive environment-emotion index ;

[0052] According to the comprehensive environment-emotion index , a matched candidate song list is dynamically generated and recommended from the preset song library;

[0053] Receiving the selection instruction made by the user from the candidate song list, and controlling the song ordering machine to play the corresponding song.

[0054] The intelligent control system and method of the song ordering machine have the following advantages compared with the prior art: through multi-modal perception and fusion technology, the intelligent closed-loop control of the song ordering machine is realized. The system can automatically collect user emotional state and environmental characteristic data, generate an environment-emotion index through algorithm fusion, and autonomously decide to generate a personalized song list based on the index, and finally execute precise playback control. The entire control process does not require human intervention, and the system continuously optimizes the control strategy through real-time data analysis and machine learning, significantly improving the intelligent level of the song ordering machine, enabling the device to adapt to different user groups and scene changes, and realizing the integrated control of intelligent perception, intelligent decision-making and intelligent execution in a true sense. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is the system block diagram of the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application will be described in detail below. The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation methods and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0057] As shown in Figure 1 , the present embodiment provides a technical solution: an intelligent control system of a song ordering machine, comprising:

[0058] A data acquisition module for acquiring facial image data and environmental audio data of a user;

[0059] An emotional analysis module connected with the data acquisition module, for calculating the current emotional state vector of the user based on the facial image data through a pre-trained emotion recognition model , wherein represents the intensity probability value of the i-th basic emotional state, i=1……n;

[0060] An environment analysis module connected with the data collection module, configured to identify and extract an environment feature parameter set of a current environment based on the environment audio data , the environment feature parameter set at least includes an environment sound pressure level, an environment rhythm pattern, and a spectrum feature of existing music in the environment;

[0061] A multi-modal fusion and decision module connected with the emotion analysis module and the environment analysis module, configured to perform multi-modal fusion on the current emotion state vector and the environment feature parameter set to generate a comprehensive environment-emotion index , and according to the comprehensive environment-emotion index , dynamically generate and recommend a matched candidate song list from a preset song library;

[0062] A control execution module connected with the multi-modal fusion and decision module, configured to receive a selection instruction made by a user from the candidate song list, and control a song ordering machine to play a corresponding song.

[0063] The process of the emotion analysis module calculating the emotion state vector includes:

[0064] Preprocessing the collected facial image, including face detection, alignment, and feature point extraction;

[0065] Inputting the processed image region into a deep convolutional neural network model optimized through transfer learning;

[0066] The output layer of the neural network model adopts a Softmax function, and the intensity probability distribution of the user corresponding to multiple basic emotion states is calculated, and the calculation process is represented as:

[0067] ;

[0068] wherein, is a high-level feature vector extracted from the fully connected layer of the neural network model, and W and b are the weight matrix and bias vector of the output layer, respectively;

[0069] The above process can accurately separate the face region from the collected original facial image through face detection, alignment, and feature point extraction, and eliminate irrelevant interference such as background and posture deviation; at the same time, aligning the face size and angle ensures the consistency of the image features input into the model, avoiding recognition deviation caused by data format confusion.

[0070] Adopting deep convolutional neural network optimized by transfer learning, without training the model from scratch: the low-level features (such as edges, textures) learned by the pre-trained network (such as VGG, ResNet) on a large number of general images can be reused, and only a small amount of facial emotion data in the karaoke scene is needed to fine-tune the high-level network, which not only reduces the demand for labeled data, but also improves the emotion recognition accuracy of the model in specific scenarios such as "KTV, home entertainment", shortens the model training and inference time, and adapts to the real-time response requirements of the karaoke machine.

[0071] The output layer uses the Softmax function to convert the high-level feature vector extracted by the neural network into a probability distribution of the intensity of multiple basic emotional states, so that abstract emotions are converted into quantifiable probability values. This quantization result can directly provide accurate emotional input for the subsequent multi-modal fusion and decision-making module, avoiding the recommendation bias caused by ambiguous emotional descriptions.

[0072] For example, in the context of a party, the user uses the karaoke machine to order songs:

[0073] Scenario premise: The high-definition camera of the data acquisition module captures the facial image of user A (the original image contains a sofa background and the user slightly lowers his head), and the emotional state vector needs to be calculated .

[0074] Step 1: Image preprocessing;

[0075] Face detection: Use the MTCNN algorithm to locate the face area of user A from the original image, with a coordinate frame of , eliminating the sofa background;

[0076] Face alignment: According to the detected "eyes, nose tip, and corners of the mouth" six feature points, correct the low head angle (about 15°) to a frontal face, and scale it to a standard size of 224x224 pixels;

[0077] Feature point extraction: Extract 68 facial key feature points (such as eyebrow curvature, mouth corner angle), forming the pre-processed face image area.

[0078] Step 2: Input transfer learning optimized CNN;

[0079] Input the pre-processed 224x224 image into the convolutional neural network based on ResNet50 transfer learning:

[0080] Freeze the first 17 layers of ResNet50 (responsible for extracting low-level features such as edges and skin color), and only fine-tune the last 5 layers (adapt to the high-frequency emotions of happy, calm, excited, and sad in the karaoke scene);

[0081] After convolution, pooling, and fully connected layer operations, extract the high-level feature vector from the fully connected layer (fc2 layer) of the network , assuming the extraction result is:

[0082] ;

[0083] 6-dimensional vector, corresponding to "eyebrow features, eye openness, mouth corner curvature", and other key emotion-related features.

[0084] Step 3: Softmax function calculates emotion state vector ;

[0085] Assuming the weight matrix W of the neural network output layer (4 rows and 6 columns, corresponding to 4 emotions) and the bias vector b are as follows:

[0086] ;

[0087] ;

[0088] Calculate the linear transformation result ;

[0089] According to the formula , calculate the linear transformation value corresponding to the 4 emotions respectively:

[0090] Happy ( ): 0.4x0.7+0.2x0.3+0.5x0.5+0.1x0.2+0.6x0.6+0.3x0.4+0.15=0.28+0.06+0.25+0.02+0.36+0.12+0.15=1.24;

[0091] Calm ( ): 0.2x0.7+0.3x0.3+0.1x0.5+0.4x0.2+0.2x0.6+0.5x0.4+0.1=0.14+0.09+0.05+0.08+0.12+0.2+0.1=0.78;

[0092] Excited ( ): 0.3x0.7+0.1x0.3+0.4x0.5+0.2x0.2+0.5x0.6+0.2x0.4+0.2=0.21+0.03+0.2+0.04+0.3+0.08+0.2=1.06;

[0093] Sad ( ): 0.1x0.7+0.4x0.3+0.2x0.5+0.3x0.2+0.1x0.6+0.4x0.4+0.05=0.07+0.12+0.1+0.06+0.06+0.16+0.05=0.62;

[0094] The Softmax function is calculated ;

[0095] The calculation formula of the Softmax function is:

[0096] z is the output vector of the linear transformation of the neural network, that is, the linear transformation result (preliminary emotion prediction value);

[0097] z1, z2, z3, and z4 are four components of the vector z, which correspond to the linear prediction scores (original outputs without probability normalization) of the four emotions "happy, calm, excited, and sad";

[0098] , , and are the exponential operation results of the linear scores, which map the linear scores to the "positive number space" using the exponential function, and amplify the differences between different emotion scores (the emotion with a higher score will have a more obvious advantage after the exponential operation;

[0099] k is an index variable for summing, which is used to sequentially traverse the serial numbers of the four basic emotions (k = 1, 2, 3, 4, corresponding to "happy, calm, excited, and sad" respectively), so as to sum the (exponential result of the linear score) corresponding to each emotion to obtain the denominator required for normalization;

[0100] Calculate the numerator (exponential value): ;

[0101] Calculate the denominator (sum): 3.45 + 2.18 + 2.89 + 1.86 = 10.38;

[0102] Calculate the emotion probability: ;

[0103] The emotion state vector of user A is:

[0104] ;

[0105] The results show that the probability of user A being happy is 33%, and the probability of being excited is 28%. The subsequent module can quantify the emotion based on this and recommend happy songs (such as birthday songs and popular dance songs).

[0106] The multi-modal fusion and decision module generates the comprehensive environment-emotion index The calculation formula is:

[0107] ;

[0108] wherein, and The fusion weight coefficient is self-adaptable, and + =1, and the initial value is preset by the system;

[0109] The preset weight of the ith emotional state is;

[0110] The environmental quantification function is used to map the environmental characteristic parameter set to a scalar value;

[0111] In the formula (emotional weight coefficient) and (environmental weight coefficient) are self-adaptable parameters, and strictly satisfy + =1, which can dynamically balance the proportion of user emotional needs and environmental atmosphere influence according to the song ordering scene.

[0112] By introducing the preset weight of the ith emotional state through (weighted emotional value), the high-value emotion can be given higher weight and the low-association emotion can be reduced in weight according to the entertainment attribute of song ordering;

[0113] The high-dimensional emotional state vector and the complex environmental characteristic parameter set are fused into a single scalar index through the formula, without subsequent module processing of multi-dimensional original data, and can be directly used as the core basis to generate the target recommendation vector, which greatly simplifies the intermediate link from data collection to song list decision, and improves the real-time performance and accuracy of the song ordering machine recommendation.

[0114] For example, in a party scene, the following parameters are known:

[0115] Emotional state vector: That is, n=4, four basic emotions;

[0116] Environmental quantification value: (Calculation basis: environmental sound pressure level , comfortable reference sound pressure level , correction coefficient , basic environmental emotional value , ;

[0117] Fusion weight coefficient (party scene preset): (focus on emotion), (taking into account the environment);

[0118] Emotional preset weight (party scene adaptation): , the weights of happiness and excitement are higher, which is in line with the atmosphere of the party.

[0119] Calculate the comprehensive environment-emotion index :

[0120] Step 1: Calculate the weighted emotion value ;

[0121] According to the formula , substitute the components of the emotion vector and the corresponding weights:

[0122] = 0.4 x 0.33 + 0.2 x 0.21 + 0.3 x 0.28 + 0.1 x 0.18 = 0.132 + 0.042 + 0.084 + 0.018 = 0.276;

[0123] Step 2: Substitute the core formula to calculate :

[0124] According to the formula , substitute all parameters:

[0125] ;

[0126] Finally (take value range 0~1), this index comprehensively reflects the user's A's happiness / excitement emotion + light happy environment atmosphere in the party scene. The subsequent multi-modal fusion and decision-making module can directly convert into the target recommendation vector , and then filter out happy songs (such as "Friend xxx") with high matching degree, avoiding recommending sad songs (such as "Sorry xxx") or too soothing songs (such as "Moon xx light"), which fully meet the entertainment needs of family parties.

[0127] Further, the calculation process of the environment quantification function includes:

[0128] Perform a fast Fourier transform on the existing music signal in the environment to extract the energy distribution of its main melody frequency band;

[0129] According to the predefined frequency spectrum-emotion mapping table, the energy distribution is mapped to a basic environment emotion value ;

[0130] Introduce the ambient sound pressure level for correction, and the final calculation formula of is:

[0131] ;

[0132] Where, The correction coefficient, The comfort reference sound pressure level preset by the system.

[0133] The step of dynamically generating the candidate song list by the multi-modal fusion and decision module includes:

[0134] The comprehensive environment-emotion index is converted into a target recommendation vector ;

[0135] The emotion label vector of each song in the song library is calculated The cosine similarity of the target recommendation vector is taken as the matching degree score , and the calculation formula is:

[0136] ;

[0137] According to the matching degree score , the songs are sorted from high to low, and the top-ranked songs are selected to form the candidate song list;

[0138] The cosine similarity is used , and the core is to measure the direction consistency of the target recommendation vector T and the song emotion label vector Sj, rather than simply comparing the vector numerical value. This way can accurately filter out songs that match the emotional tone, atmosphere attribute and scene demand, and avoid pseudo-matching recommendations caused by vector length (such as mispushing sad songs in party scenes).

[0139] The target recommendation vector T is converted from the comprehensive environment-emotion index Γ, and Γ has fused user emotions and environmental characteristics, so the song matching is essentially a screening based on the core of the scene demand, rather than a separate matching of emotions or environment. This can ensure that the candidate song list always fits the current scene and avoids the disconnection between the song list and the actual needs of the user.

[0140] By sorting from high to low, the top-ranked songs are directly selected to form the candidate song list, without the need for the user to filter from a large number of song libraries. This "precise screening + priority sorting" mode greatly reduces the user's operation cost, while ensuring that the user can see the most matched songs first, improving the efficiency and experience of song selection.

[0141] If the party scene is continued, the scene premise is:

[0142] The following parameters are known:

[0143] Comprehensive environment-emotion index: ;

[0144] Target recommendation vector T: convert Convert to 4-dimensional emotion vector (corresponding to "happy, calm, excited, sad" 4 emotions, consistent with the emotional dimensions of claims 2, 3), set to ("happy" and "excited" have a higher proportion, which meets the needs of the party);

[0145] Preset song library candidate songs and emotion label vectors (j is the song number, the vector dimension is consistent with T):

[0146] Song 1: "Friends xxx" (cheerful type) -> ;

[0147] Song 2: "xx Blessing Song" (cheerful type) -> ;

[0148] Song 3: "Sorry xxx" (sad type) -> ;

[0149] Song 4: "Moon xx light" (relaxing type) -> .

[0150] Calculate and generate a candidate song list

[0151] Step 1: Clearly define the cosine similarity formula:

[0152] The matching score calculation formula is:

[0153] ;

[0154] Where:

[0155] is the dot product of the vector (reflecting the consistency of the direction);

[0156] is the length of T, is the length of (length of the vector, used for normalization).

[0157] Step 2: Calculate the basic parameters (length of T)

[0158] First calculate :

[0159] = = ;

[0160] Step 3: Calculate the song :

[0161] Song 1 "Friends xxx" ( )

[0162] Dot product: ;

[0163] Module length: ;

[0164] Similarity: ;

[0165] (2) Song 2 "xx Blessing Song" ( )

[0166] Dot product: ;

[0167] Module length: ;

[0168] Similarity: ;

[0169] (3) Song 3 "It's a pity xxx" ( )

[0170] Dot product: ;

[0171] Module length: =

[0172] ;

[0173] Similarity: ;

[0174] (4) Song 4 "xx Moonlight" ( )

[0175] Dot product: ;

[0176] Module length: ;

[0177] Similarity: ;

[0178] Step 4: Sort and generate candidate playlists

[0179] Sort the dot products of the 4 songs from high to low:

[0180] 1. Song 1 "Friend xxx": ;

[0181] 2. Song 2 "xx Blessing Song": ;

[0182] 3. Song 4 "xx Moonlight": ;

[0183] 4. Song 3 "It's a pity xxx":​​ .

[0184] Select the top 2 songs to form a candidate song list: "Friends xxx" and "xx blessing song", which completely match the "happy and warm" scene requirements of family gatherings, avoiding recommending sad or too soothing songs.

[0185] The system also includes a feedback learning module;

[0186] The feedback learning module is used to record the user's final song selection behavior;

[0187] Compare the user's selection with the system's initial recommendation, calculate the recommendation deviation, and use the fusion weight coefficient of the deviation data , Iterative optimization and update;

[0188] The feedback learning module can adjust the values of (emotional weight) and (environmental weight) based on the deviation between the user's actual selection and the initial recommendation. Avoiding the disconnection between fixed weights and requirements due to changes in the scene or user preference migration, allowing the weights to adapt to the user's real-time selection habits.

[0189] The initial recommendation is based on objective data of emotion and environment, but users may have subjective preferences that are not reflected in objective data. The feedback module compares the highly matched songs recommended by the system with the songs actually selected by the user, quantifies the deviation and corrects the decision logic, so that subsequent recommendations are more in line with the user's implicit preferences, rather than relying solely on objective data.

[0190] In the gathering scenario, the key parameters of the prequel module are known:

[0191] Initial fusion weight (system default) (emotional weight), (environmental weight);

[0192] Initial comprehensive environment-emotion index: ;

[0193] Initial candidate song list and matching score:

[0194] 1. "Friends xxx" (happy type): (recommended rank 1);

[0195] 2. "xx blessing song" (happy type): (recommended rank 2);

[0196] 3. "Moon xx light" (soothing type): (recommended rank 3);

[0197] 4. "It's a pity xxx" (sadness category): (Ranking recommendation 4);

[0198] Feedback learning module parameters: learning rate (Control the weight adjustment range to avoid excessive fluctuations), the recommended deviation D is defined as "the matching degree score of the song selected by the user" (the lower the score, the greater the deviation, and the greater the weight adjustment range).

[0199] Weight optimization process of feedback learning module:

[0200] Step 1: Record the actual selection behavior of the user:

[0201] After the system recommends the candidate song list, the user does not choose the second-ranked happy song, but chooses the third-ranked soothing song "Moonlight", that is:

[0202] User-selected song: "Moonlight";

[0203] Initial matching degree score of selected song: .

[0204] Step 2: Calculate the recommended deviation D;

[0205] According to the definition of the feedback module, the recommended deviation D quantifies the difference between the initial recommendation and the user's choice, and the formula is:

[0206] ;

[0207] Substitute :

[0208] D = 1 - 0.516 = 0.484;

[0209] D = 0.484 indicates that there is a significant deviation between the initial recommendation and the user's preference (the higher the deviation value, the greater the difference), which requires adjustment and .

[0210] Step 3: Iteratively optimize the fusion weights based on the deviation and ;

[0211] Since the user selected a soothing song, it indicates that the initial weight of (emotional weight, focusing on "happy / excited") is too high, (environmental weight, possibly implying an "environmental signal that needs to be relaxed") is too low. The feedback module optimizes the weights by reducing and increasing , and the updated formula is:

[0212] ;

[0213] ;

[0214] For learning rate, control the magnitude of each adjustment of the weight, to avoid the weight mutation caused by a single deviation;

[0215] Substitute parameters:

[0216] ;

[0217] ;

[0218] New weight after optimization: , , the environment weight is improved, which is more in line with the user's preference for soothing songs.

[0219] Step 4: Verify the effect of the optimized recommendation (next time in the same scenario);

[0220] If the user is again in the same family gathering environment (the emotional state vector is still (0.33, 0.21, 0.28, 0.18), and the environment quantization value is still 0.76), use the new weight to calculate the new comprehensive environment-emotion index :

[0221] ;

[0222] Given , substitute parameters:

[0223] = 0.1518 + 0.342 = 0.4938 ≈ 0.49;

[0224] (Slightly higher than ), the corresponding target recommendation vector will appropriately reduce the "happy / excited" weight and increase the "calm" weight (more in line with the emotional label of soothing songs). Next time, the matching score of "Moon xx light" in the candidate song list will increase (assuming it increases to 0.72), and the ranking will rise to the second place, making it easier for the user to select, and the recommendation accuracy will be significantly improved.

[0225] The feedback learning module calculates the deviation of the user's selection once, and optimizes from 0.6 to 0.55, from 0.4 to 0.45, so that the subsequent recommendations in the same scenario are more in line with the user's implicit demand for "preferring soothing songs". If the user selects similar style songs multiple times, the deviation D will gradually decrease, and the weight will continue to iterate to the optimal value, achieving the self-optimization effect of the system getting more accurate with use.

[0226] The data acquisition module includes a rotatable high-definition camera array and a microphone array;

[0227] The control system realizes multi-user tracking through the camera array, and when multiple users are identified, the emotional state vectors of each user are weighted and averaged to calculate the comprehensive emotional vector of the group The calculation formula is:

[0228]

[0229] Where K is the number of users, is the weight of the kth user, which is determined by the user's activity, relative distance from the jukebox, and historical song selection frequency;

[0230] The multi-modal fusion and decision module uses instead of a single user for subsequent fusion and decision-making;

[0231] Traditional jukeboxes are designed for single users, and in multi-person scenarios such as family gatherings and KTV boxes, identifying only the emotional state of a single user can lead to biased recommendations (such as only considering the host's emotions and ignoring the needs of guests). This case tracks all users through a rotatable camera array while collecting multi-user emotional data, upgrading from single-user perception to group perception, and adapting to the core multi-person entertainment scenario of jukeboxes.

[0232] In a gathering scenario, the following parameters are known:

[0233] Data acquisition module: Rotatable high-definition camera array (angle covers all areas of the living room) identifies and tracks 3 users, and microphone array assists in determining user activity;

[0234] Single-user emotional state vector (emotional dimensions: happy, calm, excited, sad):

[0235] User A (initiator of the gathering, often selects songs):

[0236] User B (friend of A, actively interacts):

[0237] User C (family member of A, quietly observes):

[0238] User weight Calculation rules: activity level , relative distance (take the reciprocal, the closer the value is the larger it is), historical song selection frequency ​​​​​each occupies weight, i.e. All parameters are normalized to the interval 0-1.

[0239] The activity, distance, and historical frequency of each user:

[0240] User A: (sporadic interaction), (the closest to the jukebox), (the most historical jukebox);

[0241] User B: (frequent hand-raising interaction), (far from the jukebox), (sporadic jukebox);

[0242] User C: (almost no interaction), (the farthest from the jukebox), (rare jukebox).

[0243] Calculate the group's comprehensive sentiment vector :

[0244] Step 1: Calculate the weight of each user ;

[0245] According to the weight rule , calculate by user:

[0246] User A (k = 1):

[0247] ;

[0248] User B (k = 2):

[0249] ;

[0250] User C (k = 3):

[0251] ;

[0252] Step 2: Calculate the sum of weights :

[0253] ;

[0254] Step 3: Calculate the numerator ;

[0255] It is a 4-dimensional vector, and the weighted sum needs to be calculated according to the 4 dimensions of "happy, calm, excited, and sad":

[0256] Happy dimension (first component):

[0257] ≈0.286+0.3+0.056=0.642;

[0258] Calm dimension (2nd component):

[0259] ≈0.182+0.100+0.139=0.421;

[0260] Excited dimension (3rd component): ≈0.243+0.200+0.042=0.485;

[0261] Sad dimension (4th component):

[0262] ≈0.156+0.067+0.042=0.265;

[0263] Therefore, the molecular vector is: ;

[0264] Step 4: Calculate the group comprehensive emotion vector

[0265] According to the formula: ;

[0266] Substitute the molecular vector and the sum of weights:

[0267] ;

[0268] Compare the single-user emotion vector and the group vector:

[0269] If only user A's is used, the requirement of user B "more happy (0.45)" will be ignored;

[0270] If only user C's is used, it will be overly biased towards "calm", which does not conform to the atmosphere of the party;

[0271] The group vector integrates A's "excitement" (high weight), B's "happiness", and C's "calmness", which is more in line with the overall atmosphere of "most people want to be happy but not noisy" in a family gathering.

[0272] The subsequent multi-modal fusion and decision-making module uses instead of single-user calculation , and the generated candidate song list will be more balanced in "happy" and "warm" attributes (such as including both "Friends xxx" and "xxx lucky"), avoiding the bias towards single user preferences, and improving the overall satisfaction in a 3-person scenario.

[0273] The multi-modal fusion and decision module also introduces the historical karaoke record data of the user when generating the candidate song list;

[0274] Based on the collaborative filtering algorithm, the similarity between the songs in the song library and the user's historical preference songs is calculated to generate a user preference correction factor .

[0275] The matching degree score and the user preference correction factor are combined by weighting to obtain the final recommendation score , and the calculation formula is:

[0276] .

[0277] Among them, is the balance weight, used to adjust the proportion between environmental emotional matching and personal preference of the user;

[0278] According to the final recommendation score , the songs are sorted to generate the candidate song list;

[0279] Only by the current environment-emotional index calculation , it may not be consistent with the user's long-term habits. The introduction of historical karaoke record can make the recommendation take into account the current scene and the user's long-term preference, avoiding invalid recommendations that are suitable for the scene but not liked by the user. By calculating through the collaborative filtering algorithm, the implicit preference of the user which is not explicitly expressed can be captured, so that the recommendation is more in line with the user's real music preference.

[0280] For example, in the party (user A) scene, the matching degree score is known:

[0281] "Friend xxx" (happy): .

[0282] "xx blessing song" (happy): .

[0283] "moon xx light" (relaxed): .

[0284] "Regret xxx" (sad): .

[0285] The user A's historical karaoke preference is that among the last 10 times of karaoke, 7 times are relaxed (such as "moon xx light" and "xxx lucky"), and 3 times are happy, and the implicit preference is relaxed songs.

[0286] The preference correction factor is calculated by the collaborative filtering algorithm, and the correction factor The higher the value, the more similar to the historical preference:

[0287] "Friend xxx": , happy, low similarity to historical preference;

[0288] "xx blessing song": , happy, low similarity;

[0289] "moon xx light": , soothing, high similarity;

[0290] "Regret xxx": , sad, very low similarity.

[0291] Set the balance weight , focus on the current scene, and take into account historical preferences.

[0292] Calculate the final recommendation score And sort:

[0293] Step 1: Clearly define the core formula:

[0294] ;

[0295] Where: , .

[0296] Step 2: Calculate each song :

[0297] "Friend xxx" j=1:

[0298] ;

[0299] "xx blessing song" j=2:

[0300] ;

[0301] "moon xx light" j=3:

[0302] ;

[0303] "Regret xxx" j=4:

[0304] ;

[0305] Step 3: Sort to generate the final candidate song list 1. "Friend xxx":

[0306] ; 2. "xx blessing song":

[0307] ; ;

[0308] 3. Moonlight: ;

[0309] 4. Regret: .

[0310] Compared with the ranking without introducing historical preferences, after introducing :

[0311] Moonlight from the original 0.630, the ranking is still third, but the gap with the top two has narrowed (the original gap is 0.974-0.516=0.458, and the current gap is 0.704-0.630=0.074);

[0312] If the user's historical preferences are stronger (such as ), , can surpass xx blessing song and rank second, fully consistent with the user's long-term preferences, and achieve the balance between scene adaptation and personal preferences.

[0313] A kind of intelligent control method of jukebox, comprising the following steps:

[0314] Collect at least the facial image data of user and environmental audio data;

[0315] Based on the facial image data, the current emotional state vector of the user is calculated by the pre-trained emotion recognition model ;

[0316] Based on the environmental audio data, the environmental feature parameter set of the current environment is identified and extracted , the environmental feature parameter set at least includes environmental sound pressure level, environmental rhythm mode and spectral characteristics of existing music in the environment;

[0317] The current emotional state vector and the environmental feature parameter set are multi-modal fusion to generate a comprehensive environment-emotion index ;

[0318] According to the comprehensive environment-emotion index , a matching candidate song list is dynamically generated and recommended from the preset song library;

[0319] Receive the selection instruction made by the user from the candidate song list, and control the jukebox to play the corresponding song.

[0320] In addition, the terms "first", "second", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or a specific number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0321] In the description of the present application, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples, without contradiction.

[0322] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An intelligent control system for a karaoke machine, characterized in that, Comprise: a data acquisition module for acquiring facial image data and ambient audio data of a user; an emotion analysis module configured to calculate a current emotion state vector of the user based on the facial image data by a pre-trained emotion recognition model ; An environmental analysis module is configured to identify and extract a set of environmental feature parameters of a current environment based on environmental audio data The set of environmental feature parameters At least includes environmental sound pressure level, environmental rhythm pattern and spectral features of music already existing in the environment a multi-modal fusion and decision module, configured to perform multi-modal fusion on the current emotional state vector and the environmental feature parameter set to generate a comprehensive environmental-emotional index , and according to the comprehensive environmental-emotional index , dynamically generate and recommend a matched candidate song list from a preset song library; a control execution module for receiving a selection instruction made by the user from a candidate song list and controlling a jukebox to play a corresponding song; The multi-modal fusion and decision module generates a comprehensive environment-emotion index The calculation formula is: ； wherein, and is an adaptive fusion weight coefficient, and + = 1, and the initial value is preset by the system. preset weight for the i-th emotional state, represents the intensity probability value of the i-th basic emotional state; An environmental quantification function for mapping a set of environmental characteristic parameters to a scalar value. An environmental quantification function for mapping a set of environmental characteristic parameters to a scalar value.

2. The intelligent control system of a song ordering machine according to claim 1, characterized in that: The sentiment analysis module computes a sentiment state vector The process includes: preprocessing the acquired facial image, including face detection, alignment and feature point extraction; inputting the processed image region into a deep convolutional neural network model optimized through transfer learning; the output layer of the neural network model adopts a Softmax function, and the intensity probability distribution of the user corresponding to multiple basic emotional states is calculated, and the calculation process is represented as: ； wherein, are the high-level feature vectors extracted from the fully connected layers of the neural network model, W and b are the weight matrix and bias vector of the output layer, respectively.

3. The intelligent control system of a song ordering machine according to claim 1, characterized in that: The environmental quantification function The calculation process includes: performing fast Fourier transform on the existing music signal in the environment to extract the energy distribution of the main melody frequency band; mapping the energy distribution to a base ambient affect value according to a predefined spectrum-affect mapping table ; Introducing ambient sound pressure level Amendments are made and final .

4. The intelligent control system of a song ordering machine according to claim 1, characterized in that: the multi-modal fusion and decision module dynamically generates a candidate song list, comprising: converting the integrated environmental-emotional index to a target recommendation vector ; computing an emotional tag vector for each song in a song library cosine similarity with a target recommendation vector as a match degree score , the formula is: ； According to the matching degree score The songs are ranked from high to low, and the top ranked songs are selected to form a candidate song list.

5. The intelligent control system of a song ordering machine according to claim 4, characterized in that: The system further comprises a feedback learning module; the feedback learning module is used to record the final song selection behavior of the user; Comparing the user's selection with the system's initial recommendation, calculating a recommendation deviation, and using a fusion weight coefficient of the recommendation deviation , Iterative optimization update is performed.

6. The intelligent control system of a song ordering machine according to claim 1, characterized in that: the data acquisition module comprises a rotatable high-definition camera array and a microphone array; The control system achieves multi-user tracking through the camera array, and when multiple users are identified, the emotional state vectors of each user are weighted and averaged to calculate a composite emotional vector for the group ;​ The multi-modal fusion and decision module uses replaces a single user to perform subsequent fusion and decision making.

7. The intelligent control system of a song ordering machine according to claim 4, characterized in that: when generating the candidate song list, the multi-modal fusion and decision module also introduces the historical jukebox record data of the user; Based on the collaborative filtering algorithm, the similarity of the songs in the song library and the user historical preference songs is calculated to generate a user preference correction factor ; combining the match score with a user preference modifier combining the match score with a user preference modifier combining the match score with a user preference modifier .

8. A method for intelligent control of a jukebox based on the control system of any of claims 1-7, characterized by: comprising the following steps: acquire at least the facial image data and ambient audio data of a user; Based on the facial image data, a current emotional state vector of the user is calculated by a pre-trained emotion recognition model ; Based on the environmental audio data, identify and extract the environmental feature parameter set of the current environment , the environmental feature parameter set At least includes environmental sound pressure level, environmental rhythm mode and spectral features of music already existing in the environment; a current emotional state vector a set of environmental feature parameters perform multi-modal fusion to generate a comprehensive environment-emotion index ; According to the comprehensive environment-emotion index dynamically generating and recommending a matched candidate song list from a preset song library; receive a selection instruction made by the user from a candidate song list and control a jukebox to play a corresponding song.

Images