Music collaborative playing system and method based on virtual reality

Abstract

The invention discloses a music collaborative playing system and method based on virtual reality. The method comprises the following steps: collecting a scene panoramic video and a scene audio in real time; capturing character activities in a scene panoramic video, identifying a video background, obtaining scene audio frequency, obtaining keywords by using a machine learning model, and constructing a keyword group; obtaining adaptive audio from the database according to the keyword group, judging whether the adaptive audio is replaced or not according to the replay rate of the adaptive audio, and analyzing similar users to obtain the adaptive audio again; the adaptive audio is played, the playing satisfaction degree is obtained according to the character emotion fluctuation condition, and the keyword weight in the audio keyword group is adjusted; real-time character actions are achieved, and when a specific action is recognized, adaptive audio playing is automatically adjusted; the keyword weight of the keyword group of the adaptive audio is adjusted, so that the audio matching accuracy is improved; and the adaptive audio is obtained from the historical playing audio of the similar user according to the keyword group, so that the richness of the adaptive audio is improved.

Description

Technical Field

[0001] The present invention relates to the field of image communication technology, and in particular to a music collaborative playing system and method based on virtual reality. Background Art

[0002] In recent years, with the rapid rise of artificial intelligence technology, mobile phones and computers have gradually become indispensable necessities of life. When using mobile phones and computers for work or leisure and entertainment, playing music has become a habit for most people.

[0003] At present, the Chinese invention patent with application number 201110460491.4 provides a scene-based music playback processing method and a music playback device, which can store the mapping relationship between feature information and music lists in the music playback device in advance, and by obtaining the feature information of the application scene, if the music list corresponding to the application scene can be determined according to the mapping relationship between the feature information and the music list, the music content in the music list is played, so that the music content played by the music playback device can be automatically switched based on the application scene, thereby improving the user experience. However, for the same scene, the music playback repetition rate is prone to be too high, and the mapping relationship between the music list and the feature information is not adjusted according to the playback results, and the accuracy of the music playback cannot be guaranteed. At the same time, when adjusting the audio playback, the user is required to switch manually, which reduces the user experience. Summary of the invention

[0004] The technical problem solved by the present invention is that the prior art easily leads to a high music playback repetition rate for the same scene, and does not adjust the mapping relationship between the music list and the feature information according to the playback results, and cannot guarantee the accuracy of the music playback. At the same time, when adjusting the audio playback, the user is required to manually switch, which reduces the user experience.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a music collaborative playing method based on virtual reality, comprising the following steps:

[0006] Step S1, real-time acquisition of on-site panoramic video and on-site audio;

[0007] Step S2, capturing the activities of people in the panoramic video of the scene, identifying the background of the video and obtaining the audio frequency of the scene, using the machine learning model to analyze and obtain the activity keywords, background keywords and emotional keywords, and constructing the activity keywords, background keywords and activity keywords into a keyword group;

[0008] Step S3, obtaining the adapted audio from the database according to the keyword group, judging whether to change the adapted audio according to the replay rate of the adapted audio, re-obtaining the adapted audio by analyzing similar users using the collaborative filtering recommendation algorithm, and adjusting the adapted audio frequency according to the on-site audio volume;

[0009] Step S4, playing the adapted audio, timely capturing the emotions of the characters in the on-site panoramic video, obtaining the playback satisfaction according to the emotional fluctuations of the characters, and adjusting the keyword weights in the audio keyword group according to the playback satisfaction;

[0010] Step S5, real-time monitoring of the person's movements in the on-site panoramic video, and automatically adjusting the audio playback when a specific movement is recognized;

[0011] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0012] The step S2 specifically includes the following steps:

[0013] Step S201, using portrait segmentation technology to separate the person and the background of the on-site panoramic video, and obtaining the person frequency band and the background frequency band;

[0014] Step S202, using a target detection algorithm to capture each person target in the person frequency band, identifying the target person's action through action recognition technology, and obtaining the activity keyword corresponding to the action;

[0015] Step S203, obtaining an image matrix of the face area of ​​the target person through a face recognition model as an input image for expression recognition, and obtaining emotional keywords of the target person;

[0016] Step S204, using the background recognition model to obtain background keywords of the background frequency band;

[0017] Step S205, constructing a keyword group by matching the activity keywords, emotion keywords and background keywords according to a certain weight ratio;

[0018] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0019] The step S3 specifically comprises the following steps:

[0020] Step S301, matching audio according to the keyword group to obtain adapted audio;

[0021] Step S302, obtaining the local replay rate of the adapted audio, if the replay rate is greater than the expected threshold, obtaining similar users of the user according to the static attributes and dynamic attributes of the user portrait, wherein the static attributes include gender, age, geographic location and favorite singer, and the dynamic attributes include user historical operation records;

[0022] Step S303, reacquiring adapted audio from historical playback records of similar users according to the keyword group;

[0023] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0024] The step S4 specifically includes:

[0025] Step S401, playing adapted audio;

[0026] Step S402, obtaining the live sound frequency through the live audio, and adjusting the adapted audio playback frequency according to the live sound frequency;

[0027] Step S403, capturing the on-site panoramic video within a unit time after the adapted audio is played;

[0028] Step S404, obtaining key frames of the on-site panoramic video;

[0029] Step S405, using a face recognition model to obtain an image matrix of the face area of ​​the target person in the key frame;

[0030] Step S406, inputting the image matrix into the expression recognition model, obtaining the emotional keywords of the target person in each key frame, obtaining the emotional fluctuation of the target person by comparing the emotional keywords, and obtaining the playback satisfaction of the adapted audio according to the emotional fluctuation of the target person;

[0031] Step S407, adjusting the keyword weights in the keyword group of the adapted audio according to the value of the playback satisfaction, if the value of the playback satisfaction is greater than or equal to a certain expected threshold, then increasing the keyword weights of the keyword group of the adapted audio according to the value of the playback satisfaction, if the value of the playback satisfaction is less than or equal to the expected threshold, then decreasing the keyword weights of the keyword group of the adapted audio according to the value of the playback satisfaction;

[0032] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0033] The step S5 specifically comprises the following steps:

[0034] Step S501, real-time monitoring of human movements in the on-site panoramic video;

[0035] Step S502, comparing the character action with the set playback adjustment action to obtain the action similarity, and if the action similarity is greater than or equal to the expected threshold, adjusting the adaptive audio playback according to the playback adjustment instruction corresponding to the playback adjustment action;

[0036] The playback adjustment instructions include song switching instructions, frequency adjustment instructions and speed adjustment instructions;

[0037] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0038] The step S202 specifically includes the following steps:

[0039] Step S2021, after preprocessing and labeling the human motion data set, input it into the machine learning model for training to obtain a motion recognition model;

[0040] Step S2022, using image recognition technology to identify key points of the character band, and obtaining action group nodes corresponding to the identified action key points;

[0041] Step S2023, inputting the action group frame node into the action recognition model to obtain the corresponding activity keyword;

[0042] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0043] The step S203 specifically includes the following steps:

[0044] Step S2031, obtaining a face data set and a non-face data set, inputting the face data set and the non-face data set into a machine learning model after data processing, and obtaining a face recognition model after training;

[0045] Step S2032, inputting the character frequency band into the face recognition model to obtain an image matrix of the face area of ​​the target character;

[0046] Step S2033, obtaining a face area dataset, inputting the labeled face area dataset into a machine learning model for training, and obtaining an expression recognition model;

[0047] Step S2034, inputting the image matrix of the face area of ​​the target person into the expression recognition model to obtain the emotional keywords of the target person;

[0048] As a preferred solution of the virtual reality-based music collaborative playing method described in the present invention, wherein:

[0049] The step S301 includes:

[0050] Step S3011, obtaining an audio data packet, performing data preprocessing and keyword tagging on the audio data packet, inputting the keyword-tagged audio data packet into a machine learning model for training, obtaining an audio feature extraction model, and performing keyword extraction on the audio in the database;

[0051] Step S3012, matching the keyword group with the database audio;

[0052] Step S3013, if the matching degree between the keyword group and the keyword of the database audio is greater than the expected threshold, the matching audios are organized into an audio play list according to the matching degree from high to low, and the adapted audios are played in the order of the audio play list.

[0053] In the second aspect, a music collaborative playback system based on virtual reality includes a collection module, a music recommendation module and a music playback module;

[0054] The acquisition module is used to acquire the on-site panoramic video and on-site audio in real time;

[0055] The music recommendation module is used to obtain a scene keyword group according to the scene panoramic video, and obtain an adaptation audio according to the keyword group;

[0056] The music playing module is used to play the adapted audio and adjust the audio frequency according to the live audio frequency and adjust the audio playing according to the character's actions;

[0057] As a preferred solution of the virtual reality-based music collaborative playback system and method described in the present invention, wherein:

[0058] The music recommendation module includes a site recommendation unit and a similar user recommendation unit;

[0059] The on-site recommendation unit is used to analyze the on-site panoramic video to obtain activity keywords, emotion keywords and background keywords, and to construct keyword groups according to certain weights based on the activity keywords, emotion keywords and background keywords, and to match the adapted audio according to the keyword groups;

[0060] The user recommendation unit is used to obtain the adapted audio corresponding to the keyword group from similar users of the user when the replay rate of the adapted audio is greater than a certain expected threshold;

[0061] The music playing module includes a playing unit and a feedback unit;

[0062] The playback unit is used to play the adapted audio according to the audio playback order, adjust the adapted audio frequency according to the live audio frequency, and adjust the audio playback by real-time monitoring of the character's movements;

[0063] The feedback unit is used to obtain the playing satisfaction of the adapted audio through the emotional fluctuation of the character after the adapted audio is played, and adjust the weight of the corresponding keyword in the keyword group corresponding to the adapted audio according to the playing satisfaction of the adapted audio.

[0064] Beneficial effects of the present invention: The present invention acquires scene information by real-time acquisition of on-site panoramic video and on-site audio, and recommends adaptive music according to the scene, thereby improving the matching degree between the scene and the music, saving the user's time in selecting music, and improving the user's experience;

[0065] By analyzing the emotions of the target person in the panoramic video of the scene after the adapted music is played, the playing satisfaction of the adapted music is obtained, and the keyword weights of the keyword groups of the adapted music are adjusted according to the playing satisfaction;

[0066] When the replay rate of the adapted music is greater than the expected threshold, similar users of the user are searched, and the adapted audio is obtained from the historical playback audio of similar users based on the keyword group, which solves the problem that the played audio tends to be single; through real-time monitoring of the human movements in the live panoramic video, the audio playback is adjusted according to the human body movements, which improves the convenience and interactivity of music playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A basic flow chart of a music collaborative playing method based on virtual reality provided by one embodiment of the present invention;

[0068] Figure 2 A basic flow chart of a music collaborative playback system based on virtual reality is provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0070] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a music collaborative playing method based on virtual reality, comprising:

[0071] Step S1, real-time acquisition of on-site panoramic video and on-site audio;

[0072] Step S2, capturing the activities of people in the panoramic video of the scene, identifying the background of the video and obtaining the audio frequency of the scene, using the machine learning model to obtain activity keywords, background keywords and emotional keywords, and constructing a keyword group with the activity keywords, background keywords and activity keywords;

[0073] Step S3, obtaining the adapted audio from the database according to the keyword group, judging whether to change the adapted audio according to the replay rate of the adapted audio, re-obtaining the adapted audio by analyzing similar users using the collaborative filtering recommendation algorithm, and adjusting the adapted audio frequency according to the on-site audio volume;

[0074] Step S4, playing the adapted audio, timely capturing the emotions of the characters in the on-site panoramic video, obtaining the playback satisfaction according to the emotional fluctuations of the characters, and adjusting the keyword weights in the audio keyword group according to the playback satisfaction;

[0075] Step S5, real-time monitoring of human movements in the on-site panoramic video, and automatically adjusting the audio playback when a specific movement is recognized.

[0076] The collaborative filtering recommendation algorithm is based on the user's rating or behavior data on the product, and uses the similarity between users or the similarity between items to make recommendations;

[0077] In this embodiment, scene information is obtained by real-time acquisition of on-site panoramic video and on-site audio, and adaptive music is recommended according to the scene, thereby improving the matching degree between the scene and the music and saving the user's time in selecting music;

[0078] By analyzing the emotions of the target person in the panoramic video of the scene after the adapted music is played, the playing satisfaction of the adapted music is obtained, and the keyword weights of the keyword groups of the adapted music are adjusted according to the playing satisfaction;

[0079] When the replay rate of the adapted music is greater than the expected threshold, similar users of the user are searched, and the adapted audio is obtained from the historical playback audio of similar users based on the keyword group, which solves the problem that the played audio tends to be single; through real-time monitoring of the human movements in the live panoramic video, the audio playback is adjusted according to the human body movements, which improves the convenience of music playback.

[0080] The step S2 specifically includes the following steps:

[0081] Step S201, using portrait segmentation technology to separate the person and the background of the on-site panoramic video, and obtaining the person frequency band and the background frequency band;

[0082] Step S202, using a target detection algorithm to capture each person target in the person frequency band, identifying the target person's action through action recognition technology, and obtaining the activity keyword corresponding to the action;

[0083] Step S203, obtaining an image matrix of the face area of ​​the target person through a classifier as an input image for expression recognition, and obtaining emotional keywords of the target person;

[0084] Step S204, using the background recognition model to obtain background keywords of the background frequency band;

[0085] Step S205, constructing a keyword group by matching the activity keywords, emotion keywords and background keywords according to a certain weight ratio;

[0086] The portrait segmentation technology is essentially a refined image cutout technology, which aims to accurately separate the portrait from the background in the image;

[0087] The portrait segmentation comprises the following steps:

[0088] Image preprocessing: Obtain key frame images of the on-site panoramic video, and perform image denoising and image enhancement on the key frame images to eliminate image noise and enhance image features;

[0089] Feature extraction: Use deep learning and other algorithms to automatically extract the characteristic information of people in key frame images, such as edges, textures, etc.

[0090] Portrait segmentation: Based on the extracted feature information, separate the person from the background in the key frame image and generate a transparency mask;

[0091] Post-segmentation processing: further optimize the segmentation results, such as removing noise, filling holes, etc., to improve the accuracy and effect of segmentation;

[0092] The object detection algorithm is used to find all objects of interest in an image and determine their categories and locations. It is one of the core problems in the field of computer vision.

[0093] The image matrix represents an image abstracted into a two-dimensional or three-dimensional matrix, where the number of rows and columns of the matrix corresponds to the height and width of the image;

[0094] In this embodiment, the keyword group is constructed by respectively acquiring the activity keywords, the emotion keywords and the background keywords, which provides an accurate matching basis for the subsequent matching and adaptation of the music.

[0095] The step S3 specifically comprises the following steps:

[0096] Step S301, matching audio according to the keyword group to obtain adapted audio;

[0097] Step S302, obtaining the local replay rate of the adapted audio, if the replay rate is greater than the expected threshold, obtaining similar users of the user according to the static attributes and dynamic attributes of the user portrait, wherein the static attributes include gender, age, geographic location and favorite singer, and the dynamic attributes include user historical operation records;

[0098] Step S303, reacquiring adapted audio from historical playback records of similar users according to the keyword group;

[0099] The user's historical operation records include audio playing, audio skipping, audio switching, and audio marking as favorite;

[0100] In this embodiment, whether to re-match the adapted audio is determined based on the replay rate of the adapted music. If the adapted audio is re-matched, the adapted audio is obtained based on the historical playback records of similar users of the user, thereby avoiding the problem of audio playback tending to be single.

[0101] The step S4 specifically includes:

[0102] Step S401, playing adapted audio;

[0103] Step S402, obtaining the live sound frequency through the live audio, and adjusting the adapted audio playback frequency according to the live sound frequency;

[0104] Step S403, capturing the on-site panoramic video within a unit time after the adapted audio is played;

[0105] Step S404, obtaining key frames of the on-site panoramic video;

[0106] Step S405, using a face recognition model to obtain an image matrix of the face area of ​​the target person in the key frame;

[0107] Step S406, inputting the image matrix into the expression recognition model, obtaining the emotional keywords of the target person in each key frame, obtaining the emotional fluctuation of the target person by comparing the emotional keywords, and obtaining the playback satisfaction of the adapted audio according to the emotional fluctuation of the target person;

[0108] Step S407, adjusting the keyword weights in the keyword group of the adapted audio according to the value of the playback satisfaction, if the value of the playback satisfaction is greater than or equal to a certain expected threshold, then increasing the keyword weights of the keyword group of the adapted audio according to the value of the playback satisfaction, if the value of the playback satisfaction is less than or equal to the expected threshold, then decreasing the keyword weights of the keyword group of the adapted audio according to the value of the playback satisfaction;

[0109] The face recognition model is used to identify whether the target area is a face area;

[0110] The key frame represents the frame where the key action of the person or object in motion or change is located;

[0111] The key frame acquisition process includes taking the first frame of the live panoramic video as the key frame and the reference frame of the second frame, calculating the difference between the current reference frame and the remaining frames, and if the difference of a certain frame is greater than a certain threshold, then this frame is defined as a new key frame;

[0112] The emotional keywords include emotions and emotional saturation, the emotions include happiness, sadness, excitement, doubt, disappointment, disgust and calmness, and the emotional saturation is expressed in percentage to indicate the degree of emotional state;

[0113] The emotional fluctuation includes emotional transition and emotional saturation transition. The emotional transition is used to indicate the target person's transition between various emotions, and the emotional saturation transition is used to indicate the transition of the emotional degree under the same emotion.

[0114] The method of obtaining the playing satisfaction of the adapted audio according to the emotional fluctuation of the target person includes:

[0115] Corresponding emotion values ​​are set for different emotions, and the total emotion value is obtained in combination with the emotion saturation. The difference in the total emotion value after the adapted audio is played is obtained according to the emotion fluctuation, and the playback satisfaction of the adapted audio is obtained according to the difference.

[0116] In this embodiment, the playback satisfaction of the adapted audio is obtained by real-time monitoring of the emotional changes of the target person in the on-site panoramic video, and the keyword weight of the adapted audio is adjusted according to the playback satisfaction, thereby improving the accuracy of music matching.

[0117] The step S5 specifically comprises the following steps:

[0118] Step S501, real-time monitoring of human movements in the on-site panoramic video;

[0119] Step S502, comparing the character action with the set adjustment play action to obtain the action similarity, and if the action similarity is greater than or equal to the expected threshold, adjusting the adapted audio play according to the play adjustment instruction corresponding to the adjustment play action;

[0120] The playback adjustment instructions include song switching instructions, frequency adjustment instructions and speed adjustment instructions;

[0121] The playback adjustment instructions include song switching, pause, increasing audio frequency, and decreasing audio frequency;

[0122] In this embodiment, by real-time monitoring of the character movements in the panoramic video, playback adjustment instructions are obtained to adjust the adaptive audio playback, thereby achieving contactless adaptive audio playback adjustment and improving the interactivity and convenience of audio playback.

[0123] The step S202 specifically includes the following steps:

[0124] Step S2021, after preprocessing and labeling the human motion data set, input it into the machine learning model for training to obtain a motion recognition model;

[0125] Step S2022, using image recognition technology to identify key points of the character band, and obtaining action group nodes corresponding to the identified action key points;

[0126] Step S2023, inputting the action group frame node into the action recognition model to obtain the corresponding activity keyword;

[0127] The dataset annotation is the process of processing the collected, unprocessed raw data or primary data, including voice, pictures, text, video and other types of data, and converting them into machine-recognizable information;

[0128] In this embodiment, activity keywords are obtained based on character actions, providing accurate and specific activity keywords for constructing keyword groups that match the adapted audio, thereby improving the success rate of audio matching.

[0129] The step S203 specifically includes the following steps:

[0130] Step S2031, obtaining a face data set and a non-face data set, inputting the face data set and the non-face data set into a machine learning model after data processing, and obtaining a face recognition model after training;

[0131] Step S2032, inputting the character frequency band into the face recognition model to obtain an image matrix of the face area of ​​the target character;

[0132] Step S2033, obtaining a face area dataset, inputting the labeled face area dataset into a machine learning model for training, and obtaining an expression recognition model;

[0133] Step S2034, inputting the image matrix of the face area of ​​the target person into the expression recognition model to obtain the emotional keywords of the target person.

[0134] In this embodiment, by establishing an expression recognition model and an audio feature extraction model, emotional keywords are obtained to form keyword groups, and audio is matched according to the keyword groups, thereby improving the accuracy of audio matching.

[0135] The step S301 includes:

[0136] Step S3011, obtaining an audio data packet, performing data preprocessing and keyword tagging on the audio data packet, inputting the keyword-tagged audio data packet into a machine learning model for training, obtaining an audio feature extraction model, and performing keyword extraction on the audio in the database;

[0137] Step S3012, matching the keyword group with the database audio;

[0138] Step S3013: if the matching degree between the keyword group and the keyword of the database audio is greater than the expected threshold, the matching audios are organized into an audio playlist according to the matching degree from high to low, and the adapted audios are played in the order of the audio playlist;

[0139] In this embodiment, by establishing an expression recognition model and an audio feature extraction model, emotional keywords are obtained to form keyword groups, and audio is matched according to the keyword groups, thereby improving the accuracy of audio matching.

[0140] Example 2, reference Figure 2 Another embodiment of the present invention is different from the first embodiment in that it provides a music collaborative playback system based on virtual reality, including a collection module, a music recommendation module and a music playback module;

[0141] The acquisition module is used to acquire the on-site panoramic video and on-site audio in real time;

[0142] The music recommendation module is used to obtain a scene keyword group according to the scene panoramic video, and obtain an adaptation audio according to the keyword group;

[0143] The music playing module is used to play the adapted audio and adjust the audio frequency according to the live audio frequency and adjust the audio playing according to the character's actions;

[0144] In this embodiment, background keywords, activity keywords and emotional keywords are obtained by real-time acquisition of the on-site panoramic video to form a keyword group, and the audio is matched according to the keyword group, which improves the accuracy of audio matching and enhances the interactivity and convenience of scene-based audio playback;

[0145] The replay rate of the adapted audio determines whether to re-match, searches for similar users of the user, and re-acquires the adapted music through the historical playback audio of similar users, thus avoiding the problem of the adapted audio tending to be single and improving the richness of the adapted audio;

[0146] By real-time monitoring of human body movements and receiving audio playback adjustment instructions, contactless audio adjustment is achieved, enhancing the convenience of audio playback.

[0147] The music recommendation module includes a site recommendation unit and a similar user recommendation unit;

[0148] The on-site recommendation unit is used to analyze the on-site panoramic video to obtain activity keywords, emotion keywords and background keywords, obtain keyword groups according to the activity keywords, emotion keywords and background keywords, and match the adapted audio according to the keyword groups;

[0149] The user recommendation unit is used to obtain the adapted audio corresponding to the keyword group from similar users of the user when the replay rate of the adapted audio is greater than a certain expected threshold;

[0150] The music playing module includes a playing unit and a feedback unit;

[0151] The playback unit is used to play the adapted audio according to the audio playback order, adjust the adapted audio frequency according to the live audio frequency, and adjust the audio playback by real-time monitoring of the character's movements;

[0152] The feedback unit is used to obtain the playing satisfaction of the adapted audio through the emotional fluctuation of the character after the adapted audio is played, and adjust the weight of the corresponding keyword in the keyword group corresponding to the adapted audio according to the playing satisfaction of the adapted audio.

[0153] In this embodiment, the audio is adapted according to the keyword group matching, and whether to re-match is determined according to the replay rate of the adapted audio. Similar users of the user are searched, and the adapted audio is re-matched according to the historical playback audio of the similar users, thereby improving the accuracy and richness of the audio matching.

[0154] In this embodiment, the playback satisfaction of the adapted audio is obtained by monitoring the emotional fluctuations of the character after the audio is played, and the corresponding keyword weights in the adapted audio keyword group are adjusted according to the playback satisfaction, thereby improving the success rate and accuracy of audio matching. By monitoring the character's movements in real time and obtaining audio adjustment instructions, the convenience and interactivity of audio playback are improved.

[0155] In this embodiment, scene information is obtained by real-time acquisition of on-site panoramic video and on-site audio, and adaptive music is recommended according to the scene, thereby improving the matching degree between the scene and the music and saving the user's time in selecting music;

[0156] By analyzing the emotions of the target person in the panoramic video of the scene after the adapted music is played, the playing satisfaction of the adapted music is obtained, and the keyword weights of the keyword groups of the adapted music are adjusted according to the playing satisfaction;

[0157] When the replay rate of the adapted music is greater than the expected threshold, similar users of the user are searched, and the adapted audio is obtained from the historical playback audio of similar users based on the keyword group, which solves the problem that the played audio tends to be single; through real-time monitoring of the human movements in the live panoramic video, the audio playback is adjusted according to the human body movements, which improves the convenience and interactivity of music playback.

[0158] It should be understood by those skilled in the art that the embodiments of the present invention can be provided as methods, systems or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A music collaborative playing method based on virtual reality, characterized in that: The following steps are involved: Step S1, real-time acquisition of on-site panoramic video and on-site audio; Step S2, capturing the activities of people in the panoramic video of the scene, identifying the background of the video and obtaining the audio frequency of the scene, using the machine learning model to obtain activity keywords, background keywords and emotional keywords, and constructing a keyword group according to the activity keywords, background keywords and activity keywords according to a certain weight; Step S3, obtaining the adapted audio from the database according to the keyword group, judging whether to change the adapted audio according to the replay rate of the adapted audio, and re-obtaining the adapted audio by analyzing similar users using a collaborative filtering recommendation algorithm; Step S4, playing the adapted audio, and adjusting the adapted audio frequency according to the on-site audio volume, timely capturing the emotions of the characters in the on-site panoramic video, obtaining the playback satisfaction according to the emotional fluctuations of the characters, and adjusting the keyword weights in the audio keyword group according to the playback satisfaction; Step S5, real-time monitoring of human movements in the on-site panoramic video, and automatically adjusting the audio playback when a specific movement is recognized.

2. A music collaborative playing method based on virtual reality as claimed in claim 1, characterized in that: The step S2 specifically includes the following steps: Step S201, using portrait segmentation technology to separate the person and the background of the on-site panoramic video, and obtaining the person frequency band and the background frequency band; Step S202, using a target detection algorithm to capture each person target in the person frequency band, identifying the target person's action through action recognition technology, and obtaining the activity keyword corresponding to the action; Step S203, obtaining an image matrix of the face area of ​​the target person through a face recognition model as an input image for expression recognition, and obtaining emotional keywords of the target person; Step S205, using the background recognition model to obtain background keywords of the background frequency band; Step S206, constructing a keyword group by matching the activity keywords, emotion keywords and background keywords according to a certain weight ratio.

3. A virtual reality-based music collaborative playing method as claimed in claim 1, characterized in that: The step S3 specifically comprises the following steps: Step S301, matching audio according to the keyword group to obtain adapted audio; Step S302, obtaining the local replay rate of the adapted audio, if the replay rate is greater than the expected threshold, obtaining similar users of the user according to the static attributes and dynamic attributes of the user portrait, wherein the static attributes include gender, age, geographic location and favorite singer, and the dynamic attributes include user historical operation records; Step S303: reacquire the adapted audio according to the keyword group from the historical playback records of similar users.

4. A music collaborative playing method based on virtual reality as claimed in claim 1, characterized in that: The step S4 specifically includes: Step S401, playing adapted audio; Step S402, obtaining the live sound frequency through the live audio, and adjusting the adapted audio playback frequency according to the live sound frequency; Step S403, capturing the on-site panoramic video within a unit time after the adapted audio is played; Step S404, obtaining key frames of the on-site panoramic video; Step S405, using a face recognition model to obtain an image matrix of the face area of ​​the target person in the key frame; Step S406, inputting the image matrix into the expression recognition model, obtaining the emotional keywords of the target person in each key frame, obtaining the emotional fluctuation of the target person by comparing the emotional keywords, and obtaining the playback satisfaction of the adapted audio according to the emotional fluctuation of the target person; Step S407, adjust the keyword weights in the keyword group of the adapted audio according to the value of the playback satisfaction. If the value of the playback satisfaction is greater than or equal to a certain expected threshold, increase the keyword weights of the keyword group of the adapted audio according to the value of the playback satisfaction. If the value of the playback satisfaction is less than or equal to the expected threshold, reduce the keyword weights of the keyword group of the adapted audio according to the value of the playback satisfaction.

5. A music collaborative playing method based on virtual reality as claimed in claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S501, real-time monitoring of human movements in the on-site panoramic video; Step S502, comparing the character action with the set playback adjustment action to obtain the action similarity, and if the action similarity is greater than or equal to the expected threshold, adjusting the adaptive audio playback according to the playback adjustment instruction corresponding to the playback adjustment action; The playback adjustment instructions include song switching instructions, frequency adjustment instructions and speed adjustment instructions.

6. A music collaborative playing method based on virtual reality as claimed in claim 2, characterized in that: The step S202 specifically includes the following steps: Step S2021, after preprocessing and labeling the human motion data set, input it into the machine learning model for training to obtain a motion recognition model; Step S2022, using image recognition technology to identify key points of the character band, and obtaining action group nodes corresponding to the identified action key points; Step S2023, input the action group frame node into the action recognition model to obtain the corresponding activity keyword.

7. A music collaborative playing method based on virtual reality as claimed in claim 2, characterized in that: The step S203 specifically includes the following steps: Step S2031, obtaining a face data set and a non-face data set, inputting the face data set and the non-face data set into a machine learning model after data processing, and obtaining a face recognition model after training; Step S2032, inputting the character frequency band into the face recognition model to obtain an image matrix of the face area of ​​the target character; Step S2033, obtaining a face area dataset, inputting the labeled face area dataset into a machine learning model for training, and obtaining an expression recognition model; Step S2034, inputting the image matrix of the face area of ​​the target person into the expression recognition model to obtain the emotional keywords of the target person.

8. A music collaborative playing method based on virtual reality as claimed in claim 3, characterized in that: The step S301 includes: Step S3011, obtaining an audio data packet, performing data preprocessing and keyword tagging on the audio data packet, inputting the keyword-tagged audio data packet into a machine learning model for training, obtaining an audio feature extraction model, and performing keyword extraction on the audio in the database; Step S3012, matching the keyword group with the database audio; Step S3013, if the matching degree between the keyword group and the keyword of the database audio is greater than the expected threshold, the matching audios are organized into an audio play list according to the matching degree from high to low, and the adapted audios are played in the order of the audio play list.

9. A music collaborative playing system based on virtual reality, characterized in that: Including collection module, music recommendation module and music playback module: The acquisition module is used to acquire the on-site panoramic video and on-site audio in real time; The music recommendation module is used to obtain a scene keyword group according to the scene panoramic video, and obtain an adaptation audio according to the keyword group; The music playing module is used to play the adapted audio and adjust the audio frequency according to the live audio frequency and adjust the audio playing according to the character's actions.

10. A music collaborative playing system based on virtual reality as claimed in claim 9, characterized in that: The music recommendation module includes a site recommendation unit and a similar user recommendation unit; The on-site recommendation unit is used to analyze the on-site panoramic video to obtain activity keywords, emotion keywords and background keywords, and to construct keyword groups according to certain weights based on the activity keywords, emotion keywords and background keywords, and to match the adapted audio according to the keyword groups; The user recommendation unit is used to obtain the adapted audio corresponding to the keyword group from similar users of the user when the replay rate of the adapted audio is greater than a certain expected threshold; The music playing module includes a playing unit and a feedback unit; The playback unit is used to play the adapted audio according to the audio playback order, adjust the adapted audio frequency according to the live audio frequency, and adjust the audio playback by real-time monitoring of the character's movements; The feedback unit is used to obtain the playing satisfaction of the adapted audio through the emotional fluctuation of the character after the adapted audio is played, and adjust the weight of the corresponding keyword in the keyword group corresponding to the adapted audio according to the playing satisfaction of the adapted audio.

Images