Scenic spot information recommendation method based on 3D digital interaction

Abstract

The invention belongs to the field of artificial intelligence, and relates to a tourist attraction information recommendation method based on 3D digital interaction, which comprises the following steps: acquiring tourist dynamic information, and determining tourist position information; initiating a session request according to the tourist position information; the system obtains a session area state and obtains random session data; extracting the action information and the broadcast text, and converting the broadcast text into audio data; obtaining phoneme information in the audio data, and constructing lip animation data; the action information is updated; feeding back the updated action information, voice information and lip animation data to the tourist; if the tourist does not respond to the feedback data, the random session data is repeatedly generated; if the tourist responds to the feedback data, real-time conversation is carried out, and recommendation information is generated; according to the invention, the face recognition technology is adopted to recognize the tourist of the user, interaction is realized through voice, and humanized, intelligent and efficient service experience is provided for the tourist in combination with the large language model and the voice synthesis technology.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence, and in particular relates to a tourist attraction information recommendation method based on 3D digital interaction. Background Art

[0002] The development of artificial intelligence technology has provided a more natural and convenient service method for the public's cultural and tourism travel. With the increase in the number of tourists, traditional service forms cannot meet service needs in terms of time, personalization, etc. Therefore, how to integrate artificial intelligence technology to better serve people has become a problem that needs to be thought about and solved. In cultural tourism and natural scenic spots, natural language interactive digital humans interact with people in natural language in a human-like way, understand user needs, answer questions, and present them in a visual form through the behavioral expression language and facial movements of digital virtual humans, providing people with a more natural interactive experience and more humane service.

[0003] Among the existing intelligent voice tour guide systems, the patent application number 201710649957.2, "An Intelligent Voice Tour Guide System and Its Tour Guide Method", discloses: The system mainly includes a voice terminal device, a wireless network, and a management server, and the voice terminal device is wirelessly connected to the management server via a wireless network. The system can push tour guide information to the voice terminal device in a timely manner through a wireless network based on real-time factors such as tourist roles (including needs), locations, and time, and can make intelligent judgments and analyses based on real-time data such as scenic spot capacity, personnel density, tourist journeys, and weather changes, and promptly prompt tourists with practical information such as the best tour routes and public service projects. It can also answer tourists' inquiries or solve problems that tourists need to solve through voice interaction with tourists. The above-mentioned prior art cannot actively communicate with users, but can only passively ask and answer questions with customers, which leads to fuzzy user query information and low efficiency. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention proposes a tourist attraction information recommendation method based on 3D digital interaction, which includes: building a tourist attraction information recommendation system; obtaining dynamic information of tourists, and determining the location information of tourists according to the dynamic information of tourists; initiating a session request according to the location information of tourists; according to the session request, the system obtains the session area status, and obtains random session data according to the current time, location and weather; extracts action information and broadcast text in the session data, and converts the broadcast text into audio data; obtains phoneme information in the audio data, generates facial micro-expressions according to the phoneme information, and constructs lip animation data; updates the action information according to the animation data; feeds back the voice information, lip animation data and updated action information to the tourists; if the tourists do not respond to the feedback data, repeatedly generate random session data, extract corresponding data and feed it back to the tourists, if there is no response for multiple rounds, end the session; if the tourists respond to the feedback data, the tourists have a real-time dialogue with the system and generate recommendation information.

[0005] Beneficial effects of the present invention:

[0006] The present invention adopts face recognition technology to identify user tourists, actively conducts conversations with tourists, realizes interaction through voice, and combines large language models and speech synthesis technology to provide tourists with a friendly, humane, intelligent and efficient service experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is the overall flow chart of the present invention;

[0008] Figure 2 A flowchart for converting text to speech and driving the digital human's movements and lip shapes of the present invention;

[0009] Figure 3 are the phonemes and corresponding weight factors of the present invention;

[0010] Figure 4 It is a flowchart of converting phoneme information into lip shape animation of the present invention;

[0011] Figure 5 It is a micro-expression simulation flow chart of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] A tourist attraction information recommendation method based on 3D digital interaction, such as Figure 1 to Figure 5 The method includes: building a tourist attraction information recommendation system; obtaining dynamic information of tourists, and determining the location information of tourists according to the dynamic information of tourists; initiating a session request according to the location information of tourists; according to the session request, the system obtains the session area status, and obtains random session data according to the current time, location and weather; extracting action information and broadcast text in the session data, and converting the broadcast text into audio data; obtaining phoneme information in the audio data, generating facial micro-expressions according to the phoneme information, and constructing lip animation data; updating the action information according to the animation data; feeding back the voice information, lip animation data and updated action information to the tourists; if the tourists do not respond to the feedback data, repeatedly generating random session data, extracting corresponding data and feeding back to the tourists, and terminating the session if there is no response for multiple rounds; if the tourists respond to the feedback data, the tourists have a real-time dialogue with the system and generate recommendation information.

[0014] An implementation method of a tourist attraction information recommendation method based on 3D digital interaction, such as Figure 1 and Figure 2 As shown, the method includes: obtaining the state of the conversation area; obtaining random conversation data through the current time, location and weather; obtaining action information and broadcast text in the conversation data, and generating audio from the broadcast text; obtaining phoneme information in the audio; constructing lip shape animation data; simulating facial micro-expressions; playing and updating actions and lip shapes; humanized user interaction response; obtaining the content of tourist conversations or analyzing tourist needs through interactive operations to complete the tour guide service.

[0015] Acquiring the status of the conversation area includes: acquiring surrounding images through the camera, performing face detection on the images, obtaining the coordinates of key points of the face after detecting the face, and measuring the distance between the face and the camera. If the distance is less than the set distance, it means that the visitor is in the conversation area, and the conversation is initiated actively. After entering the conversation state, if the tracking face distance is greater than the set distance, it means that the visitor has left the conversation area, and the conversation is terminated at this time.

[0016] Measuring the distance between a face and a camera includes: setting a human eye distance constant M, acquiring an image at a distance d from the camera with the human eye distance being m, calculating the camera focal length constant F = (m*d) / M, and using the constants M and F, through d = (M*F) / m, the distance between the face and the camera can be estimated.

[0017] like Figure 1 As shown, when a visitor enters the conversation area, the system will trigger the conversation state. At this time, the system will randomly obtain a conversation content based on time, location, weather and other conditions. The conversation content includes 3D digital human actions and broadcast text.

[0018] Obtaining action information and broadcast text in session data includes: To improve session efficiency, the text is optimally segmented to minimize the broadcast waiting time, ensure uninterrupted context broadcast, and minimize the overhead of the text-to-speech service. Set the maximum length of the broadcast text, split the short sentences from the session data by punctuation marks, regularly obtain the action information and broadcast text in the short sentences, and splice the broadcast text. The length not exceeding the maximum length is used as the content for one broadcast.

[0019] Obtaining phoneme information in the audio includes: Obtaining the start time and end time of the pronunciation of the text in the audio through speech recognition technology, and obtaining the corresponding pinyin or phonetic symbols of the text or English words as phoneme information. Set the pronunciation weights of the open vowels and closed vowels in the phonemes, and simplify the initial consonants and vowels with the same pronunciation. For example, the phoneme of "上" is shang, and there are only two BS animations during actual pronunciation. Finally, the BS curves corresponding to the obtained phoneme data are "sh" and "ang" instead of pronouncing "s", "h", "a", "n", "g" one by one. Finally, set the pronunciation time and maximum value of each phoneme according to the total pronunciation duration and phoneme weight.

[0020] Constructing lip animation data includes, as Figure 4 shown: Calculating the phoneme pronunciation weight; Calculating the pronunciation duration according to the pronunciation weight; Calculating the frame data according to the pronunciation duration and frame rate; Generating animation data.

[0021] In this embodiment, the total weight of the current phoneme is calculated as:

[0022] W sum =w s *W s +w j *W j +w y *W y

[0023] Among them, W sum represents the total phoneme weight, w s represents the weight factor of the specific phoneme of the initial consonant, W s represents the pronunciation weight of the initial consonant. Similarly, w j and W j represent the phoneme weight factor and pronunciation weight of the medial respectively, and w y and W y represent the phoneme weight factor and pronunciation weight of the final respectively.

[0024] Calculate the actual pronunciation duration of each phoneme corresponding to the current text:

[0025]

[0026] Among them, T is the pronunciation duration of a single phoneme, T eis the end time of the current pronunciation, T s is the pronunciation start time, w is the phoneme weight factor, W is the pronunciation weight, W sum The total weight of the pronunciation.

[0027] In this embodiment, calculating the frame data according to the pronunciation duration and the frame rate includes: calculating the frame data of the pronunciation curve, and the expression thereof is:

[0028] v t =f(T a -T s )=f(d t )

[0029] Among them, v t is the curve value of the current frame, T a is the duration of audio playback, T s is the pronunciation start time, f(d t ) is the formula for calculating the current frame curve value:

[0030]

[0031] Among them, v t′ is the value of the previous frame of the curve, d t is the difference between the audio time and the pronunciation start time, V max is the maximum value of the phoneme corresponding curve, T is the phoneme pronunciation duration, S open S is the pronunciation speed of the phoneme, and W is the pronunciation weight of special phonemes, such as closed mouth sounds. close The speed at which the phoneme is restored to its starting value after pronunciation is completed.

[0032] The obtained value v t Restricted within a reasonable range, the restriction formula is:

[0033] v t =Clamp(v t ,0,V max )

[0034] In this embodiment, generating facial micro-expressions according to phoneme information includes: simulating eyeball movements, blinking movements, eyebrow movements, nose movements, and lip movements according to factor information; and fusing basic shapes according to the simulated movements.

[0035] 1. Simulating eye movements includes: obtaining random factors for eye movements up and down and left and right:

[0036] R ud = Random(-1, 1)*W ud

[0037] R lr = Random(-1, 1)*Wlr

[0038] Where R ud Indicates the random factor moving up and down, R lr The random factor for left and right movement, W ud Controls the maximum range of up and down movement, W lr Control the maximum range of left and right movement; then use the formula to obtain the linear change value of the eyeball in the next frame:

[0039] v ud =Lerp(V ud′ , R ud , d t )

[0040] v lr =Lerp(V lr′ , R lr , d t )

[0041] Where V ud V is the upper and lower weight value of the eyeball in the next frame. lr is the weight value for the next frame, V ud′ is the upper and lower weight value of the current frame, V lr′ The left and right weight values ​​of the current frame, d t is the frame time. Then through the formula:

[0042] w ud = sin(D ud +d t *S ud )*v ud

[0043] w lr = cos(D lr +d t *S lr )*v lr

[0044] Where w represents the final BS weight, S is the speed weight, d t is the unit time, D is the speed S and the unit time d t When v is equal to R, reset R to 0 so that the eyeball finally returns to the starting position, then set a certain interval time and execute the same process.

[0045] 2. Simulating blinking action includes: obtaining random factors:

[0046] R = Random(0, 1)

[0047] Then through linear change:

[0048] w=Lerp(w′,R,dt*S)

[0049] Where w is the weight value of the blink curve in the next frame, w′ is the current weight value, R is the target value, and d t is the unit frame time, S is the random blink speed, when w is equal to R, set R to 0 and return to the starting position

[0050] 3. Simulating eyebrow movements and simulating nose movements include: obtaining random factors for left and right changes, obtaining the weight value of the next frame through linear changes according to the current frame weight, and when the weight value is equal to the random value, setting the random value to 0 and returning to the starting position.

[0051] 4. Simulating lip movements The actions include: obtaining the random factor of lip movements, obtaining the frame data of the lip shape animation, and using the formula:

[0052] w=W*w lips +(1-W)*w emo

[0053] Where w is the weight value of the same curve of lip movement and lip shape animation, W is the proportion of lip shape animation in the whole mouth animation, and w lips is the lip animation weight value, w emo The weight value for the lip action

[0054] 5. Fusion of basic shapes: Obtain the current facial basic shape data, and then obtain the shape data generated by facial micro-expressions, separate the same curves from the basic shape data and micro-expression shape data, and sum the weight values ​​of the same curves in proportion to the basic expression and micro-expression shape values. The formula is:

[0055] w=W*w base +(1-W)*w new

[0056] Where w is the weight value of the same curve of lip movement and lip shape animation, W is the proportion of the basic shape in the entire face, and w base is the base shape weight value, w new is the weight value of micro-expression action.

[0057] In this embodiment, the action and lip shape are played and updated as follows Figure 2 As shown, specifically including:

[0058] Play and update the motion and lip shape relative to the lip shape animation data constructed above.

[0059] An implementation method, relative to method 1 of constructing lip animation data: playing audio and lip animation simultaneously, obtaining audio playback progress, obtaining bound actions according to the playback progress, playing action animation and expression animation corresponding to the action.

[0060] Another implementation method is the method 2 of constructing lip animation data: first play the audio, record the audio playback progress AudioTime in real time, and then get the values ​​of all BS in the current frame according to the audio playback progress, and reset all the BS values ​​that are not in this interval. In this method, it is particularly important to process closed-mouth sounds. If there is a large change in the open sound in the previous frame or the closed-mouth sound pronunciation time is too short, the BS of the closed-mouth sound pronunciation will not be obvious, or the pronunciation time is less than the frame interval and is directly skipped.

[0061] When the animation of the digital human model's expression is triggered, method 1 needs to merge the expression animation and lip shape animation, but method 2 does not have this problem. When the action interval between two segments of text is short, the animation needs to be merged, or the problem can be avoided directly from the text end, otherwise the animation will twitch.

[0062] Humanized user interaction response includes: when the user initiates a conversation or interrupts the digital human conversation through voice, the language model responds to the user's question and answer needs; when the user does not respond to the digital human's response, the digital human should initiate a new round of conversation according to the tour guide's explanation script or continue the conversation topic before being interrupted and continue to explain the content. When the digital human receives a normal notification, the digital human will interrupt before initiating a new round of topics when no one answers the current topic. When the digital human receives an emergency notification, it will naturally stop the conversation, reveal the reason and broadcast the emergency notification.

[0063] In this embodiment, the digital human tour guide explanation script should include the following topics:

[0064] 1. Tour guide service: explain your identity and introduce yourself to welcome tourists; get closer to tourists and gain resonance by asking them for non-sensitive information; create an immersive atmosphere, guide tourists' emotional rhythm, and enter the slow pace of life in the scenic area; suggest time arrangements for sightseeing and risks of scenic spots, and travel precautions; organize and attract tourists to participate in interactions and soothe their emotions; express retention, convey memories, and express expectations for the next reunion when leaving.

[0065] 2. Information broadcast: real-time broadcast of weather, traffic conditions, passenger flow, ticket purchase conditions, etc., push of the latest activities and notifications, recommended tour routes, time arrangements, relevant attractions, introductions to scenic spots, and recommendations for related activity information; reminders of public information such as catering and activities.

[0066] 3. Specialized knowledge of the cultural history of the scenic area.

[0067] 4. Specialized knowledge of the scenic area’s natural ecology.

[0068] 5. Specialized knowledge on ethnic culture of scenic spots.

[0069] In this embodiment, obtaining the content of tourist conversations or analyzing tourist needs through interactive operations includes: introducing attractions or guiding routes to tourists through tourist interactive operations; obtaining tourist audio through a microphone, calling a voice recognition service to recognize the audio content, using the audio content to call a large language model, analyzing user needs, and providing guided tour services to tourists.

[0070] In this embodiment, other facial key points can be used as the basis for estimating the distance between faces, and the facial features of people of different ages can be distinguished. Distance verification using multiple points can improve the measurement accuracy. When segmenting the text, it can also be segmented according to the maximum length of the broadcast text, and then the sentence break. Lip shape and action are not limited to being created as animation files, and the BS value of the current frame can also be calculated in real time in the refresh method, and the method of updating the BS of the model can be called for processing. The device can be used to perform facial motion capture for each pronunciation phoneme to make the lip shape more accurate.

[0071] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for recommending tourist attraction information based on 3D digital interaction, characterized in that: include: Build a tourist attraction information recommendation system; Obtain tourist dynamic information, and determine the tourist's location information based on the tourist dynamic information; Initiate a session request based on the visitor's location information; According to the session request, the system obtains the session area status and obtains random session data according to the current time, location and weather; extracts the action information and broadcast text in the session data, and converts the broadcast text into audio data; obtains the phoneme information in the audio data, generates facial micro-expressions based on the phoneme information, and constructs lip animation data; updates the action information according to the animation data; feeds back the voice information, lip animation data and updated action information to the tourist; if the tourist does not respond to the feedback data, repeatedly generates random session data, extracts the corresponding data and feeds it back to the tourist, and ends the session if there is no response for multiple rounds; if the tourist responds to the feedback data, the tourist has a real-time conversation with the system and generates recommendation information.

2. A method for recommending tourist attraction information based on 3D digital interaction according to claim 1, characterized in that: The acquired session area status includes: acquiring surrounding images through the camera, performing face detection and tracking on the images; acquiring the coordinates of key points of the face, and measuring the distance between the face and the camera.

3. The method for recommending tourist attraction information based on 3D digital interaction according to claim 1, characterized in that: Extracting action information and broadcast text from conversation data includes: setting a maximum length of the broadcast text, segmenting short sentences according to punctuation symbols in the conversation data, obtaining action information and broadcast text from the short sentences, splicing the broadcast text, and using text less than or equal to the maximum length as a broadcast content.

4. The method for recommending tourist attraction information based on 3D digital interaction according to claim 1, characterized in that: Acquiring the phoneme information in the audio data includes: acquiring the start time and the end time of the pronunciation of the text in the audio through voice recognition, and acquiring the pinyin or phonetic symbol corresponding to the text or English word as the phoneme information.

5. The method for recommending tourist attraction information based on 3D digital interaction according to claim 1, characterized in that: Constructing lip animation data includes: calculating phoneme pronunciation weights; calculating pronunciation duration according to the pronunciation weights; calculating frame data according to the pronunciation duration and frame rate; and generating animation data according to the frame data.

6. The method for recommending tourist attraction information based on 3D digital interaction according to claim 5, characterized in that: The formula for calculating frame data based on pronunciation duration and frame rate is: v t =f(T a -T s )=f(d t ) in t =Clamp(in t ,0,V max ) Among them, v t is the curve value of the current frame, T a is the duration of audio playback, T s is the pronunciation start time, f(d t ) is the formula for calculating the current frame curve value, v t′ is the value of the previous frame of the curve, d t is the difference between the audio time and the pronunciation start time, V max is the maximum value of the phoneme corresponding curve, T is the phoneme pronunciation duration, S open is the phoneme pronunciation speed, W is the special phoneme pronunciation weight, S close The speed at which the phoneme is restored to its starting value after pronunciation is completed.

7. The method for recommending tourist attraction information based on 3D digital interaction according to claim 1, characterized in that: Generating facial micro-expressions based on phoneme information includes: simulating eyeball movements, blinking movements, eyebrow movements, nose movements, and lip movements based on factor information; and fusing basic shapes based on simulated movements.

8. The method for recommending tourist attraction information based on 3D digital interaction according to claim 7, characterized in that: Simulating eye movements includes: obtaining random factors for eye movements up and down and left and right: R ud =Random(-1,1)*W ud R lr =Random(-1,1)*W lr Where R ud Indicates the random factor moving up and down, R lr The random factor for left and right movement, W ud Controls the maximum range of up and down movement, W lr Control the maximum range of left and right movement; The linear change value of the next frame of the eyeball is calculated based on the random factor, and its expression is: v ud =Lerp(V ud′ ,R ud ,d t ) v lr =Lerp(V lr′ ,R lr ,d t ) Among them, V ud V is the upper and lower weight value of the eyeball in the next frame. lr is the weight value for the next frame, V ud′ is the upper and lower weight value of the current frame, V lr′ The left and right weight values ​​of the current frame, d t is the frame time; The eye movement is simulated according to the linear change value of the next frame of the eye, and its expression is: w ud =sin(D ud +d t *S ud )*v ud w lr =cos(D lr +d t *S lr )*v lr Among them, w represents the final BS weight, S is the speed weight, and d t is the unit time, D is the speed S and the unit time d t Total amount produced; When v is equal to R, reset R to 0 so that the eyeball finally returns to the starting position, set the interval time, and execute the same process as above.

9. The method for recommending tourist attraction information based on 3D digital interaction according to claim 7, characterized in that: The lip motion simulation includes: obtaining a random factor of the lip motion, calculating a weight value of the same curve of the lip motion and the lip shape animation according to the random factor of the lip motion; generating frame data of the lip shape animation according to the weight value; the formula for calculating the weight value is: w=W*w lips +(1-W)*in emo Among them, w is the weight value of the same curve of lip action and lip shape animation, W is the proportion of lip shape animation in the whole mouth animation, and w lips is the lip animation weight value, w emo The weight value of the lip movement.

10. The method for recommending tourist attraction information based on 3D digital interaction according to claim 1, characterized in that: Updating the action information according to the animation data includes: playing audio and animation at the same time, obtaining the audio playing progress, obtaining the bound action according to the playing progress, and playing the action animation and the expression animation corresponding to the action.

Images