Automatic regulation and control method of box equipment based on virtual digital human, medium and equipment

Abstract

The invention relates to an automatic regulation and control method for box equipment based on virtual digital humans, which comprises the following steps of: acquiring consumption data of a user and song requesting data in a box, acquiring behavior data of the user in the box through image acquisition equipment, acquiring sound data of the user through audio acquisition equipment, and sending the sound data to the user through the audio acquisition equipment; generating multi-modal data of the user according to the consumption data, the song requesting data, the behavior data and the sound data; inputting the multi-modal data of the user into the trained neural network model to obtain a scene mode corresponding to the current box, and determining an interaction strategy of the virtual digital human according to the scene mode corresponding to the current box; and carrying out automatic regulation and control on box equipment based on an interaction strategy of the virtual digital human. According to the scheme, the box scene mode can be determined according to the multi-modal data, then the interaction strategy is determined based on the box scene mode, and the box equipment is automatically regulated and controlled through the virtual digital human, so that the user operation can be effectively simplified, and the interaction experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of box interaction, and in particular to an automatic control method for box equipment based on virtual digital humans, a computer-readable storage medium, and an electronic device. Background Art

[0002] As the private room entertainment industry rapidly develops, consumers' expectations for in-room entertainment experiences continue to rise, placing higher demands on intelligent and convenient entertainment environments. However, existing private room equipment still relies on manual user control, requiring explicit user commands to adjust operating parameters such as lighting and sound effects. This inconvenient operation affects the user's interactive experience within the private room. Summary of the Invention

[0003] To this end, it is necessary to provide an automatic control method for box equipment based on virtual digital people to solve the problems in the existing technology that the control of box equipment requires the user to issue clear instructions to complete, which is inconvenient to operate and affects the user's interactive experience.

[0004] To achieve the above objectives, in a first aspect, the present application provides an automatic control method for box equipment based on a virtual digital human, wherein the box equipment includes lighting equipment, audio playback equipment or display equipment, and an image acquisition device and an audio acquisition device are also provided in the box;

[0005] The method comprises the following steps:

[0006] S1: Set the image and style of the virtual digital person based on the current merchant portrait, box portrait, user portrait, and the reserved box theme information;

[0007] S2: Obtain the user's consumption data and song request data in the box, and collect the user's behavior data in the box through an image acquisition device, and collect the user's voice data through an audio acquisition device, and generate the user's multimodal data based on the consumption data, song request data, behavior data and voice data;

[0008] S3: Input the user's multimodal data into the trained neural network model to obtain the scene mode corresponding to the current box, and determine the interaction strategy of the virtual digital human based on the scene mode corresponding to the current box;

[0009] S4: Automatically adjust and control the box equipment based on the interactive strategy of the virtual digital human.

[0010] Furthermore, step S3 includes:

[0011] Based on whether there is a birthday cake in the image captured in real time by the image acquisition device and whether the user in the box is blowing out candles, the virtual digital human sets the scene mode corresponding to the current box to the birthday scene mode;

[0012] Determining the interaction strategy of the virtual digital human based on the scene mode corresponding to the current box includes: executing any one or more of the following interaction strategies through the virtual digital human: controlling the audio playback device to play birthday blessing-related songs, controlling the display device to display the lyrics of birthday blessing-related songs, and controlling the lighting equipment to present lighting effects that match the current birthday scene mode.

[0013] Furthermore, step S3 includes:

[0014] Determining whether the user has made a photo-taking action based on the image captured in real time by the image acquisition device and / or determining whether the user has issued a photo-taking request based on the sound data collected by the audio acquisition device, if so, the virtual digital human sets the scene mode corresponding to the current box to the photo-taking scene mode;

[0015] Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes:

[0016] The virtual digital human issues a photo-taking guidance prompt, and the background light at the user's location is adjusted by adjusting the brightness of the lighting device, wherein the photo-taking guidance information includes position guidance information, facial expression guidance information, and body movement guidance information;

[0017] The standing position guidance information and / or facial expression guidance information and / or body movement guidance information includes user address information corresponding to the need to adjust the standing position, facial expression, and body movement when taking pictures. The user address information is obtained from a portrait-address mapping library based on the user portrait information. The portrait-address mapping library is generated in the following manner: the portrait information and behavior information of all users in the current box are collected, and the collected user voice data is subjected to denoising and data cleaning, and then the address features in the voice data are extracted. Combined with the behavior information of the voice initiator and the behavior information of the user who responds to the voice, the mapping relationship between the address features and the portrait information is determined to obtain the portrait-address mapping library.

[0018] Furthermore, step S3 includes:

[0019] Based on the behavioral data collected by the image acquisition device, it is determined whether the user repeatedly performs the first specific action that conforms to the comfort scene mode within a unit time, or based on the sound data collected by the audio acquisition device, the user's current emotional information is analyzed and the emotional information is determined to be of the sad type. If so, the virtual digital human sets the scene mode corresponding to the current box to the comfort scene mode;

[0020] Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes:

[0021] The virtual digital human plays song information adapted to the comfort scene mode through an audio playback device and / or a display device.

[0022] Furthermore, the method includes:

[0023] Determining whether the user has performed a second specific action based on the behavioral data collected by the image acquisition device, and if so, switching the virtual digital human to a real-time conversation mode;

[0024] When the virtual digital human is in real-time dialogue mode, the image acquisition device captures the user's head movement trajectory and facial expressions in real time, adjusts the virtual digital human's head movement trajectory and facial expressions based on the captured user's head movement trajectory and facial expressions, and responds to the sound data emitted by the user in real time.

[0025] Furthermore, the method includes:

[0026] After detecting that the box has reached the upper limit of usage time, object recognition analysis is performed based on image data collected by image acquisition devices set at different angles in the box to determine whether there are any missing items in the current box. If so, the location of the missing items is determined based on the image data, and a first prompt message is issued through the virtual digital human. The first prompt message includes the object type of the identified missing object and the location of the missing object.

[0027] Furthermore, the method includes:

[0028] In response to the user's response data regarding the singing intention, control the virtual digital human to enter a duet mode, dividing the song information being sung into a first part sung by the user and a second part sung by the virtual digital human;

[0029] Collect first singing feature information when the user sings the first part of the current song information, and adjust second singing feature information when the virtual digital person sings the second part based on the first singing feature information. The first singing feature information and / or the second singing feature information include any one or more of the body movements, expressions, timbre, pitch, and rhythm during singing.

[0030] Furthermore, the response data of the duet intention includes a user's confirmation instruction for the duet intention, and the duet intention is generated according to the following method:

[0031] When it is detected that only one user is singing, the virtual digital human generates a duet intention, which is sent to the user in the form of a voice prompt. The voice prompt includes the user's name, which is obtained by analyzing the user portrait or the collected user voice data.

[0032] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for automatically controlling box equipment based on a virtual digital human as described in the first aspect of the present application is implemented.

[0033] In a third aspect, the present application also provides an electronic device comprising a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the method for automatically controlling the box equipment based on the virtual digital human as described in the first aspect of the present application is implemented.

[0034] Different from existing technologies, the above-mentioned method for automatically controlling box equipment based on a virtual digital human sets the virtual digital human's image and style based on the current merchant portrait, box portrait, user portrait, and predetermined box theme information. It then obtains the user's consumption data and song request data within the box, collects the user's behavior data within the box through an image acquisition device, and collects the user's voice data through an audio acquisition device. Multimodal data of the user is generated based on the consumption data, song request data, behavior data, and voice data. The user's multimodal data is input into a trained neural network model to obtain the scene mode corresponding to the current box. The virtual digital human's interaction strategy is then determined based on the scene mode corresponding to the current box. The box equipment is automatically controlled based on the virtual digital human's interaction strategy. The above-mentioned solution can determine the box scene mode based on the multimodal data, and then determine the interaction strategy based on the box scene mode. Automatically controlling the box equipment through the virtual digital human can effectively simplify user operations and enhance the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a method for automatically controlling box equipment based on a virtual digital human according to a first exemplary embodiment of the present invention;

[0036] Figure 2 This is a flow chart of a method for automatically controlling box equipment based on a virtual digital human according to a second exemplary embodiment of the present invention;

[0037] Figure 3 This is a flow chart of a method for automatically controlling box equipment based on a virtual digital human according to a third exemplary embodiment of the present invention;

[0038] Figure 4This is a flow chart of a method for automatically controlling box equipment based on a virtual digital human according to a fourth exemplary embodiment of the present invention;

[0039] Figure 5 This is a flow chart of a method for automatically controlling box equipment based on a virtual digital human according to a fifth exemplary embodiment of the present invention;

[0040] Figure 6 A schematic diagram of a module of an electronic device according to the present invention;

[0041] Reference numerals:

[0042] 10. Electronic equipment;

[0043] 101. Processor;

[0044] 102. Storage medium. DETAILED DESCRIPTION

[0045] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of this application, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application and are therefore only examples and are not intended to limit the scope of protection of this application.

[0046] References to "embodiments" herein mean that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the word "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or relevance to other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the various technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution.

[0047] Unless otherwise defined, the technical terms used herein have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms herein is only for describing specific embodiments and is not intended to limit this application.

[0048] In the description of this application, the term "and / or" is used to describe a logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and both A and B exist. In addition, the character " / " in this document generally indicates that the objects before and after are in a logical "or" relationship.

[0049] In this application, terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, priority or sequence relationship between these entities or operations.

[0050] Without further limitations, in this application, the words "include", "comprise", "have" or other similar expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product that includes the elements, so that the process, method or product that includes a series of elements may include not only those limited elements, but also other elements that are not explicitly listed, or also include elements that are inherent to such process, method or product.

[0051] Consistent with the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceed" are understood to exclude the number itself; expressions such as "above," "below," and "within" are understood to include the number itself. Furthermore, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this manner, such as "multiple groups," "multiple times," etc., unless otherwise specifically defined.

[0052] In the first aspect, the present application provides an automatic control method for box equipment based on virtual digital people, wherein the box equipment includes lighting equipment, audio playback equipment or display equipment, and image acquisition equipment and audio acquisition equipment are also provided in the box.

[0053] The lighting equipment in the box includes basic lighting equipment, atmosphere-creating lighting equipment, effect-enhancing lighting equipment, etc. Audio playback equipment includes audio sound-generating equipment (such as speakers, microphones), audio signal processing equipment (such as amplifiers, mixers, sound effect processors), audio playback source equipment (such as karaoke machines), and audio transmission equipment (such as audio cables or wireless transmission equipment). Display equipment includes video display equipment, LED light curtains, projectors, etc. Image acquisition equipment can be cameras distributed at different angles in the box, such as panoramic cameras, depth cameras, thermal imaging cameras, etc. Audio acquisition equipment can be microphones distributed at different locations in the box, such as dynamic microphones, condenser microphones, array microphones, and wireless microphones.

[0054] like Figure 1 As shown, the method includes:

[0055] S1: Set the image and style of the virtual digital person based on the current merchant portrait, box portrait, user portrait and the reserved box theme information.

[0056] In step S1, the merchant portrait includes basic merchant information (such as merchant name, geographical location, contact information), business information (such as the number and type of boxes, business hours, charging mode, etc.), service features (such as audio configuration, music library, value-added services, etc.), customer groups (such as customers' age distribution, consumption habits, etc.), market competition (such as market reputation, evaluation), etc.

[0057] A private room can be a KTV room, a cinema, a game room, an immersive experience center, a multi-functional conference room, etc. The private room portrait includes information about the space and layout (such as the area, seating arrangement, and the presence of a private bathroom), facilities and equipment (such as audio equipment, karaoke system, and display device configuration), decoration style information (such as color matching), usage history information (such as reservation frequency and peak usage hours), and cost-related information (such as charging standards and additional charges).

[0058] The user profile includes basic user information (such as age, gender, region, occupation, membership status, etc.), music preferences, and behavioral characteristics (such as the frequency of requesting various types of songs). The reserved box themes include birthday themes, Valentine's Day themes, anime themes, rock themes, etc.

[0059] The image and style of the virtual digital person can be determined based on a comprehensive consideration of the user's preferences, the overall decoration style of the box and the merchant, and the theme information when the current box is booked. For example, when the user portrait is male, the image of the virtual digital person can be set to female. Conversely, when the user portrait is female, the image of the virtual digital person can be set to male, and the clothing designed for the virtual digital person matches the decoration style of the merchant or the box.

[0060] S2: Obtain the user's consumption data and song request data in the box, and collect the user's behavior data in the box through image acquisition equipment, and collect the user's voice data through audio acquisition equipment, and generate the user's multimodal data based on the consumption data, song request data, behavior data and voice data.

[0061] In step S2, user consumption data covers a wide range of aspects, including product purchases, service consumption, entertainment activities, and other miscellaneous expenses. This data can reflect user consumption behaviors and preferences, helping businesses optimize their business strategies and providing accurate service recommendations for digital humans. Song request data within private rooms is key to understanding user musical preferences and entertainment behaviors. This data covers song selection, request operations, playback duration and order, and related interactive information, providing a basis for KTV businesses to optimize their song libraries, improve services, and for digital humans to provide personalized recommendations.

[0062] The behavioral data of users in the private rooms can reflect their activities and preferences during the entertainment process. It covers aspects such as body movements, vocal language, participation in entertainment activities, and interaction with equipment and the environment. User audio data includes voice data of conversations between users, singing data of users, and audio data expressing users' emotions (such as laughter, crying, screaming, sighing, etc.).

[0063] When generating multimodal data, collected consumption data, song request data, behavioral data, and sound data can be integrated and processed to form the user's multimodal data. Specifically, techniques such as data annotation and feature extraction can be used to convert different types of data into a unified format and feature representation. For example, information such as product names and prices in consumption data can be encoded, song features can be extracted from song request data, and movement features from behavioral data and audio features from sound data can be quantified. These features can then be combined into a multimodal data vector.

[0064] S3: Input the user's multimodal data into the trained neural network model to obtain the scene mode corresponding to the current box, and determine the interaction strategy of the virtual digital human based on the scene mode corresponding to the current box.

[0065] When implementing step S3, the neural network model can be pre-trained using a large amount of multimodal data, allowing the model to learn the characteristics and patterns of multimodal data in different scene modes. When the user's multimodal data is input, the model uses forward propagation to calculate and output the scene mode probability corresponding to the current box. For example, after training, the model can recognize that in the case of a birthday party, the multimodal data may contain image features of a birthday cake, audio features of birthday-related songs, and birthday set meals in consumption data. When the input multimodal data meets these characteristics, the model determines that the current scene is a birthday party.

[0066] S4: Automatically adjust and control the box equipment based on the interactive strategy of the virtual digital human.

[0067] In step S4, a corresponding interaction strategy for the virtual digital human is formulated based on the scenario mode output by the neural network model. For example, if the current scenario mode is a birthday party, the virtual digital human's interaction strategy might include: controlling the audio playback device to play songs related to birthday wishes; controlling the display device to display lyrics and animations related to birthday wishes; sending birthday wishes to the user via voice, and organizing other users to sing a birthday song together. For a more casual singing entertainment scenario, the interaction strategy might include recommending popular songs and exchanging singing techniques with the user.

[0068] The above solution can determine the box scene mode based on multimodal data, and then determine the interaction strategy based on the box scene mode. By automatically adjusting the box equipment through virtual digital humans, it can effectively simplify user operations and enhance the user's interactive experience.

[0069] In some embodiments, step S3 includes:

[0070] Based on whether there is a birthday cake in the image captured in real time by the image acquisition device and whether the user in the box is blowing out candles, the virtual digital human sets the scene mode corresponding to the current box to the birthday scene mode;

[0071] Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes:

[0072] Execute any one or more of the following interactive strategies through the virtual digital human: control the audio playback device to play birthday blessing related songs, control the display device to display the lyrics of birthday blessing related songs, and control the lighting device to present lighting effects that match the current birthday scene mode.

[0073] Specifically, for birthday cake recognition, we can leverage deep learning object detection algorithms (such as Faster R-CNN and YOLO) to train the model with a large number of images containing birthday cakes. This allows the model to learn the characteristics of birthday cakes, including shape, color, and texture. After the image acquisition device captures the image in real time and feeds it into the trained model, the model can then determine whether a birthday cake is present in the image.

[0074] To identify candle-blowing actions, posture estimation and action recognition technologies, such as OpenPose algorithms, can be used to detect human posture and movement. First, data on the user's candle-blowing actions, including changes in the position of joints and the time series of the action, is collected to train an action recognition model. By analyzing continuous images captured by the image acquisition device, the model can determine whether the user in the box is blowing out the candles.

[0075] When the virtual human recognizes the current scene mode as birthday mode, it communicates with the audio player, display device, and lighting equipment. For the audio player, the virtual human selects and plays a birthday song by calling the corresponding music playback interface. For the display device, the virtual human uses graphics rendering technology to display the lyrics of the birthday song on the screen. For the lighting equipment, the virtual human controls parameters such as color, brightness, and flickering frequency to create lighting effects that match the birthday scene mode, such as a warm yellow hue or a simulated candle flickering effect.

[0076] Through the above solution, an automated process from image recognition to scene judgment to device control can be realized. The virtual digital human can execute a series of interactive strategies according to the birthday scene mode, such as playing songs, displaying lyrics and adjusting lighting effects. Users no longer need to manually adjust the operating parameters of the box equipment, which effectively simplifies user operations and improves the user's interactive experience.

[0077] In some embodiments, as Figure 2 As shown, step S3 includes:

[0078] Step S201: determining whether the user has made a photo-taking action based on the image captured in real time by the image acquisition device and / or determining whether the user has issued a photo-taking request based on the sound data collected by the audio acquisition device. If so, the virtual digital human sets the scene mode corresponding to the current box to the photo-taking scene mode;

[0079] Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes:

[0080] Step S202: issuing a photo-taking guidance message prompt via the virtual digital human, and adjusting the background light at the user's location by adjusting the brightness of the lighting device.

[0081] In step S201, when a photo request is identified through a captured image, computer vision technology can be used with deep learning algorithms, such as a convolutional neural network (CNN)-based action recognition model. A large amount of image data containing various photo actions (holding a camera, holding a phone in front of the face, striking a pose, etc.) is first collected to train the model, allowing it to learn the key characteristics of photo actions. After the image acquisition device captures the image of the room in real time, the image is input into the trained model. The model analyzes the posture, body position, and movement trajectory of the person in the image to determine whether the user has made a photo action.

[0082] When a photo request is identified through collected sound data, speech recognition technology can be used, utilizing a speech recognition system based on a hidden Markov model (HMM) or deep learning. Voice samples containing various photo requests, such as "take a photo" and "take a photo for me," are collected and trained to learn the acoustic features and semantic information of these voices. Audio acquisition equipment collects sound data from the room in real time, converts it into digital signals, and then inputs it into the speech recognition system. The system performs feature extraction, acoustic model matching, and language model decoding on the speech to determine whether the user has issued a photo request.

[0083] In step S202, the photo-taking guide information includes standing position guide information, facial expression guide information and body movement guide information; the standing position guide information and / or facial expression guide information and / or body movement guide information include user address information corresponding to the need to adjust the standing position, facial expression and body movement when taking photos, and the user address information is obtained from a portrait-address mapping library based on the user portrait information. The portrait-address mapping library is generated in the following manner: the portrait information and behavior information of all users in the current box are collected, and the voice data of the collected users are denoised and cleaned, and then the address features in the voice data are extracted. In combination with the behavior information of the voice initiator and the behavior information of the user who responds to the voice, the mapping relationship between the address features and the portrait information is determined to obtain the portrait-address mapping library.

[0084] For example, if there are 10 users in a private room, their portrait information can be captured by cameras placed at different angles within the room, or by comparing the images captured by the cameras with the user portrait information stored in the backend system, and then using the user identification avatars in the user portrait information as the user's portrait information. As for how users address each other, the audio of the conversation after the user enters the room can be collected, and the multimodal model can be used to analyze the names of each user in the gathering using other users. This name can then be mapped and stored with the user's portrait information. Of course, the user's address information can also be manually adjusted or corrected through a conversation between the user and the virtual digital human. For example, a user can say to the virtual digital human, "Change the name of player A to B," and the virtual digital human will adaptively adjust the mapping relationship based on the received instruction.

[0085] When users need to be guided on their positions for photos, the virtual human analyzes suitable positions based on a pre-modeled layout of the private room and real-time occupant location information captured by the image acquisition device. For example, if there are multiple people in the private room, the virtual human can calculate positions that ensure everyone is clearly in the frame and the layout is reasonable, based on the number of people and the size of the room. It then generates corresponding guidance messages, such as "Please move closer together and stand in two rows. Those in the back row should stand on their toes slightly." The virtual human conveys these guidance messages to users in a clear and friendly voice via an audio output device.

[0086] When it is necessary to guide the user's facial expression when taking photos, the virtual digital human can also analyze the user tags collected by the image acquisition device. For example, if it detects that the expression of one or more users is relatively stiff when taking photos, a variety of facial expression guidance words suitable for taking photos can be preset, such as "Please raise the corners of your mouth and show a bright smile" and "Relax your eyes and look at the camera naturally."

[0087] When guiding users' body movements during photos, the virtual human pre-accesses a rich database of body movement guidance templates. Upon detecting a photo scene, the virtual human generates personalized body movement guidance information based on the number of users, gender, age, and other information. For example, for a group photo shoot with young people, the virtual human might suggest, "Everyone, please strike some lively and interesting poses, such as making a heart shape or waving your hands." For a family gathering, the virtual human might suggest, "Elders can naturally place their hands in front of them, while younger generations stand to the side, leaning slightly." The virtual human then conveys these guidance messages to the user via voice.

[0088] When the background lighting at the user's location needs to be adjusted, the virtual human establishes a communication connection with the lighting control system. Based on the user's location information obtained by the image acquisition device and an analysis of the current lighting conditions, the background model algorithm calculates the appropriate lighting brightness for the photo. For example, if the user is in a dimly lit corner, the virtual human sends a control command to the lighting system to increase the brightness in that area. If the background light is too strong, the brightness is reduced to avoid overexposure in the photo. By monitoring light changes in real time and dynamically adjusting the lighting brightness, the user's background lighting is always optimal for photographing.

[0089] like Figure 3 As shown, in some embodiments, step S3 includes:

[0090] Step S301: Based on the behavioral data collected by the image acquisition device, it is determined whether the user repeatedly performs the first specific action that conforms to the comfort scene mode within a unit time, or based on the sound data collected by the audio acquisition device, the user's current emotional information is analyzed and the emotional information is determined to be of the sad type. If so, the virtual digital human sets the scene mode corresponding to the current box to the comfort scene mode;

[0091] Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes:

[0092] Step S302: The virtual digital human plays song information adapted to the comfort scene mode through an audio playback device and / or a display device.

[0093] In step S301, an image acquisition device (such as a camera) is used to continuously collect image data in the box, and computer vision technology is used to analyze the image in real time. A motion recognition model is constructed using a deep learning algorithm, such as a convolutional neural network (CNN). During the model training phase, a large number of image samples containing first specific actions such as clapping the table are collected, and the key feature points of the action are marked, such as the position of the hand at the moment of contact with the table, the trajectory of the hand movement, etc. By learning from these samples, the model can identify whether the action of the character in the image is the first specific action that conforms to the comfort scene mode. At the same time, a unit time (such as 1 minute) is set, and the number of times the first specific action is detected within the time period is counted. When the number exceeds a preset threshold (such as 5 times), it is determined that the user has repeatedly made the action within the unit time.

[0094] In step S301, the audio acquisition device (such as a microphone) collects the sound data in the box in real time and converts it into a digital signal. Using speech emotion recognition technology, an emotion analysis system including an acoustic feature extraction module and a classification model is constructed. The acoustic feature extraction module extracts features such as pitch, volume, and resonance peaks from the sound data, and the classification model (such as support vector machine SVM, recurrent neural network RNN, etc.) learns the acoustic feature patterns corresponding to different emotions (such as sadness, happiness, anger, etc.) during the training phase. By performing feature extraction and model matching on the collected sound data, it is determined whether the user's current emotional information is of the sad type. If it is determined to be a sad emotion, combined with the action recognition result (if any), the judgment of the comfort scene mode is triggered.

[0095] In step S302, once behavioral or sound data analysis determines that the current box should switch to comfort scene mode, the virtual digital human selects appropriate song information (such as soothing and healing music) from the pre-built song library based on the needs of the comfort scene mode. Through the control interface of the audio playback device, a song playback instruction is sent to the audio playback device to specify the selected comfort song. The song lyrics, album cover, and other related information can be displayed on the screen through the graphics rendering interface of the display device, creating an immersive and comforting atmosphere for the user.

[0096] Furthermore, in the comfort scene mode, the virtual human can access a database storing user consumption data to obtain the user's consumption records in the box, including information such as the amount spent and items consumed. The virtual human can organize this data, along with relevant information triggered by the comfort scene mode (such as the number of motion recognitions and emotion analysis results), and send a feedback request to the merchant's front-end management system via network communication protocols (such as HTTP and TCP / IP). Upon receiving the request, the merchant's front-end management system arranges for staff to provide services such as fruit plates to the user based on pre-set rules (such as a fruit plate awarded for historical consumption reaching a certain threshold).

[0097] Through the above solution, it is possible to automatically identify whether the current box is in comfort scene mode, and when the scene mode is identified, soothing songs are played to enrich the user's overall experience in the box, making the venue more distinctive and competitive.

[0098] like Figure 4 As shown, in some embodiments, the method includes:

[0099] Step S401: determining whether the user has performed a second specific action based on the behavior data collected by the image acquisition device, and if so, switching the virtual digital human to a real-time conversation mode;

[0100] Step S402: When the virtual digital human is in real-time conversation mode, the image acquisition device captures the user's head movement trajectory and facial expressions in real time, adjusts the virtual digital human's head movement trajectory and facial expressions based on the captured user's head movement trajectory and facial expressions, and responds to the sound data emitted by the user in real time.

[0101] In this embodiment, the second specific action can be a gesture used by a user to greet a virtual digital human. In practice, a gesture recognition model can be constructed using a convolutional neural network (CNN) from deep learning. During the model training phase, a large number of image samples containing user greeting gestures (such as waving) toward a virtual digital human are collected and annotated. By learning the features of these samples, the model can accurately identify greeting gestures in images. After preprocessing the image captured by the image acquisition device, it is input into the gesture recognition model. The model determines whether the second specific action (greeting gesture) is present. If so, the virtual digital human switches to real-time conversation mode.

[0102] To capture the head's motion trajectory, target tracking algorithms (such as KCF and CSRT) can be used to track the user's head in the image sequence captured in real time by the image acquisition device. By detecting the position changes of the head's feature points (such as the eyes, nose, and mouth) in different frames, the head's motion trajectory, including translation and rotation information, can be calculated.

[0103] When recognizing a user's facial expression, a deep learning-based facial expression recognition model, such as one based on the ResNet architecture, can be used. This model, trained on a large dataset of facial images with expression labels (such as happiness, sadness, and anger), can accurately recognize a user's facial expression. The facial image captured by the image acquisition device is analyzed, facial features are extracted, and input into the expression recognition model to determine the user's current facial expression.

[0104] After capturing the user's head motion and facial expressions, this information can be transmitted to the virtual human's control system. The control system uses animation rendering technology to adjust the virtual human's head motion and facial expressions, ensuring that the virtual human's performance aligns with the user's, enhancing the realism of the interaction.

[0105] Through the above solution, users can easily switch the virtual digital human to real-time conversation mode with simple gestures. During the conversation, the virtual digital human's head movement and facial expressions are synchronized with the user, making the interaction more natural and real, greatly enhancing the user's entertainment experience in the box.

[0106] In some embodiments, the method comprises:

[0107] After detecting that the box has reached the upper limit of usage time, object recognition analysis is performed based on image data collected by image acquisition devices set at different angles in the box to determine whether there are any missing items in the current box. If so, the location of the missing items is determined based on the image data, and a first prompt message is issued through the virtual digital human. The first prompt message includes the object type of the identified missing object and the location of the missing object.

[0108] Specifically, the virtual digital human can connect to the private room reservation management system to obtain the time information for the private room reservation. When the time in the isolated private room reservation system reaches the reserved end time, a detection signal is triggered to indicate that the private room has been used up. Alternatively, a timing device can be installed in the private room, starting from the moment the user opens the room and signaling that the room has been used up after a preset time has elapsed.

[0109] Image acquisition devices (such as multiple high-definition cameras) positioned at different angles within the private room immediately begin capturing images upon receiving a signal indicating the room has been occupied. These devices cover all areas of the private room. The captured image data is transmitted to a multimodal model for analysis. Image preprocessing, including denoising and contrast enhancement, is performed to improve image quality for subsequent analysis. The multimodal model then uses deep learning target detection algorithms, such as FasterR-CNN and the YOLO series, to perform object recognition analysis on the preprocessed images. The model is pre-trained with a large number of images containing various commonly missed items (such as mobile phones, wallets, coats, and umbrellas) to learn the characteristics of these items, including shape, color, and texture. After the private room images are fed into the trained model, the model detects and classifies the objects in the image, determines whether any are missed, and identifies the type of item.

[0110] The missing object's actual physical location within the box is then calculated using algorithms such as triangulation, based on the image acquisition device's installation position, shooting angle, and the pixel coordinates of the object in the image. For example, if the positions and shooting angles of two cameras, as well as the pixel coordinates of the missing object in the two camera images, are known, geometric calculations can be used to determine its location in three-dimensional space.

[0111] After receiving the type and location of the missing item, the virtual human generates a first prompt message, such as "User A, you left a phone on the left armrest of the sofa in the private room." The virtual human then plays the prompt message in voice format through an audio output device (such as a speaker), reminding the user to check for the missing item. Regarding the confirmation of the user's name in the first prompt message, please refer to the method described above for analyzing and confirming the user's voice data, which will not be elaborated here.

[0112] This solution uses a virtual digital human to automatically detect and remind users of items left behind in private rooms. This reduces the workload for staff members to manually check private rooms, improves management efficiency, and reduces labor costs. It also reduces the risk of lost or stolen items and mitigates subsequent disputes and management issues caused by users leaving items behind.

[0113] like Figure 5 As shown, in some embodiments, the method includes:

[0114] Step S501: In response to the user's response data regarding the singing intention, the virtual digital human is controlled to enter a duet mode, and the song information being sung is divided into a first part sung by the user and a second part sung by the virtual digital human;

[0115] Step S502: Collect the first singing feature information when the user sings the first part of the current song information, and adjust the second singing feature information when the virtual digital person sings the second part based on the first singing feature information.

[0116] Preferably, the first singing feature information and / or the second singing feature information include any one or more of body movements, facial expressions, timbre, pitch, and rhythm during singing.

[0117] In this embodiment, the virtual digital human can monitor the user's input response data in real time. Specifically, voice commands can be captured through a microphone, or a duet option can be set on the song request interface, and the user clicks it to generate operation command data. If the command is a voice command, speech recognition technology is used to convert the audio signal into text. Natural language processing technology is then used to parse the text to determine whether it contains keywords indicating duet intention, such as "duet" or "sing together." If the command data is an operation command, the user's selection of the duet function is directly recognized. Once the user's duet intention is confirmed, the virtual digital human is triggered to enter the duet mode process.

[0118] After confirming the entry into duet mode, the virtual digital human obtains the audio data and related information of the song being sung, such as the song's duration, beat, and melody. Using a music analysis algorithm, the song is divided into the first part sung by the user and the second part sung by the virtual digital human, based on the song's structure (such as the verse and chorus), rhythm changes, and preset division rules. For example, based on the song's paragraphs, odd-numbered paragraphs can be assigned to the user and even-numbered paragraphs to the virtual digital human; or the song can be divided evenly by duration, ensuring that both parties have roughly equal singing time, while also taking into account the climax of the song and assigning it to the more suitable person to improve the performance.

[0119] Then, multimodal data of the user singing the first part of the song can be collected through various data collection devices. Specifically, the user's body movements and facial expressions can be captured in real time through image collection devices (such as cameras), and computer vision technology can be used to identify the user's body movements (such as waving, swaying the body) and facial expressions (such as smiling, frowning) with the help of deep learning models. Alternatively, the user's singing voice can be collected through audio collection devices (such as microphones), and digital signal processing technology and acoustic analysis algorithms can be used to extract timbre, pitch, rhythm and other features. For example, pitch information can be obtained by performing Fourier transform analysis on the audio signal; rhythm features can be extracted by analyzing the energy changes and periodicity of the audio signal; and timbre characteristics can be determined by analyzing the spectral characteristics of the audio.

[0120] A feature mapping relationship is then established based on the collected feature information of the user's first performance. For example, if the user's singing pitch is high and the rhythm is brisk, the pre-trained model is used to generate the pitch and rhythm parameters for the second part of the virtual digital human's performance, so that the style is consistent with the user's performance. For body movements and expressions, the user's body movements and facial expressions are also converted into corresponding movement and expression instructions for the virtual digital human based on pre-set association rules or through machine learning models. For example, if the user frequently waves their hands during the performance, the virtual digital human will also add arm swinging movements when performing the second part. The adjusted second performance feature information is then applied to the virtual digital human's performance generation process. The virtual digital human's body movements and facial expressions are generated through animation rendering technology, and the timbre, pitch, and rhythm of the virtual digital human's singing are adjusted through speech synthesis technology.

[0121] This solution enables duet interaction between users and virtual digital beings, transforming the traditional solo singing experience. This allows users to experience collaboration with their virtual partner, enhancing the sense of participation and enjoyment in the private room. Furthermore, the virtual being's singing performance can be adjusted in real time based on the user's singing style, allowing both parties to more closely match their singing styles, creating a sense of duet with a partner who is familiar with their singing habits, effectively enhancing the interactive experience.

[0122] In some embodiments, the response data of the duet intention includes the user's confirmation instruction for the duet intention, and the duet intention is generated according to the following method: when it is detected that only one user is singing, the virtual digital human generates a duet intention, and the duet intention is issued to the user in the form of a voice prompt, and the voice prompt includes the user's name, and the user name is obtained by analyzing the user portrait or the collected user voice data.

[0123] In this way, when the virtual digital human detects that only one user is singing, it triggers a duet intention generation mechanism. Specifically, based on pre-set rules and strategies, the virtual digital human can initiate a duet invitation to the user. For example, if a user sings a certain number of songs consecutively or the singing duration reaches a certain threshold, it can be considered that the user may want a singing partner, and the virtual digital human will generate a duet intention.

[0124] Then, the virtual digital human system generates a voice prompt containing a duet invitation through the audio playback device based on the generated duet intention. At the same time, the voice prompt of the duet invitation can include the user's name. If the user has previously established a user portrait, the virtual digital human can extract the user's name and other address information from the user portrait database; if there is no user portrait, it is analyzed based on the collected user voice data. Use speech recognition technology to convert the user's voice into text, and then use natural language processing technology to analyze the address-related information in the text (such as the name in the user's self-introduction, or the address of the user who is singing independently by other users in the box, etc.), or match the voice characteristics (such as voiceprints, etc.) with the existing database to obtain possible user addresses. Finally, the user's address is integrated into the voice prompt to send a duet invitation to the user in a more personalized way.

[0125] Through this solution, when a single user is detected singing, the virtual human can proactively invite them to a duet, breaking the monotony of traditional solo performances and providing users with a new form of entertainment and interaction. Duets between users and the virtual human add interactivity and fun to the entertainment process, making the private room entertainment more engaging and enriching.

[0126] In a second aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic control method for box equipment based on virtual digital humans as described in the first aspect of the present invention.

[0127] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.

[0128] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface storage, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface storage may be a magnetic disk storage or a magnetic tape storage.

[0129] The volatile memory may be a random access memory (RAM) that is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronized dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The computer-readable storage medium described in the embodiments of the present invention is intended to include these and any other suitable types of memory.

[0130] like Figure 6 As shown, in the third aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, on which a computer program is stored. When the computer program is executed by the processor, the automatic control method of the box equipment based on the virtual digital human as described in the first aspect of the present invention is implemented.

[0131] In some embodiments, the processor can be implemented through software, hardware, firmware or a combination thereof, and can use at least one of a circuit, a single or multiple application-specific integrated circuits (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor, so that the processor can execute some or all of the steps or any combination of the steps in the automatic control method of the box equipment based on the virtual digital human described in the various embodiments of the present application.

[0132] It should be noted that although the above embodiments have been described herein, this does not limit the scope of patent protection of the present invention. Therefore, based on the innovative concept of the present invention, changes and modifications to the embodiments described herein, or equivalent structural or equivalent process transformations made using the contents of the present invention's specification and drawings, and direct or indirect application of the above technical solutions to other related technical fields, are all included in the scope of patent protection of the present invention.

Claims

1. An automatic control method for box equipment based on virtual digital human, characterized in that: The box equipment includes lighting equipment, audio playback equipment or display equipment, and image acquisition equipment and audio acquisition equipment are also provided in the box; The method comprises the following steps: S1: Set the image and style of the virtual digital person based on the current merchant portrait, box portrait, user portrait, and the reserved box theme information; S2: Obtain the user's consumption data and song request data in the box, and collect the user's behavior data in the box through an image acquisition device, and collect the user's voice data through an audio acquisition device, and generate the user's multimodal data based on the consumption data, song request data, behavior data and voice data; S3: Input the user's multimodal data into the trained neural network model to obtain the scene mode corresponding to the current box, and determine the interaction strategy of the virtual digital human based on the scene mode corresponding to the current box; S4: Automatically adjust and control the box equipment based on the interactive strategy of the virtual digital human.

2. The automatic control method for box equipment based on virtual digital human according to claim 1, characterized in that: Step S3 includes: Based on whether there is a birthday cake in the image captured in real time by the image acquisition device and whether the user in the box is blowing out candles, the virtual digital human sets the scene mode corresponding to the current box to the birthday scene mode; Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes: Execute any one or more of the following interactive strategies through the virtual digital human: control the audio playback device to play birthday blessing related songs, control the display device to display the lyrics of birthday blessing related songs, and control the lighting device to present lighting effects that match the current birthday scene mode.

3. The automatic control method of the box equipment based on the virtual digital human according to claim 1 is characterized in that: Step S3 includes: Determining whether the user has made a photo-taking action based on the image captured in real time by the image acquisition device and / or determining whether the user has issued a photo-taking request based on the sound data collected by the audio acquisition device, if so, the virtual digital human sets the scene mode corresponding to the current box to the photo-taking scene mode; Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes: The virtual digital human issues a photo-taking guidance prompt, and the background light at the user's location is adjusted by adjusting the brightness of the lighting device, wherein the photo-taking guidance information includes position guidance information, facial expression guidance information, and body movement guidance information; The standing position guidance information and / or facial expression guidance information and / or body movement guidance information includes user address information corresponding to the need to adjust the standing position, facial expression, and body movement when taking pictures. The user address information is obtained from a portrait-address mapping library based on the user portrait information. The portrait-address mapping library is generated in the following manner: the portrait information and behavior information of all users in the current box are collected, and the collected user voice data is subjected to denoising and data cleaning, and then the address features in the voice data are extracted. Combined with the behavior information of the voice initiator and the behavior information of the user who responds to the voice, the mapping relationship between the address features and the portrait information is determined to obtain the portrait-address mapping library.

4. The automatic control method for box equipment based on virtual digital human according to claim 1 is characterized in that: Step S3 includes: Based on the behavioral data collected by the image acquisition device, it is determined whether the user repeatedly performs the first specific action that conforms to the comfort scene mode within a unit time, or based on the sound data collected by the audio acquisition device, the user's current emotional information is analyzed and the emotional information is determined to be of the sad type. If so, the virtual digital human sets the scene mode corresponding to the current box to the comfort scene mode; Determining the interactive strategy of the virtual digital human based on the scene mode corresponding to the current box includes: The virtual digital human plays song information adapted to the comfort scene mode through an audio playback device and / or a display device.

5. The automatic control method of the box equipment based on the virtual digital human according to claim 1 is characterized in that: The method comprises: Determining whether the user has performed a second specific action based on the behavioral data collected by the image acquisition device, and if so, switching the virtual digital human to a real-time conversation mode; When the virtual digital human is in real-time dialogue mode, the image acquisition device captures the user's head movement trajectory and facial expressions in real time, adjusts the virtual digital human's head movement trajectory and facial expressions based on the captured user's head movement trajectory and facial expressions, and responds to the sound data emitted by the user in real time.

6. The automatic control method of the box equipment based on the virtual digital human according to claim 1 is characterized in that: The method comprises: After detecting that the box has reached the upper limit of usage time, object recognition analysis is performed based on image data collected by image acquisition devices set at different angles in the box to determine whether there are any missing items in the current box. If so, the location of the missing items is determined based on the image data, and a first prompt message is issued through the virtual digital human. The first prompt message includes the object type of the identified missing object and the location of the missing object.

7. The automatic control method for box equipment based on virtual digital human according to claim 1 is characterized in that: The method comprises: In response to the user's response data regarding the singing intention, control the virtual digital human to enter a duet mode, dividing the song information being sung into a first part sung by the user and a second part sung by the virtual digital human; Collect first singing feature information when the user sings the first part of the current song information, and adjust second singing feature information when the virtual digital person sings the second part based on the first singing feature information. The first singing feature information and / or the second singing feature information include any one or more of the body movements, expressions, timbre, pitch, and rhythm during singing.

8. The automatic control method for box equipment based on virtual digital human according to claim 1 is characterized in that: The response data of the duet intention includes a user's confirmation instruction for the duet intention, and the duet intention is generated according to the following method: When it is detected that only one user is singing, the virtual digital human generates a duet intention, which is sent to the user in the form of a voice prompt. The voice prompt includes the user's name, which is obtained by analyzing the user portrait or the collected user voice data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the automatic control method of the box equipment based on the virtual digital human as described in any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: It includes a processor and a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by the processor, the automatic control method of the box equipment based on the virtual digital human as described in any one of claims 1 to 8 is implemented.

Images