Interaction method and device, storage medium, equipment and program product

Abstract

The invention discloses an interaction method and device, a storage medium, equipment and a program product, and the method comprises the steps: displaying a target page through a user side, and displaying at least one virtual character on the target page; obtaining multi-modal input data, wherein the multi-modal input data comprises at least one of position information data, microphone data and touch screen data; sending the multi-modal input data to a server, so that the server generates interaction feedback information for the target page or the virtual character according to the multi-modal input data; and receiving interaction feedback information sent from the server, and displaying the interaction feedback information on the target page. According to the scheme provided by the embodiment of the invention, the depth and breadth of user experience can be enhanced, so that the user can obtain a response in real time in an interaction process, and the reality sense and interactivity of interaction are enhanced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an interaction method, apparatus, storage medium, device, and program product. Background Art

[0002] In the current booming development of the Internet, interactive platforms have become important places for people's daily communication, entertainment, and learning. However, although there are a variety of interactive platforms on the market, most of them are still limited to a single interaction method, such as the transmission of text or pictures, which to a certain extent restricts the depth and breadth of the user experience. This single interaction mode not only makes it difficult to meet the user's demand for in-depth interaction, but also seems inadequate in the instant feedback mechanism. The information sent by users on the platform often cannot be quickly responded to, which not only reduces the communication efficiency, but also weakens the user's enthusiasm and initiative to participate in the interaction. Summary of the Invention

[0003] Embodiments of this application provide an interaction method, apparatus, storage medium, device, and program product, which can achieve in-depth interaction between users and target pages and virtual characters, not only enriching the user's interaction experience, but also enhancing the interactivity between the user and the platform to which the target page belongs.

[0004] On the one hand, embodiments of this application provide an interaction method, which includes:

[0005] Display a target page through a user terminal, where at least one virtual character is displayed on the target page;

[0006] Obtain multimodal input data, where the multimodal input data includes at least one of position information data, microphone data, and touch screen data;

[0007] Send the multimodal input data to a server so that the server generates interaction feedback information for the target page or the virtual character according to the multimodal input data;

[0008] Receive the interaction feedback information sent from the server and display the interaction feedback information on the target page.

[0009] On the other hand, embodiments of this application provide another interaction method, which includes:

[0010] Receive multimodal input data sent from a user terminal, where the multimodal input data includes at least one of position information data, microphone data, and touch screen data;

[0011] Generate interaction feedback information for a target page or a virtual character according to the multimodal input data, where the target page is a page displayed on the client, and at least one virtual character is displayed on the target page;

[0012] Send the interaction feedback information to the client so that the client displays the interaction feedback information on the target page.

[0013] On the other hand, an embodiment of the present application provides an interaction device, and the device includes:

[0014] A display unit for displaying a target page through a client, and at least one virtual character is displayed on the target page;

[0015] An acquisition unit for acquiring multimodal input data, where the multimodal input data includes at least one of position information data, microphone data, and touch screen data;

[0016] A first sending unit for sending the multimodal input data to a server so that the server generates interaction feedback information for the target page or the virtual character according to the multimodal input data;

[0017] An interaction unit for receiving the interaction feedback information sent from the server and displaying the interaction feedback information on the target page.

[0018] On the other hand, an embodiment of the present application provides another interaction device, and the device includes:

[0019] A receiving unit for receiving multimodal input data sent from a client, where the multimodal input data includes at least one of position information data, microphone data, and touch screen data;

[0020] A generating unit for generating interaction feedback information for a target page or a virtual character according to the multimodal input data, where the target page is a page displayed on the client, and at least one virtual character is displayed on the target page;

[0021] A second sending unit for sending the interaction feedback information to the client so that the client displays the interaction feedback information on the target page.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the interaction method described in any of the above embodiments.

[0023] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory. A computer program is stored in the memory, and the processor is configured to execute the interaction method described in any of the foregoing embodiments by calling the computer program stored in the memory.

[0024] On the other hand, an embodiment of the present application provides a computer program product, including computer instructions, which implement the interaction method described in any of the foregoing embodiments when executed by a processor.

[0025] In an embodiment of the present application, a target page is displayed on a user terminal, and at least one virtual character is displayed on the target page; multi-modal input data is obtained, and the multi-modal input data includes at least one of position information data, microphone data, and touch screen data; the multi-modal input data is sent to a server so that the server generates interaction feedback information for the target page or the virtual character according to the multi-modal input data; the interaction feedback information sent from the server is received and the interaction feedback information is displayed on the target page. In the embodiment of the present application, by collecting multi-modal input data such as position information data, microphone data, and touch screen data, richer interaction intentions and behaviors of the user can be captured, enhancing the depth and breadth of the user experience. In addition, the multi-modal input data is sent to the server for processing, and the server can quickly generate interaction feedback information for the target page or the virtual character according to these data. This mechanism of instant processing and feedback significantly improves the efficiency of communication, enabling the user to obtain an immediate response during the interaction process and enhancing the authenticity and interactivity of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the architecture of the interaction system provided by an embodiment of the present application.

[0027] Figure 2 It is a first flowchart of the interaction method provided by an embodiment of the present application.

[0028] Figure 3 It is a schematic diagram of the target page provided by an embodiment of the present application.

[0029] Figure 4 It is a schematic diagram of the target page including target activity information provided by an embodiment of the present application.

[0030] Figure 5 It is a schematic diagram of the page including lighting feedback information provided by an embodiment of the present application.

[0031] Figure 6 It is a schematic diagram of the touch effective area and the non-active touch area provided by an embodiment of the present application.

[0032] Figure 7It is a second process schematic diagram of the interaction method provided by an embodiment of the present application.

[0033] Figure 8 It is a first structural schematic diagram of the interaction device provided by an embodiment of the present application.

[0034] Figure 9 It is a second structural schematic diagram of the interaction device provided by an embodiment of the present application.

[0035] Figure 10 It is a structural schematic diagram of the computer device provided by an embodiment of the present application.

[0036] Figure 11 It is a schematic diagram of the storage medium provided by an embodiment of the present application. Detailed implementation manners

[0037] Embodiments of the present application provide an interaction method, device, storage medium, device and program product. Specifically, the interaction method of the embodiments of the present application can be executed by a computer device, where the computer device can be a terminal or a server and other devices. The terminal can be a smart phone, a tablet computer, a notebook computer, a smart TV, a smart speaker, a wearable smart device, a smart vehicle terminal and other devices. The terminal can also include a client, and the client can be a video client, a browser client, an instant messaging client or a small program, etc. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0038] It should be noted that the information (including but not limited to: information input by users, etc., for example, information input by users into the input box), data (including but not limited to data for analysis, stored data, displayed data, etc., for example, context code, all codes of the current project, service pressure corresponding to operations performed on all codes of the current project, code development status of the current project), and signals involved in the present application are all authorized by users or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards. For example, the context code, operations performed on all codes of the current project, service pressure corresponding to the operations, and code development status involved in the present application are all obtained under full authorization.

[0039] Embodiments of the present application can be applied to various scenarios such as virtual activities, e-commerce services, social platforms, game entertainment, etc.

[0040] First, some nouns or terms that appear in the description of the embodiments of the present application are explained as follows:

[0041] Multimodal interaction: Multimodal interaction refers to the human-computer interaction method through multiple sensory channels (such as sound, touch, etc.).

[0042] Fast Fourier Transform: Fast Fourier Transform is an efficient algorithm used to convert time domain signals into frequency domain signals.

[0043] Kalman Filter: A Kalman filter is a recursive filter that estimates the state of a system and filters out noise.

[0044] Nowadays, interactive platforms have become important places for communication, entertainment and learning. However, most platforms in related technologies are still limited to text or picture interaction, which restricts user experience. This single mode is difficult to meet the needs of deep interaction and lacks immediate feedback. User information often does not receive a quick response, which reduces communication efficiency and weakens participation enthusiasm.

[0045] Taking into account the problems existing in the above-mentioned related technologies, the embodiments of the present application provide an interactive method, apparatus, storage medium, device and program product to solve the technical problems that most interactive platforms in the related technologies still rely on a single interactive method (such as text or pictures), lack deep interaction and instant feedback mechanism, and affect the user's interactive experience.

[0046] Please refer to Figure 1 , Figure 1 The schematic diagram of the architecture of the interactive system provided in the embodiment of the present application. The system can implement the interactive method. The interactive system may include a user terminal device 10 and a server terminal device 20. The server terminal device 20 and the user terminal device 10 are connected via a network, such as a wired or wireless network connection.

[0047] Among them, the user terminal device 10 can be used to display the target page. In the embodiment of the present application, the user terminal device 10 can display the target page, and the target page displays at least one virtual character. The user terminal device 10 obtains multimodal input data, and the multimodal input data includes at least one of position information data, microphone data, and touch screen data. The user terminal device 10 sends the multimodal input data to the server device 20, and the server device 20 generates interactive feedback information for the target page or the virtual character according to the received multimodal input data. The server device 20 sends the generated interactive feedback information for the target page or the virtual character to the user terminal device 10, and the user terminal device 10 receives the interactive feedback information sent from the server device 20, and displays the interactive feedback information on the target page.

[0048] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0049] Please refer to Figure 2 , Figure 2 which is the first flowchart of the interaction method provided by the embodiment of the present application. This method can be applied to a client device such as Figure 1 shown, and this method may include the following steps 110 to step 140.

[0050] Step 110, display a target page through the client, and at least one virtual character is displayed on the target page;

[0051] The client is an application or interface deployed on the client device for users to access and display the target page. For example, it can be an e-commerce platform client, a game application client, etc.

[0052] The target page is a page for users to interact with virtual characters. For example, it can be a browser page, a platform page, an in-app page, a mini-program page, etc. More specifically, the target page can be an e-commerce platform activity page, the main interface of a role-playing game, a chat page of a social media application containing the user's virtual image, the main activity page corresponding to a public welfare stray cat rescue activity containing the stray cat to be rescued, etc.

[0053] The virtual character is a virtual person or animal displayed in the form of an image or animation on the target page. For example, it can be a game character, a virtual assistant, a virtual animal character, etc. More specifically, the virtual character can be a virtual kitten, and the virtual character can interact with the user.

[0054] Please refer to Figure 3 , Figure 3 which is the page schematic diagram of the target page provided by the embodiment of the present application. Figure 3 Taking the target page as the public welfare rescue page corresponding to the public welfare stray cat rescue activity containing the stray cat to be rescued as an example, the virtual character is the stray cat to be rescued.

[0055] In some embodiments, when displaying the target page through the client, the following steps 1101 to step 1104 are further included:

[0056] Step 1101, obtain a login request for the target page, and the login request includes user information and device information corresponding to the client;

[0057] In some embodiments, the user information may include information such as a username and a user email, and the device information may include information such as a device model, an operating system version, and a hardware specification.

[0058] Step 1102: Send the login request to the server so that the server can determine whether the client meets the participation conditions for the event based on the login request.

[0059] In some embodiments, the participation conditions include that the current user corresponding to the client indicated by the user information exists in the whitelist and the device information meets the device preconditions. The device preconditions include that the touch screen of the client supports the pressure sensing function and the multi-touch function, and the client has at least a controllable linear vibration motor, a gravity sensor, a distance sensor, and a speaker.

[0060] Specifically, the server parses the user information and the device information from the login request. The server first queries the whitelist database according to the identifier information (such as the username, user email, etc.) that uniquely indicates the user identity in the user information. If the user is in the whitelist database, the server further determines whether the touch screen of the client supports the pressure sensing function and the multi-touch function according to the device information. If it supports, the server further determines whether the client has a controllable linear vibration motor, a gravity sensor, a distance sensor, and a speaker according to the device information. If all are available, it is determined that the client meets the participation conditions for the event.

[0061] In some embodiments, the device information includes the client device model information. The server maintains a device database that contains various device models and their corresponding hardware characteristic information. When receiving the login request, the server can parse the device model and find the corresponding hardware characteristics in the database, so as to determine whether the touch screen of the client supports the pressure sensing function and the multi-touch function and whether the client has a controllable linear vibration motor, a gravity sensor, a distance sensor, and a speaker, etc.

[0062] In other embodiments, the server can call the system-level application programming interface (API) (such as Android's PackageManager or iOS's UIDevice, etc.) according to the device information to obtain the hardware information of the client device, so as to determine whether the device information meets the device preconditions. Among them, the system-level API is an interface for querying and verifying the hardware functions and characteristics of the client device. The system-level API is usually provided by the operating system (such as Android, iOS, etc.) or specific hardware manufacturers, allowing the server to query the hardware specifications, function support conditions, etc. of the device.

[0063] Step 1103: Receive the login request response information sent by the server when it determines that the client meets the participation conditions for the event.

[0064] If the client meets the participation conditions of the event, the server generates a login request response message and returns the login request response message to the client via HTTP or WebSocket.

[0065] Step 1104, display a virtual interaction interface and operation guidance information corresponding to the virtual interaction interface on the target page according to the login request response message. The virtual interaction interface is used for the current user corresponding to the client to perform virtual interaction with the virtual character.

[0066] Specifically, the login request response message may include event information. The client displays operation guidance information (such as text description, icon hint, etc.) corresponding to the virtual interaction interface on the target page according to the login request response message. Among them, the virtual interaction interface is an interface for the user to enter the target page. The virtual interaction interface can be, for example, a button, an input box, etc. The operation guidance information corresponding to the virtual interaction interface can be a text description, an icon hint, etc.

[0067] In a specific embodiment provided by the present application, the target page may be a public welfare assistance page, and the virtual character may be a virtual image of an animal in need of assistance, such as a virtual kitten. The client displays a "start assistance" button on the target page according to the login request response message. The client responds to the click operation of the user on the "start assistance" button and displays the target page.

[0068] Step 120, obtain multimodal input data, where the multimodal input data includes at least one of location information data, microphone data, and touch screen data;

[0069] Among them, the location information data may include the geographical location information of the user (or the client device) or the movement information in space, etc. The microphone data is the audio information collected by the microphone of the client device, such as the sound wave intensity when the user blows. The touch screen data is the operation data when the user operates on the touch screen of the client, such as click strength, click times, slide strength, slide duration, etc.

[0070] In this embodiment, the server can obtain multimodal input data to implement multimodal interaction between the client and the target page.

[0071] Step 130, send the multimodal input data to the server so that the server generates interaction feedback information for the target page or the virtual character according to the multimodal input data;

[0072] After receiving the multimodal input data, the server analyzes and processes the multimodal input data to generate corresponding interaction feedback information.

[0073] Step 140, receive the interaction feedback information sent from the server and display the interaction feedback information on the target page.

[0074] After the client receives the interaction feedback information sent from the server, it displays the interaction feedback information on the target page according to the linear vibration motor, gravity sensor, distance sensor and speaker of the client device.

[0075] In some embodiments, the multimodal input data includes location information data. Obtaining the multimodal input data includes: obtaining the location information data of the client.

[0076] In some embodiments, obtaining the location information data of the client includes:

[0077] Obtain a first interaction mode instruction for participating in the activity input through the target page, and send the first interaction mode instruction to the server. The first interaction mode instruction represents an activation instruction for the client to participate in the activity;

[0078] Receive a first permission request for obtaining location information data sent by the server in response to the first interaction mode instruction, and display a first permission acquisition pop-up window on the target page;

[0079] In response to the first authorization information input through the first permission acquisition pop-up window, obtain the location information data of the client.

[0080] The activities in the embodiments of the present application can be various network online activities, such as online public welfare activities, online shopping activities, online social activities, etc. More specifically, for example, it can be a public welfare stray cat rescue activity.

[0081] Specifically, the user inputs a first interaction mode instruction for participating in the activity through the target page. The first interaction mode instruction can be a button click, form submission or other forms of user interaction, indicating that the user hopes to participate in the activity. The client sends the first interaction mode instruction to the server so that the server can confirm that the user has triggered the activity participation process. Further, after receiving the first interaction mode instruction, the server sends a first permission request for obtaining the location information data permission to the client. After receiving the first permission request, the client will display a first permission acquisition pop-up window on the target page, prompting the user to authorize the acquisition of location information data. The first permission acquisition pop-up window can include detailed information of the first permission request, buttons for the user to agree or refuse, etc. By reading the first permission acquisition pop-up window, the user can choose to agree or refuse to authorize. If the user chooses to agree, the first authorization information will be input (usually by clicking the "agree" button or performing other forms of confirmation). After the user agrees to authorize, the client will start collecting the user's location information data.

[0082] In this embodiment, it can ensure that the client can obtain the location information data of the user safely and compliantly, providing an important basis for the subsequent processing and analysis of the activity. At the same time, it also ensures the legality and security of the user data.

[0083] In some embodiments, the multi-modal input data includes location information data, the interaction feedback information includes target activity information, receiving the interaction feedback information sent from the server, and displaying the interaction feedback information on the target page, including: receiving the target activity information sent from the server, the target activity information being determined by the server according to the location information data of the user terminal, the target activity information at least including the activity location, activity content, and participation method of the target activity; displaying the target activity information on the target page.

[0084] Among them, the location information data may specifically be the city information where the user (or the user terminal device) is located, and different location information data corresponds to different target activity information.

[0085] For example, the target activity may be a public welfare stray cat rescue activity, and the target activity information may include the public welfare stray cat rescue process, the cities participating in the activity, etc.

[0086] Please refer to Figure 4 , Figure 4 which is a schematic diagram of the target page including the target activity information provided by the embodiment of the present application. As Figure 4 shown, the activity location "City A" of the target activity is displayed on the page, and the activity content may include: qualification acquisition, pet food donation and distribution, achievement publicity, etc. The activity participation method is to invite friends to help and receive badges, and the participation process may also be included on the page.

[0087] In some embodiments, the interaction feedback information further includes a city heat map. Displaying the interaction feedback information on the target page includes:

[0088] Displaying the city heat map on the target page, and marking the lit city logo of the city where the activity location of the target activity is located on the city heat map. The city heat map is used to reflect the participation activity of each city participating in the activity through the target platform in the current cycle;

[0089] Displaying the activity content and participation method of the target activity through the target page.

[0090] In some embodiments, the upper half of the target page is used to display the city heat map. The number of people participating in the activity in the city where the activity location of the target activity is located may also be displayed on the city heat map. The higher the brightness of the city heat map, the higher the participation activity of the city where the activity location is located. The target activity is periodic, and each cycle is independent of each other.

[0091] Taking the target activity as a public welfare stray cat rescue activity as an example, as Figure 4As shown, the activity content of the target activity can be receiving city badges, participating in the distribution of public welfare pet food, uploading rescue photos for publicizing the list, etc.; the participation method can be inviting friends to help to receive city badges, and after obtaining the city badges, one can participate in the public welfare stray cat rescue activity.

[0092] In some embodiments, the city heat map is implemented by the server through the following steps 101 to 104:

[0093] Step 101, obtain the social proximity among the user terminals participating in the activity in the current cycle;

[0094] In some embodiments, the social proximity among the user terminals participating in the activity in the current cycle can be determined according to the number of common interest points of the user terminals participating in the activity in the social network. The common interest points can include jointly participated projects, jointly liked, commented and shared projects, etc.

[0095] Specifically, the following formula 1 can be used to calculate the social proximity D between user terminals:

[0096] Formula 1: D = 1 / (1 + e (k*(common_interests-threshold)) )

[0097] Where k is a preset influence factor, common_interests is the number of common interest points between user terminals, and threshold is a preset interest point threshold.

[0098] For example, "between user terminals" refers to the connection or relationship between users who participate in a specific activity through their respective user terminal devices within the current cycle. These user terminal devices can be any device capable of accessing the Internet and participating in social activities, such as smartphones, tablets, personal computers, etc. "Between user terminals" not only refers to the connection between devices, but more importantly, the social relationship and interaction between users participating in the activity through these devices.

[0099] Step 102, obtain the user terminal attribute values of each user terminal participating in the activity in the current cycle;

[0100] In some embodiments, the user terminal attributes include but are not limited to: the frequency f of the user terminal participating in activities of the same type as the target activity, the number of times p of the user terminal participating in activities of the same type as the target activity, the consumption amount d of the user terminal participating in activities of the same type as the target activity, the activity level u of the user terminal participating in activities of the same type as the target activity, etc.

[0101] More specifically, for example, if the target activity is a public welfare stray cat rescue activity, the user terminal attributes include but are not limited to: the frequency of the user terminal participating in public welfare activities, the total amount donated by the user terminal to public welfare activity projects, the total number of times the user terminal participates in public welfare activities, the consumption level of the user on the public welfare rescue platform, etc.

[0102] In some embodiments, for each client, the client attribute of the client may be determined according to the contribution willingness behavior data of the activities of the same type as the target activity of the client.

[0103] In some embodiments, the client attribute value A may be determined using the following formula 2:

[0104] Formula 2:

[0105] where w j is the weight of the j-th client attribute, and w j can be reasonably adjusted according to business requirements, and attribute j is the attribute value of the j-th client attribute of the client, and the attribute value can be reasonably configured.

[0106] Step 103: Calculate the current client participation heat index of each city participating in the activity in the current cycle according to the number of clients participating in the activity in the current cycle, the social proximity between clients, the client attribute value, and the behavior influence strength, where the behavior influence strength is obtained by conversion according to the historical client participation heat index of each city participating in the activity in the previous cycle;

[0107] In some embodiments, the following formula 3 may be used to determine the current client participation heat index HI of each city city :

[0108] Formula 3: HI city = w1 * user i + w2 * D + w3 * A + w4 * impact i ;

[0109] where w1, w2, w3, and w4 are weight coefficients, which can be adjusted according to current business needs, user i represents the number of clients participating in the activity in the current cycle, D represents the social proximity between clients, A represents the client attribute value, and impact i represents the behavior influence strength, and impact i can specifically be obtained by multiplying the historical client participation heat index of each city participating in the activity in the previous cycle by a preset heat coefficient.

[0110] Step 104: Determine the city heat map for display on the target page according to the current client participation heat index of each city.

[0111] In some embodiments, a city heat map for display on a target page is determined according to the current client participation heat index of each city, including: mapping the current client participation heat index of each city to the same interval (such as mapping to the interval of 0-1), and determining the city heat map for display on the target page according to the current client participation heat index of each city after the mapping process.

[0112] In some embodiments, the following formula 4 can be used to perform a mapping process on the current client participation heat index of each city to obtain the current client participation heat index of each city after the mapping process:

[0113] Formula 4: HI mapped =(HI city -min(HI)) / (max(HI)-min(HI));

[0114] where min(HI) and max(HI) are respectively the minimum value and the maximum value of the current client participation heat index of all cities.

[0115] In some embodiments, the server is further configured to:

[0116] set a corresponding basic exchange value for each client;

[0117] calculate the dynamic exchange value of each client according to the current client participation heat index of each city;

[0118] update the integral value of each client according to the dynamic exchange value to obtain the updated integral value of each client, and send the updated integral value of each client to the target page of each client for display.

[0119] Specifically, the server first sets a basic exchange value V for each client base .

[0120] This basic exchange value can be determined after comprehensive evaluation according to the user's registration information (such as region, age, gender, etc.), historical behavior data (such as purchase records, browsing records, etc.) and the platform's operation strategy. For example, for newly registered users, a relatively high basic exchange value can be set to encourage their participation in platform activities; for active users, the basic exchange value can be dynamically adjusted according to their historical behavior data.

[0121] The server calculates the dynamic exchange value V of each client according to the current client participation heat index HI city , user .

[0122] In some embodiments, the server can calculate the dynamic redemption value V of each client using the following formula 5 user

[0123] Formula 5: V user = V base *(1 + λ(HI city - HI median ));

[0124] where λ is a coefficient that can be freely set, and HI median is the median of the current client participation heat index for all cities.

[0125] According to the calculated dynamic redemption value, the server updates the integral value of each client, and the server sends the updated integral value of each client to the target page of each client for display.

[0126] In some embodiments, the form of the integral value can be diversified and can be designed as various symbolic or interesting symbols. For example, in the embodiments of the present application, the integral can be in the form of a spark.

[0127] In some embodiments, the method further includes:

[0128] Obtain the real-scene marker image of the current user of the client placing a service marker at the activity location, and send the real-scene marker image to the server, so that the server associates the real-scene marker image with the city heat map to obtain an updated city heat map; display the updated city heat map received from the server, and display a highlighted marker effect at the map coordinates corresponding to the activity location in the updated city heat map.

[0129] Specifically, the client displays a prompt message to prompt the current user that a virtual public welfare marker can be placed at the current activity location. After the user confirms to place the public welfare marker, the client captures the real-scene marker image of the current activity location, and the client sends the captured real-scene marker image and the current activity location of the current user to the server. After receiving the real-scene marker image and the current activity location of the current user, the server associates the real-scene marker image with the city heat map according to the current activity location of the current user and updates the heat value of the current activity location. After receiving the updated city heat map, the client parses and displays the updated city heat map. In the updated city heat map, the map coordinates corresponding to the location where the user places the public welfare marker will display a highlighted marker effect to reflect the public welfare participation degree and increase the city heat value.

[0130] In some embodiments, the interactive feedback information further includes lighting feedback information. Displaying the interactive feedback information on the target page includes: displaying the lighting feedback information on the target page, where the lighting feedback information indicates that the lighting city identifier of the city where the activity location of the target activity is marked on the city heat map has been successfully marked.

[0131] Please refer to Figure 5 , Figure 5 which is a schematic diagram of a page including lighting feedback information provided by an embodiment of the present application. For example, if the city where the activity location is located is City A, the lighting feedback information can be: "Congratulations on lighting the xxxth guardian in the dim light of City A".

[0132] In some embodiments, the interactive feedback information includes at least one of interactive virtual animations of virtual characters, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, and vibration feedback information; the interactive feedback information is determined by the server according to multi-modal input data to predict the user's intention and the user's emotional intensity, and according to the user's intention and the user's emotional intensity to determine the type of user interaction behavior, and according to the type of user interaction behavior.

[0133] Among them, the interactive virtual animation of the virtual character can be a dynamic reaction made by the virtual character, such as an action. The tactile feedback information can be physical vibration or pressure change information of the user terminal device, used to convey information or feelings to the user. The temperature feedback information can be information that changes the temperature of the surface of the user terminal device, used to convey information or feelings to the user. The ambient light change information can be information that changes the light conditions around the user terminal device or on the user's touch screen, used to convey information or create an atmosphere. The audio feedback information can be sounds (such as beeps, music, voices, etc.), used to convey information or feedback to the user. The character expression change information can be different expression information shown by the virtual character. The vibration feedback information can be physical vibration used to feedback the user's emotional state.

[0134] In some embodiments, obtaining multi-modal input data includes:

[0135] Obtaining a second interaction mode instruction input through a virtual interaction interface and sending the second interaction mode instruction to the server, where the second interaction mode instruction represents an opening instruction for the current user corresponding to the user terminal to perform virtual interaction with the virtual character;

[0136] Obtaining a second permission request for obtaining multi-modal input data sent by the server in response to the second interaction mode instruction, and displaying a second permission acquisition pop-up window on the target page;

[0137] In response to the second authorization information input through the second permission acquisition pop-up window, obtaining multi-modal input data;

[0138] Among them, the multi-modal input data at least includes microphone data and touch screen data;

[0139] The microphone data includes the sound signal generated when the current user blows air collected by the microphone of the user terminal at a preset sampling frequency;

[0140] The touch screen data includes the touch force corresponding to the touch event generated when the current user acts on the touch screen captured by the sensor of the touch screen of the user terminal.

[0141] Specifically, the user inputs a second interaction mode instruction for virtual interaction with the virtual character through the virtual interaction interface. The second interaction mode instruction can be a button click, form submission, or other forms of user interaction, indicating that the user hopes to have virtual interaction with the virtual character. The user terminal sends the second interaction mode instruction to the server so that the server can confirm that the user has triggered virtual interaction with the virtual character. Further, after receiving the second interaction mode instruction, the server sends a second permission request for obtaining the multi-modal input data permission to the user terminal. After receiving the second permission request, the user terminal will display a second permission acquisition pop-up window on the target page, prompting the user to authorize the acquisition of multi-modal input data. The second permission acquisition pop-up window can include detailed information of the second permission request, buttons for the user to agree or refuse, etc. By reading the second permission acquisition pop-up window, the user can choose to agree or refuse to authorize. If the user chooses to agree, the second authorization information will be input (usually by clicking the "agree" button or performing other forms of confirmation). After the user agrees to authorize, the user terminal will start collecting the user's multi-modal input data.

[0142] In some embodiments, the server generates interaction feedback information for the target page or the virtual character according to the multi-modal input data, including the following steps 1 to step 3:

[0143] Step 1, the server predicts the user's intention and the user's emotional intensity according to the multi-modal input data;

[0144] In some embodiments, the multi-modal input data at least includes microphone data and touch screen data; predicting the user's intention and the user's emotional intensity according to the multi-modal input data includes the following steps 11 to step 13:

[0145] Step 11, perform data processing on the microphone data to obtain the blowing intensity of the current user corresponding to the user terminal;

[0146] In some embodiments, the microphone data includes the sound signal generated when the current user blows air collected by the microphone of the user terminal at a preset sampling frequency. Among them, the preset sampling frequency can be reasonably set according to the actual application scenario. Preferably, the preset sampling rate can be 44.1 kHz;

[0147] The server processes the microphone data to obtain the blowing intensity of the current user corresponding to the client, including the following steps 111 to 114:

[0148] Step 111: Divide the sound signal into multiple sound frames according to a preset length, and apply a Hamming window function to each sound frame for window function processing to obtain the sound frame after window function processing;

[0149] Among them, the preset length can be reasonably set according to the application scenario. Preferably, the preset length can be set to 256 sample points or 512 sample points.

[0150] In some embodiments, the Hamming window function can be represented by the following formula 6:

[0151] Formula 6:

[0152] Among them, α is the coefficient of the Hamming window function, and its value range is [0.46, 0.54]. Preferably, α can take 0.54, n is the sample serial number within each sound frame, and N is the length of the sound frame.

[0153] Each sound frame can be processed by the window function through the following formula 7:

[0154] Formula 7: X windowed [n] = w[n] * x[n];

[0155] Among them, x[n] is the sound frame, and X windowed [n] is the sound frame after window function processing.

[0156] Step 112: Perform a fast Fourier transform on the sound frame after window function processing to obtain the spectral components corresponding to each sound frame, and calculate the complex power of each spectral component;

[0157] In some embodiments, the complex power X[k] of each sound frame after window function processing can be calculated through the following formula 8:

[0158] Formula 8:

[0159] Step 113: Calculate the power spectral density of each spectral component according to the complex power of each spectral component and the total number of frequency components;

[0160] In some embodiments, the power spectral density PSD(f) of each spectral component is calculated through the following formula 9:

[0161] Formula 9:

[0162] Step 114: Determine the spectral component with the highest calculated power spectral density as the blowing intensity of the current user corresponding to the user terminal, and perform normalization processing on the blowing intensity to obtain the normalized blowing intensity.

[0163] In some embodiments, the spectral component B with the highest power spectral density is determined by the following formula 10 peak :

[0164] Formula 10: B peak = max(PSD);

[0165] When performing normalization processing on the blowing intensity, linear normalization, general normalization, Z-score standardization, or any other normalization method can be used.

[0166] In a specific embodiment provided by the present application, the linear normalization method is used to perform normalization processing on the blowing intensity. Specifically, the blowing intensity can be normalized through the following formula 11 to obtain the normalized blowing intensity B nrom :

[0167] Formula 11:

[0168] Step 12: Perform data processing on the touch screen data to obtain the touch force, touch duration, and touch position change rate of the current user corresponding to the user terminal;

[0169] In some embodiments, the touch screen data includes touch signals corresponding to touch events generated when the current user acts on the touch screen through the sensors of the touch screen of the user terminal; performing data processing on the touch screen data to obtain the touch force, touch duration, and touch position change rate of the current user corresponding to the user terminal includes the following steps 121 to 123:

[0170] Step 121: Determine the touch force of the current user corresponding to the user terminal according to the touch signal, and perform normalization processing on the touch force to obtain the normalized touch force;

[0171] During the period when the user's finger remains in contact with the touch screen of the user terminal device, the sensors of the touch screen will continuously detect the pressure, and the user terminal will continuously record the force magnitude of each touch point, and use all the touch force data points during this period as this touch event.

[0172] In some embodiments, the touch force includes the maximum force T max and the average force T avg , and the normalized touch force includes the normalized maximum force T max,nrom and the normalized average force T avg,nrom .

[0173] Among them, the maximum force T max is the maximum force point in a single touch event, and the average force T avg is the average value of all force points in a single touch event. The average force T can be determined by the following formula 12 avg :

[0174]

[0175] where m is the total number of touch force data points in this touch event, and T i is the i-th touch force data point.

[0176] When normalizing the maximum force T max and the average force T avg in the touch force, linear normalization, general normalization, Z-score standardization, or any other normalization method can be used.

[0177] In a specific embodiment provided by this application, the linear normalization method is used to normalize the maximum force T max and the average force T avg in the touch force. Specifically, the maximum force T max in the touch force can be normalized through the following formula 13 to obtain the normalized maximum force T max,nrom , and the average force T avg in the touch force can be normalized through the following formula 14 to obtain the normalized average force T avg,nrom :

[0178] Formula 13:

[0179] Formula 14:

[0180] where min(T max ) is the minimum value of all maximum forces in all touch events corresponding to all client terminals, max(T max ) is the maximum value of all maximum forces in all touch events corresponding to all client terminals, min(T avg ) is the minimum value of all average forces in all touch events corresponding to all client terminals, and max(T avg ) is the maximum value of all average forces in all touch events corresponding to all client terminals.

[0181] In this embodiment, the touch force is normalized to obtain the normalized touch force, which can eliminate the differences between different client devices and different users, thereby improving the accuracy and consistency of touch force detection.

[0182] Step 122: Calculate the touch duration of the current user corresponding to the client based on the touch start timestamp and touch end timestamp of the touch event, and normalize the touch duration to obtain the normalized touch duration.

[0183] In some embodiments, in a single touch event, the moment when the pressure detected by the sensor of the touch screen exceeds the first preset pressure threshold is determined as the touch start timestamp T of this touch event. start The moment when the pressure detected by the sensor of the touch screen is less than the second preset pressure threshold is determined as the touch end timestamp T of this touch event. end Among them, the specific values of the first preset pressure threshold and the second preset pressure threshold can be reasonably set according to the type of the touch screen, the application scenario, and the user's habits, and this application does not limit this.

[0184] In some embodiments, in this interaction, the current user may trigger multiple touch events. For each touch event, subtract the touch start timestamp T of each touch event. end from the touch end timestamp T start to obtain the touch duration of each touch event. Divide the sum of the touch durations of all touch events in this interaction by the total number of touch events to obtain the average duration D of a single touch event in this interaction. avg .

[0185] When normalizing the touch duration, linear normalization, general normalization, Z-score normalization, or any other normalization method can be used.

[0186] In a specific embodiment provided by this application, the linear normalization method is used to normalize the touch duration. Specifically, the touch duration can be normalized through the following formula 15 to obtain the normalized touch duration D. avg,norm :

[0187] Formula 15:

[0188] where min(D avg ) is the minimum value among all touch durations in all touch events corresponding to all client devices, and max(D avg ) is the maximum value among all touch durations in all touch events corresponding to all client devices.

[0189] Step 123: Determine the touch position change rate of the current user corresponding to the client based on the distance change amount and time difference between adjacent touch points among the continuous touch points of the touch event, and perform normalization processing on the touch position change rate to obtain the normalized touch position change rate. Among them, the distance change amount between adjacent touch points is obtained by calculating the Euclidean distance between adjacent touch points according to the effective position coordinates of the continuous touch points.

[0190] Specifically, when the user interacts with the virtual character through the target page, the sensor of the touch screen will capture the user's continuous touch point records, which are recorded as a series of continuous two-dimensional coordinate points: P 1 (x 1 ,y 1 ), P 2 (x 2 ,y 2 )... P n (x n ,y n ).

[0191] In some embodiments, the distance change amount between the adjacent i-th touch point P i and the (i + 1)-th touch point P i+1 can be calculated by the following formula 16:

[0192] Formula 16:

[0193] In some embodiments, the time difference between the touch time point t i corresponding to the i-th touch point P i and the touch time point t i+1 corresponding to the (i + 1)-th touch point P i+1 is denoted as Δt i , Δt i = t i+1 - t i .

[0194] Then, the touch position change rate between the adjacent i-th touch point P i and the (i + 1)-th touch point P i+1 can be calculated by the following formula 17:

[0195] Formula 17:

[0196] When performing normalization processing on the touch position change rate, linear normalization, general normalization, Z-score standardization or any other normalization method can be used.

[0197] In a specific embodiment provided in the present application, a linear normalization method is used to normalize the touch position change rate. Specifically, the touch position change rate can be normalized by the following formula 18 to obtain the normalized touch position change rate V norm :

[0198] Formula 18:

[0199] Among them, min(V) is the minimum value of all touch position change rates in all touch events corresponding to all user terminals, and max(V) is the maximum value of all touch position change rates in all touch events corresponding to all user terminals.

[0200] In some embodiments, when determining the touch position change rate of the current user corresponding to the user terminal, it is determined based on the distance change and time difference between adjacent valid touch points in the continuous valid touch points of the touch event, wherein the valid touch point is a touch point located in a preset touch valid area (AA area) among all recorded touch points, and the invalid touch point is a touch point located in a preset inactive touch area (VA area) among all recorded touch points.

[0201] In some embodiments, a Kalman filter may be used to eliminate invalid touch data corresponding to invalid touch points in all touch point data.

[0202] Specifically, the Kalman filter is used to remove invalid touch data corresponding to invalid touch points in all touch point data, including:

[0203] Initialize the Kalman filter, including setting the filter parameters, and initializing the state estimation and estimation error covariance; collect all touch point data of the touch event, including the touch point coordinates, force, etc.; use the Kalman filter to predict and update the touch point data to obtain a more accurate state estimation; compare the state estimation value with the preset "touch valid area (AA area)"; if the state estimation value falls outside the "inactive touch area (VA area)", it is considered to be a valid touch point; otherwise, it is an invalid touch point; remove all invalid touch data identified as invalid touch points.

[0204] See also Figure 6 , Figure 6 A schematic diagram of a touch-enabled area and an inactive touch-enabled area provided in an embodiment of the present application.

[0205] Step 13, predicting the user intention and the user emotion intensity according to the blowing intensity, the touch force, the touch duration and the touch position change rate.

[0206] In some embodiments, the user intention and the user emotional intensity are predicted according to the blowing intensity, the touch force, the touch duration, and the touch position change rate, including the following steps 131 to 132:

[0207] Step 131, predicting the user intention according to the normalized average force T avg,nrom , the normalized touch duration D avg,norm and the normalized touch position change rate V norm ;

[0208] In some embodiments, the user intention I can be predicted by the following formula 19:

[0209] Formula 19: I = tanh(αT avg,nrom + βD avg,norm + γV norm );

[0210] where α, β, and γ are weight coefficients, and machine learning methods can be used to adjust the parameters of α, β, and γ. The tanh function limits the output result I within the range of [-1, 1] and is used to anchor different user intentions.

[0211] Step 132, predicting the user emotional intensity according to the normalized blowing intensity B nrom , the normalized average force T avg,nrom , the normalized maximum force T max,nrom and the normalized touch duration D avg,norm .

[0212] In some embodiments, the user emotional intensity AvgRange(E) can be predicted by the following formula 20 and formula 21:

[0213] Formula 20: E = sin(δB nrom + εT avg,nrom + θT max,nrom + μD avg,norm );

[0214] Formula 21:

[0215] where δ, ε, θ, and μ are weight coefficients, and machine learning methods can be used to adjust the parameters of δ, ε, θ, and μ. The sin function limits the output result E within the range of [-1, 1]. E is the user emotional intensity corresponding to a single touch event, AvgRange(E) is the average value of the user emotional intensity within a preset time period, n is the total number of touch events within the preset time period, and the average value of the user emotional intensity AvgRange(E) within the preset time period is used as the current user emotional intensity.

[0216] Step 2: The server determines the type of user interaction behavior according to the user's intention and the user's emotional intensity;

[0217] In some embodiments, the categories of user interaction behaviors include, but are not limited to, soothing, playing, feeding, etc. The server determines the type of user interaction behavior according to the user's intention and the user's emotional intensity, which may include:

[0218] If the user's intention I is less than the preset low intention threshold I low , and the user's emotional intensity AvgRange(E) is within the preset low emotional intensity range E low _ range , determine that the type of user interaction behavior is soothing;

[0219] If the user's intention I is greater than the preset high intention threshold I high , and the user's emotional intensity AvgRange(E) is within the preset medium emotional intensity range E medium _ range , determine that the type of user interaction behavior is playing;

[0220] If the user's intention I is greater than the preset low intention threshold I high , and the user's emotional intensity AvgRange(E) is within the preset high emotional intensity range E high _ range , determine that the type of user interaction behavior is feeding.

[0221] Among them, the preset low intention threshold I low , the preset high intention threshold I high , the preset low emotional intensity range E low _ range , the preset medium emotional intensity range E medium _ range , and the preset high emotional intensity range E high _ range are all greater than 0 and less than or equal to 1, and the specific values can be reasonably set according to the application scenario.

[0222] Preferably, the preset low intention threshold I low is taken as 0.3, the preset high intention threshold I high is taken as 0.7, the preset low emotional intensity range E low _ range is taken as [0.1, 0.2], the preset medium emotional intensity range E medium _ range is taken as [0.6, 0.8], and the preset high emotional intensity range E high _ range is taken as [0.8, 1.0].

[0223] Step 3: The server determines the interaction feedback information for the virtual character according to the user interaction behavior type. The interaction feedback information includes at least one of the virtual character's interactive virtual animation, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, and vibration feedback information.

[0224] In some embodiments, the server stores an interaction feedback database of the interaction feedback templates of the virtual character corresponding to each user interaction behavior type. Each interaction feedback template includes at least one of the virtual character's interactive virtual animation, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, and vibration feedback information. The server determines the corresponding interaction feedback template from the interaction feedback database according to the user interaction behavior type, and uses the queried interaction feedback template as the interaction feedback information for the virtual character.

[0225] The interaction feedback information is used to enhance the immersion and realism of the interaction between the user and the virtual character, can simulate the real scenario when the user interacts with the virtual character in reality, and provide a real sensory experience.

[0226] For example, if the user interaction behavior type is soothing, the corresponding first interaction feedback information for the virtual character includes: the first interactive virtual animation of the virtual character, the first tactile feedback, the first temperature feedback parameter, and the first ambient light change parameter. Among them, the first interactive virtual animation of the virtual character can be to show the behavior of the virtual character being quiet and relaxed, such as closing its eyes or making a purring sound. The first tactile feedback simulates the touch when the user strokes a cat in real life. The first temperature feedback parameter simulates the temperature when the user strokes a cat in real life, such as slightly increasing the temperature of the touch screen of the user device. The first ambient light change parameter simulates the warm brightness and color tone of the touch screen of the user device, such as the color tone of the touch screen of the user device becoming soft and warm.

[0227] For another example, if the type of user interaction behavior is playing, the corresponding second interaction feedback information for the virtual character includes: the second interactive virtual animation of the virtual character: showing the lively and curious behaviors of the virtual character, such as chasing virtual toys, jumping or rolling; the first character expression change information: the facial expression of the virtual character should change with the user's interaction, such as twinkling eyes, rotating ears, rising corners of the mouth, etc., to show its joy and excitement and enhance the authenticity of the interaction; the second tactile feedback: simulating the touch feeling when the user plays with the virtual character in real life, for example, short vibrations; the second temperature feedback parameter: simulating the temperature when the user plays with the virtual character in real life, such as slightly increasing the temperature of the touch screen of the user device; the second ambient light change parameter: the ambient light can change with the mood of the virtual character, such as changing to a more active and bright color to create a cheerful atmosphere; the first audio feedback: used to play sound effects related to playing, such as cheerful cries, toy noises or background music to enhance the user's auditory experience.

[0228] For another example, if the type of user interaction behavior is feeding, the corresponding third interaction feedback information for the virtual character includes: the third interactive virtual animation of the virtual character: showing the animation of the virtual character eating, including actions such as chewing and swallowing, as well as wagging the tail or a satisfied expression. The animation should be delicate and realistic and synchronized with the user's feeding action; the third tactile feedback: when the user performs a specific feeding action, such as swiping the screen to send food to the mouth of the virtual cat, the device generates corresponding vibrations through a linear vibration motor to simulate the touch feeling during feeding; the first vibration feedback: adjusting the intensity and duration of the vibration according to the user's feeding action and emotional state. For example, when the user feeds gently, the vibration should be gentle and short; while when the user feeds continuously, the vibration can last and have a slight rhythm.

[0229] In some embodiments, the method further includes:

[0230] Obtaining the interaction behavior data corresponding to the interaction feedback information and sending the interaction behavior data to the server, so that the server generates the activity participation information of the user device for the target activity according to the interaction behavior data, and determines the activity progress feedback information for display on the target page according to the activity participation information;

[0231] Receiving the activity progress feedback information sent from the server and displaying the activity progress feedback information on the target page. The activity progress feedback information includes at least one of an integral progress bar, an activity effect diagram corresponding to the activity object, and an interactive virtual animation change diagram corresponding to the virtual character. The virtual character is the virtual character corresponding to the activity object of the target activity.

[0232] Among them, the interaction behavior data corresponding to the interaction feedback information may refer to the interaction data between the user and the virtual character. The interaction behavior data reflects the user's activity participation degree for the target activity. The more interactions the user has with the virtual character, the higher the activity participation degree.

[0233] The integral progress bar includes the integral accumulation situation of the user side and the progress of the integral required for the next item exchange. The items exchanged by the user side can be used to feed the activity object. The activity effect diagram corresponding to the activity object is the feeding effect after the activity object obtains the items fed by the user. The interactive virtual animation change diagram corresponding to the virtual character is the different display effects when the virtual character has different integral progress bars. The integral accumulation mechanism and the reaction mechanism between the virtual character and the activity object can be customized and adjusted according to business requirements to ensure that each positive behavior of user interaction can obtain timely integral accumulation and corresponding feedback, thereby enhancing the user's sense of participation and satisfaction.

[0234] In this embodiment, determining the activity progress feedback information according to the interaction behavior data of the user side, that is, visualizing the user's interaction behavior, can provide a more intuitive and interesting interaction experience for the user, and can greatly improve the user experience and participation degree.

[0235] Taking the target activity as a public welfare stray cat rescue activity and the virtual character as a stray cat as an example, the user interacts with the stray cat through the target page. The user side can obtain the interaction behavior data between the user and the virtual character, send the interaction positioning data to the server side. The server side generates the activity progress feedback information of the user side for the public welfare stray cat rescue activity according to the received interaction behavior data, and sends the activity progress feedback information to the user side. The activity progress feedback information includes the integral progress bar determined according to the interaction behavior data (the more interactions between the interaction behavior data indicates the user side and the virtual character, the more integral), the rescue effect diagram corresponding to the real stray cat after the user uses the integral to exchange for cat food to feed the stray cat, and the effect diagram corresponding to the virtual stray cat determined according to the integral (the more integral, the better the living environment and health condition of the virtual stray cat).

[0236] In some embodiments, the method further includes:

[0237] Obtain the real interaction sound collected when the current user corresponding to the user side interacts with the activity object at the activity location, and send the real interaction sound to the server side, so that the server side associates the real interaction sound with the virtual sound of the interactive virtual animation of the virtual character to obtain the updated virtual sound;

[0238] Output the updated virtual sound received from the server side.

[0239] For example, the server can also process the received real interactive sound. For example, it can process the user's voice in the real interactive sound and associate the processed real interactive sound with the virtual sound of the interactive virtual animation of the virtual character to obtain an updated virtual sound.

[0240] Specifically, the target activity allows users to exchange items with points, and the activity platform will deliver the items exchanged by the users to the addresses specified by the users. Users can use the exchanged items to interact with the activity object offline. The activity location can be the offline real location where the user uses the item on the activity object. During the process of using the activity object, the user terminal can collect the real interactive sound when the current user interacts with the activity object at the activity location and send the real interactive sound to the server. After receiving the real interactive sound, the server processes the user's voice in the real interactive sound, then associates the processed real interactive sound with the virtual sound of the interactive virtual animation of the virtual character corresponding to the activity object to obtain an updated virtual sound, and sends the updated virtual sound to the user terminal. The user terminal outputs the virtual sound received from the server.

[0241] Taking the target activity as a public welfare stray cat rescue activity and the virtual character as a stray cat as an example, users can exchange points for physical cat food. The public welfare platform to which the activity belongs will deliver the physical cat food to the address specified by the user. Users can use the physical cat food to feed the stray cat offline. At this time, the user terminal collects the real interactive sound when the stray cat eats food or interacts with the user and sends the real interactive sound to the server. After receiving the real interactive sound, the server processes the user's voice in the real interactive sound, then associates the processed real interactive sound with the virtual sound of the interactive virtual animation of the virtual stray cat corresponding to the stray cat to obtain an updated virtual sound, and sends the updated virtual sound to the user terminal. The user terminal outputs the virtual sound received from the server.

[0242] In this embodiment, a way for users to experience the target activity offline is provided, and the offline interaction between the user and the activity object can be tracked and presented online, enhancing the user experience.

[0243] In some embodiments, the method further includes:

[0244] Sending the real interactive sound to the server so that the server generates a voice association response animation of the virtual character according to the real interactive sound, and receiving the voice association response animation of the virtual character sent by the server and displaying the voice association response animation on the target page; or

[0245] Send the real interactive sound to the server so that the server generates an updated integral value for the client based on the real interactive sound, and receive the updated integral value sent by the server and display the updated integral value on the target page.

[0246] Specifically, after receiving the real interactive sound, the server can generate a voice-associated response animation of the virtual character adapted to the real interactive sound, so as to realize the dynamic response of the virtual character to the real interactive sound.

[0247] Taking the target activity as a public welfare stray cat rescue activity and the virtual character as a stray cat as an example, after the server receives the real interactive sound when the stray cat eats food or interacts with the user collected by the client, it generates a specific animation of the virtual stray cat adapted to the real interactive sound, such as the virtual stray cat making a grateful gesture and making a satisfied expression.

[0248] Alternatively, the server can increase the cumulative integral of the client according to the real interactive sound, so as to obtain the updated integral value of the client, and send the updated integral value to the server, and the server displays the updated integral value on the target page.

[0249] In some embodiments, the method further includes:

[0250] Receive the achievement sharing information sent by the server and display the achievement sharing information on the target page. The achievement sharing information is generated by the server according to the interaction behavior data and real interactive sound collected by the client, as well as other interaction behavior data and other real interactive sound collected by other clients;

[0251] Among them, the achievement sharing information includes a sound gallery, a map and achievement badges; the sound gallery is used to display the real interactive sounds collected by the client and other real interactive sounds collected by other clients; the map is used to display a highlighted marking effect at the map coordinates corresponding to the activity location in the updated city heat map; the achievement badges are used to display the badges obtained by the client.

[0252] The user can listen to the sounds in the sound gallery by clicking or swiping.

[0253] The map displays a highlighted marking effect at the map coordinates corresponding to the activity location according to the activity location. These highlighted markings not only represent the actual locations of the public welfare activities, but also witness the user's public welfare journey. The user can view more details through operations such as zooming in and dragging, and explore the public welfare stories behind each highlighted marking.

[0254] In the "Achievement Badge" area of the target page, the user terminal displays the achievement badges obtained by the user based on public welfare behaviors. These badges include, but are not limited to, "Voice Guardian" and "City Guardian", which represent the user's contributions and achievements in public welfare activities. The user can add these badges to their personal profile to showcase their public welfare spirit and achievements.

[0255] Any combination of the above technical solutions can form optional embodiments of this application, which will not be elaborated one by one here.

[0256] In the embodiments of this application, the target page is displayed through the user terminal, and at least one virtual character is displayed on the target page; multi-modal input data is obtained, and the multi-modal input data includes at least one of position information data, microphone data, and touch screen data; the multi-modal input data is sent to the server so that the server generates interaction feedback information for the target page or the virtual character according to the multi-modal input data; the interaction feedback information sent from the server is received and the interaction feedback information is displayed on the target page. In the embodiments of this application, by collecting multi-modal input data such as position information data, microphone data, and touch screen data, richer interaction intentions and behaviors of the user can be captured, enhancing the depth and breadth of the user experience. In addition, the multi-modal input data is sent to the server for processing, and the server can quickly generate interaction feedback information for the target page or the virtual character according to these data. This mechanism of instant processing and feedback significantly improves the efficiency of communication, enabling the user to obtain an immediate response during the interaction process and enhancing the authenticity and interactivity of the interaction.

[0257] Corresponding to the above method embodiments, the embodiments of the present invention also provide an interaction method applied to a server device. The following described interaction method applied to a server device can be mutually corresponding and referred to with the interaction method applied to a client device described above.

[0258] Please refer to Figure 7 , Figure 7 which is the second process schematic diagram of the interaction method provided by the embodiments of this application. This method is applied to a server device as shown in Figure 1 and this method may include the following steps 210 to step 230.

[0259] Step 210, receive the multi-modal input data sent from the user terminal, and the multi-modal input data includes at least one of position information data, microphone data, and touch screen data;

[0260] Step 220, generate interaction feedback information for the target page or the virtual character according to the multi-modal input data, where the target page is the page displayed on the user terminal, and at least one virtual character is displayed on the target page;

[0261] Step 230: Send the interactive feedback information to the user terminal so that the user terminal can display the interactive feedback information on the target page.

[0262] In some embodiments, the multimodal input data includes the location information data of the user terminal, and the interactive feedback information includes the target activity information. Generating the interactive feedback information corresponding to the target page or the virtual character according to the multimodal input data includes:

[0263] The target activity information filtered from the activity database corresponding to the target platform according to the location information data of the user terminal and related to the location information data;

[0264] Wherein, the target activity information at least includes the activity location, activity content, participation method and virtual interaction interface of the target activity. The virtual interaction interface is used for the user terminal to perform virtual interaction with the virtual character corresponding to the activity object of the target activity;

[0265] The activity database stores the activity location, activity content, participation method and interaction requirements of all activities held by the target platform in the current cycle. The target platform is the platform to which the target page belongs.

[0266] For example, if the target activity is a public welfare stray cat rescue activity, the target platform can be the public welfare platform corresponding to the public welfare stray cat rescue activity.

[0267] In some embodiments, the target activity information further includes a city heat map, which is used to reflect the participation activity of each city participating in the activity through the target platform in the current cycle; The method further includes:

[0268] Obtain the social proximity among the user terminals participating in the activity in the current cycle;

[0269] Obtain the user terminal attribute values of each user terminal participating in the activity in the current cycle;

[0270] Calculate the current user terminal participation heat index of each city participating in the activity in the current cycle according to the number of user terminals participating in the activity in the current cycle, the social proximity among the user terminals, the user terminal attribute values and the behavior influence strength. Wherein, the behavior influence strength is obtained by converting the historical user terminal participation heat index of each city participating in the activity in the previous cycle;

[0271] Determine the city heat map to be displayed on the target page according to the current user terminal participation heat index of each city.

[0272] In some embodiments, obtaining the social proximity among the user terminals participating in the activity in the current cycle includes:

[0273] Obtain the interaction data among the user terminals participating in the activity in the current cycle. The interaction data includes at least one of the number of interaction items participated together, the number of likes, the number of comments, and the number of shares;

[0274] Calculate the social proximity degree among the user terminals according to the interaction data.

[0275] In some embodiments, obtain the user terminal attribute values of each user terminal participating in the activity in the current cycle, including:

[0276] Obtain the user terminal behavior data of each user terminal participating in the activity in the current cycle, and perform digital quantization on the user terminal behavior data to obtain the user terminal attribute data of each user terminal. The user terminal attribute data includes at least one of the following: the frequency of the user terminal participating in the activity, the payment amount of the user terminal participating in the activity, the total number of times the user terminal participates in the target activity, and the consumption level of the user terminal on the target platform;

[0277] Calculate the user terminal attribute values of each user terminal according to the user terminal attribute data of each user terminal.

[0278] In some embodiments, determine the city heat map for display on the target page according to the current user terminal participation heat index of each city, including:

[0279] Normalize the current user terminal participation heat index of each city according to the current user terminal participation heat index of each city, the minimum value of the current user terminal participation heat index of all cities, and the maximum value of the current user terminal participation heat index of all cities to obtain the normalized current user terminal participation heat index of each city;

[0280] Determine the city heat map for display on the target page according to the normalized current user terminal participation heat index of each city.

[0281] In some embodiments, the method further includes:

[0282] Set a corresponding basic exchange value for each user terminal;

[0283] Calculate the dynamic exchange value of each user terminal according to the current user terminal participation heat index of each city;

[0284] Update the integral value of each user terminal according to the dynamic exchange value to obtain the updated integral value of each user terminal, and send the updated integral value of each user terminal to the target page of each user terminal for display.

[0285] In some embodiments, the interaction feedback information further includes lighting feedback information. Generate the interaction feedback information for the target page or the virtual character corresponding to the multimodal input data, including:

[0286] Generate a lighting feedback message based on the location information data of the client and the city heat map. The lighting feedback message indicates that the lighting city identifier of the city where the activity location of the target activity is marked on the city heat map has been successfully marked.

[0287] In some embodiments, the method further includes:

[0288] Receive a real-scene marking image sent by the client. The real-scene marking image is a real-scene marking image where the current user of the client places a service mark at the activity location.

[0289] Associate the real-scene marking image with the city heat map to obtain an updated city heat map.

[0290] Send the updated city heat map to the client so that the client can display a highlighted marking effect at the map coordinates corresponding to the activity location in the updated city heat map.

[0291] In some embodiments, the method further includes:

[0292] Receive the interaction behavior data corresponding to the interaction feedback message sent by the client.

[0293] Generate the activity participation information of the client for the target activity according to the interaction behavior data.

[0294] Determine the activity progress feedback message for display on the target page according to the activity participation information.

[0295] Send the activity progress feedback message to the client so that the client can display the activity progress feedback message on the target page. The activity progress feedback message includes at least one of an integral progress bar, an activity effect diagram corresponding to the activity object, and an interactive virtual animation change diagram corresponding to the virtual character. The virtual character is the virtual character corresponding to the activity object of the target activity.

[0296] In some embodiments, the method further includes:

[0297] Receive the real interactive sound sent by the client. The real interactive sound is the real interactive sound collected by the current user corresponding to the client when interacting with the activity object at the activity location.

[0298] Associate the real interactive sound with the virtual sound of the interactive virtual animation of the virtual character to obtain an updated virtual sound.

[0299] Send the updated virtual sound to the client so that the client outputs the updated virtual sound.

[0300] In some embodiments, the method further includes:

[0301] Generate a voice - associated response animation for a virtual character based on real interactive voices, and send the voice - associated response animation of the virtual character to the client side so that the client side can display the voice - associated response animation on the target page; or

[0302] Generate an updated integral value for the client side according to the real interactive voices, and send the updated integral value to the client side so that the client side can display the updated integral value on the target page.

[0303] In some embodiments, the method further includes:

[0304] Generate achievement sharing information according to the interaction behavior data and real interactive voices collected by the client side, as well as other interaction behavior data and other real interactive voices collected by other client sides, and send the achievement sharing information to the client side so that the client side can display the achievement sharing information on the target page;

[0305] Among them, the achievement sharing information includes a voice gallery, a map, and achievement badges; the voice gallery is used to display the real interactive voices collected by the client side and other real interactive voices collected by other client sides; the map is used to display a highlighted marking effect at the map coordinates corresponding to the activity location in the updated city heat map; the achievement badges are used to display the badges obtained by the client side.

[0306] In some embodiments, generate interaction feedback information for the target page or the virtual character according to the multimodal input data, including:

[0307] Predict the user's intention and the user's emotional intensity according to the multimodal input data;

[0308] Determine the type of user interaction behavior according to the user's intention and the user's emotional intensity;

[0309] Determine the interaction feedback information for the virtual character according to the type of user interaction behavior, and the interaction feedback information includes at least one of the virtual character's interactive virtual animation, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, and vibration feedback information.

[0310] In some embodiments, the multimodal input data includes at least microphone data and touch - screen data;

[0311] Predict the user's intention and the user's emotional intensity according to the multimodal input data, including:

[0312] Process the microphone data to obtain the blowing intensity of the current user corresponding to the client side;

[0313] Process the touch - screen data to obtain the touch force, touch duration, and touch position change rate of the current user corresponding to the client side;

[0314] Predict the user's intention and the intensity of the user's emotion based on the blowing intensity, touch force, touch duration, and touch position change rate.

[0315] In some embodiments, the microphone data includes sound signals generated when the current user blows air, collected by the microphone of the user terminal at a preset sampling frequency;

[0316] Perform data processing on the microphone data to obtain the blowing intensity of the current user corresponding to the user terminal, including:

[0317] Divide the sound signal into multiple sound frames according to a preset length, and apply a Hamming window function to each sound frame for window function processing to obtain the window function processed sound frames;

[0318] Perform a fast Fourier transform on the window function processed sound frames to obtain the spectral components corresponding to each sound frame, and calculate the complex power of each spectral component;

[0319] Calculate the power spectral density of each spectral component according to the complex power of each spectral component and the total number of frequency components;

[0320] Determine the spectral component with the highest calculated power spectral density as the blowing intensity of the current user corresponding to the user terminal, and perform normalization processing on the blowing intensity to obtain the normalized blowing intensity.

[0321] In some embodiments, the touch screen data includes touch signals corresponding to touch events captured by the sensors of the touch screen of the user terminal when the current user acts on the touch screen;

[0322] Perform data processing on the touch screen data to obtain the touch force, touch duration, and touch position change rate of the current user corresponding to the user terminal, including:

[0323] Determine the touch force of the current user corresponding to the user terminal according to the touch signal, and perform normalization processing on the touch force to obtain the normalized touch force;

[0324] Calculate the touch duration of the current user corresponding to the user terminal according to the touch start timestamp and touch end timestamp of the touch event, and perform normalization processing on the touch duration to obtain the normalized touch duration;

[0325] Determine the touch position change rate of the current user corresponding to the user terminal according to the distance change amount and time difference between adjacent touch points among the continuous touch points of the touch event, and perform normalization processing on the touch position change rate to obtain the normalized touch position change rate, where the distance change amount between adjacent touch points is obtained by calculating the Euclidean distance between adjacent touch points according to the effective position coordinates of the continuous touch points.

[0326] For the corresponding implementation manners in the interaction method of the server end in the embodiments of the present application, reference may be made to the specific implementation manners in the interaction method of the client end described above. For the sake of brevity, they will not be elaborated herein.

[0327] Any combination of the above all technical solutions can form optional embodiments of the present application, which will not be elaborated one by one herein.

[0328] To facilitate better implementation of the interaction method in the embodiments of the present application, the embodiments of the present application further provide an interaction device. Please refer to Figure 8 , Figure 8 which is the first structural schematic diagram of the interaction device provided in the embodiments of the present application. The interaction device 300 may include:

[0329] A display unit 310, configured to display a target page through a client end, where the target page displays at least one virtual character;

[0330] An acquisition unit 320, configured to acquire multimodal input data, where the multimodal input data includes at least one of position information data, microphone data, and touch screen data;

[0331] A first sending unit 330, configured to send the multimodal input data to a server end, so that the server end generates interaction feedback information for the target page or the virtual character according to the multimodal input data;

[0332] An interaction unit 340, configured to receive the interaction feedback information sent from the server end and display the interaction feedback information on the target page.

[0333] Optionally, the multimodal input data includes position information data. When the acquisition unit 320 is configured to acquire the multimodal input data, it is specifically configured to:

[0334] Acquire the position information data of the client end;

[0335] The interaction feedback information includes target activity information. When the interaction unit 340 is configured to receive the interaction feedback information sent from the server end and display the interaction feedback information on the target page, it is specifically configured to:

[0336] Receive the target activity information sent from the server end, where the target activity information is determined by the server end according to the position information data of the client end, and the target activity information at least includes the activity position, activity content, and participation method of the target activity;

[0337] Display the target activity information on the target page.

[0338] Optionally, the interaction feedback information further includes a city heat map. When the interaction unit 340 is configured to display the interaction feedback information on the target page, it is specifically configured to:

[0339] Display a city heat map on the target page, and mark the lighted city identifier of the city where the activity location of the target activity is located on the city heat map. The city heat map is used to reflect the participation activity of each city participating in the activity through the target platform in the current cycle;

[0340] Display the activity content and participation method of the target activity through the target page.

[0341] Optionally, the city heat map is implemented by the server through the following steps:

[0342] Obtain the social proximity among the user terminals participating in the activity in the current cycle;

[0343] Obtain the user terminal attribute values of each user terminal participating in the activity in the current cycle;

[0344] Calculate the current user terminal participation heat index of each city participating in the activity in the current cycle according to the number of user terminals participating in the activity in the current cycle, the social proximity among the user terminals, the user terminal attribute values, and the behavior influence strength, where the behavior influence strength is obtained by converting the historical user terminal participation heat index of each city participating in the activity in the previous cycle;

[0345] Determine the city heat map to be displayed on the target page according to the current user terminal participation heat index of each city.

[0346] Optionally, the interaction feedback information further includes lighted feedback information. When the interaction unit 340 is used to display the interaction feedback information on the target page, it is specifically used for:

[0347] Display the lighted feedback information on the target page. The lighted feedback information indicates that the lighted city identifier of the city where the activity location of the target activity is located on the city heat map has been marked successfully.

[0348] Optionally, the device is further used for:

[0349] Obtain the real-scene marker image of the current user of the user terminal placing a service marker at the activity location, and send the real-scene marker image to the server, so that the server associates the real-scene marker image with the city heat map to obtain an updated city heat map;

[0350] Display the updated city heat map received from the server, and display a highlighted marker effect at the map coordinates corresponding to the activity location in the updated city heat map.

[0351] Optionally, when the obtaining unit 320 is used to obtain the location information data of the user terminal, it is specifically used for:

[0352] Obtain the first interaction mode instruction for participating in the activity input through the target page, and send the first interaction mode instruction to the server. The first interaction mode instruction represents the start instruction for the user side to participate in the activity;

[0353] Receive the first permission request for obtaining location information data sent by the server in response to the first interaction mode instruction, and display a first permission acquisition pop-up window on the target page;

[0354] In response to the first authorization information input through the first permission acquisition pop-up window, obtain the location information data of the user side.

[0355] Optionally, the device is further configured to:

[0356] Obtain the interaction behavior data corresponding to the interaction feedback information, and send the interaction behavior data to the server, so that the server generates the activity participation degree information of the user side for the target activity according to the interaction behavior data, and determines the activity progress feedback information for display on the target page according to the activity participation degree information;

[0357] Receive the activity progress feedback information sent from the server, and display the activity progress feedback information on the target page. The activity progress feedback information includes at least one of an integral progress bar, an activity effect diagram corresponding to the activity object, and an interactive virtual animation change diagram corresponding to the virtual character. The virtual character is the virtual character corresponding to the activity object of the target activity.

[0358] Optionally, the device is further configured to:

[0359] Obtain the real interaction sound collected when the current user corresponding to the user side interacts with the activity object at the activity location, and send the real interaction sound to the server, so that the server associates the real interaction sound with the virtual sound of the interactive virtual animation of the virtual character to obtain the updated virtual sound;

[0360] Output the updated virtual sound received from the server.

[0361] Optionally, the device is further configured to:

[0362] Send the real interaction sound to the server, so that the server generates a voice association response animation for the virtual character according to the real interaction sound, and receive the voice association response animation of the virtual character sent by the server, and display the voice association response animation on the target page; or

[0363] Send the real interaction sound to the server, so that the server generates an updated integral value for the user side according to the real interaction sound, and receive the updated integral value sent by the server, and display the updated integral value on the target page.

[0364] Optionally, the device is further configured to:

[0365] Receive the achievement sharing information sent by the server and display the achievement sharing information on the target page. The achievement sharing information is generated by the server based on the interaction behavior data and real interaction voices collected by the client, as well as other interaction behavior data and other real interaction voices collected by other clients;

[0366] Among them, the achievement sharing information includes a sound gallery, a map, and achievement badges; the sound gallery is used to display the real interaction voices collected by the client and other real interaction voices collected by other clients; the map is used to display a highlight marking effect at the map coordinates corresponding to the activity location in the updated city heat map; the achievement badges are used to display the badges obtained by the client.

[0367] Optionally, when the display unit 310 is used to display the target page through the client, it is also used to:

[0368] Obtain a login request for the target page, where the login request includes the user information and device information corresponding to the client;

[0369] Send the login request to the server so that the server can determine whether the client meets the participation conditions for the activity based on the login request;

[0370] Receive the login request response information sent by the server when it determines that the client meets the participation conditions for the activity;

[0371] Display a virtual interaction interface and operation guide information corresponding to the virtual interaction interface on the target page according to the login request response information. The virtual interaction interface is used for the current user corresponding to the client to perform virtual interaction with the virtual character.

[0372] Optionally, the participation conditions include that the current user corresponding to the client indicated by the user information exists in the white list and the device information meets the device preconditions. The device preconditions include that the touch screen of the client supports the pressure sensing function and the multi-touch function, and the client has at least a controllable linear vibration motor, a gravity sensor, a distance sensor, and a speaker.

[0373] Optionally, when the acquisition unit 320 is used to acquire multi-modal input data, it is specifically used to:

[0374] Obtain a second interaction mode instruction input through the virtual interaction interface and send the second interaction mode instruction to the server. The second interaction mode instruction represents an activation instruction for the current user corresponding to the client to perform virtual interaction with the virtual character;

[0375] Obtain a second permission request for acquiring multi-modal input data sent by the server in response to the second interaction mode instruction, and display a second permission acquisition pop-up window on the target page;

[0376] Upon receiving the second authorization information input through the pop-up window for the second permission, obtain multimodal input data;

[0377] Among them, the multimodal input data includes at least microphone data and touch screen data;

[0378] The microphone data includes the sound signal generated when the current user blows air, collected by the microphone of the user device at a preset sampling frequency;

[0379] The touch screen data includes the touch force corresponding to the touch event generated when the current user acts on the touch screen, captured by the sensor of the touch screen of the user device.

[0380] Optionally, the interaction feedback information includes at least one of the interactive virtual animation of the virtual character, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, vibration feedback information;

[0381] The interaction feedback information is determined by the server according to the predicted user intention and user emotion intensity from the multimodal input data, and according to the user intention and user emotion intensity to determine the type of user interaction behavior, and according to the type of user interaction behavior.

[0382] The embodiment of the present application also provides another interaction device. Please refer to Figure 9 , Figure 9 is the second structural schematic diagram of the interaction device provided by the embodiment of the present application. The interaction device 400 may include:

[0383] A receiving unit 410, configured to receive multimodal input data sent from the user device, where the multimodal input data includes at least one of location information data, microphone data, and touch screen data;

[0384] A generating unit 420, configured to generate interaction feedback information for the target page or virtual character according to the multimodal input data, where the target page is the page displayed on the user device, and at least one virtual character is displayed on the target page;

[0385] A second sending unit 430, configured to send the interaction feedback information to the user device, so that the user device displays the interaction feedback information on the target page.

[0386] Optionally, the multimodal input data includes the location information data of the user device, and the interaction feedback information includes target activity information. When generating the interaction feedback information corresponding to the target page or virtual character according to the multimodal input data, the generating unit 420 is specifically configured to:

[0387] Filter out the target activity information related to the location information data from the activity database corresponding to the target platform according to the location information data of the user device;

[0388] Among them, the target activity information at least includes the activity location, activity content, participation method, and virtual interaction interface of the target activity. The virtual interaction interface is used for the user terminal to perform virtual interaction with the virtual character corresponding to the activity object of the target activity;

[0389] The activity database stores the activity location, activity content, participation method, and interaction requirements of all activities held by the target platform in the current cycle. The target platform is the platform to which the target page belongs.

[0390] Optionally, the target activity information further includes a city heat map, which is used to reflect the participation activity levels of each city participating in activities through the target platform in the current cycle; The device is further used for:

[0391] Obtain the social proximity degree among the user terminals participating in activities in the current cycle;

[0392] Obtain the user terminal attribute values of each user terminal participating in activities in the current cycle;

[0393] According to the number of user terminals participating in activities in the current cycle, the social proximity degree among user terminals, the user terminal attribute values, and the behavior influence strength, calculate the current user terminal participation heat index of each city participating in activities in the current cycle, where the behavior influence strength is obtained by conversion according to the historical user terminal participation heat index of each city participating in activities in the previous cycle;

[0394] Determine the city heat map to be displayed on the target page according to the current user terminal participation heat index of each city.

[0395] Optionally, when the device is used to obtain the social proximity degree among the user terminals participating in activities in the current cycle, it is specifically used for:

[0396] Obtain the interaction data among the user terminals participating in activities in the current cycle. The interaction data includes at least one of the number of interaction items participated in together, the number of likes, the number of comments, and the number of shares;

[0397] Calculate the social proximity degree among user terminals according to the interaction data.

[0398] Optionally, when the device is used to obtain the user terminal attribute values of each user terminal participating in activities in the current cycle, it is specifically used for:

[0399] Obtain the client behavior data of each client participating in the activity in the current period, and numerically quantify the client behavior data to obtain the client attribute data of each client. The client attribute data includes at least one of the following: the frequency of the client participating in the activity, the payment amount of the client participating in the activity, the total number of times the client participates in the target activity, and the consumption level of the client on the target platform;

[0400] Calculate the client attribute value of each client according to the client attribute data of each client.

[0401] Optionally, when the device is used to determine the city heat map to be displayed on the target page according to the current client participation heat index of each city, it is specifically used for:

[0402] Normalize the current client participation heat index of each city according to the current client participation heat index of each city, the minimum value of the current client participation heat index of all cities, and the maximum value of the current client participation heat index of all cities, to obtain the normalized current client participation heat index of each city;

[0403] Determine the city heat map to be displayed on the target page according to the normalized current client participation heat index of each city.

[0404] Optionally, the device is also used for:

[0405] Set a corresponding basic exchange value for each client;

[0406] Calculate the dynamic exchange value of each client according to the current client participation heat index of each city;

[0407] Update the integral value of each client according to the dynamic exchange value to obtain the updated integral value of each client, and send the updated integral value of each client to the target page of each client for display.

[0408] Optionally, the interactive feedback information further includes a lighting feedback information. When the generating unit 420 is used to generate the interactive feedback information corresponding to the target page or the virtual character according to the multimodal input data, it is specifically used for:

[0409] Generate a lighting feedback information according to the location information data of the client and the city heat map. The lighting feedback information indicates that the lighting city identifier of the city where the activity location of the target activity is marked on the city heat map has been marked successfully.

[0410] Optionally, the device is also used for:

[0411] Receive the real-scene marked image sent by the user terminal, where the real-scene marked image is the real-scene marked image of the current user of the user terminal placing a service mark at the activity location;

[0412] Associate the real-scene marked image with the urban heat map to obtain an updated urban heat map;

[0413] Send the updated urban heat map to the user terminal so that the user terminal can display a highlighted mark effect at the map coordinates corresponding to the activity location in the updated urban heat map.

[0414] Optionally, the device is further configured to:

[0415] Receive the interaction behavior data corresponding to the interaction feedback information sent by the user terminal;

[0416] Generate the activity participation information of the user terminal for the target activity according to the interaction behavior data;

[0417] Determine the activity progress feedback information for display on the target page according to the activity participation information;

[0418] Send the activity progress feedback information to the user terminal so that the user terminal can display the activity progress feedback information on the target page, where the activity progress feedback information includes at least one of an integral progress bar, an activity effect diagram corresponding to the activity object, and an interactive virtual animation change diagram corresponding to the virtual character, and the virtual character is the virtual character corresponding to the activity object of the target activity.

[0419] Optionally, the device is further configured to:

[0420] Receive the real interactive sound sent by the user terminal, where the real interactive sound is the real interactive sound collected by the current user corresponding to the user terminal when interacting with the activity object at the activity location;

[0421] Associate the real interactive sound with the virtual sound of the interactive virtual animation of the virtual character to obtain an updated virtual sound;

[0422] Send the updated virtual sound to the user terminal so that the user terminal outputs the updated virtual sound.

[0423] Optionally, the device is further configured to:

[0424] Generate a voice association response animation of the virtual character according to the real interactive sound, and send the voice association response animation of the virtual character to the user terminal so that the user terminal can display the voice association response animation on the target page; or

[0425] Generate an updated integral value of the user terminal according to the real interactive sound, and send the updated integral value to the user terminal so that the user terminal can display the updated integral value on the target page.

[0426] Optionally, the device is further configured to:

[0427] Generate achievement sharing information based on the interaction behavior data and real interaction sounds collected by the client, as well as other interaction behavior data and other real interaction sounds collected by other clients, and send the achievement sharing information to the client so that the client can display the achievement sharing information on the target page;

[0428] The achievement sharing information includes a sound gallery, a map, and achievement badges; the sound gallery is used to display the real interaction sounds collected by the client and other real interaction sounds collected by other clients; the map is used to display a highlighted marking effect at the map coordinates corresponding to the activity location in the updated city heat map; the achievement badges are used to display the badges obtained by the client.

[0429] Optionally, when the generating unit 420 is configured to generate interaction feedback information for the target page or the virtual character according to the multimodal input data, it is further configured to:

[0430] Predict the user intention and user emotion intensity according to the multimodal input data;

[0431] Determine the user interaction behavior type according to the user intention and user emotion intensity;

[0432] Determine the interaction feedback information for the virtual character according to the user interaction behavior type, where the interaction feedback information includes at least one of the interaction virtual animation of the virtual character, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, and vibration feedback information.

[0433] Optionally, the multimodal input data includes at least microphone data and touch screen data;

[0434] Predicting the user intention and user emotion intensity according to the multimodal input data includes:

[0435] Process the microphone data to obtain the blowing intensity of the current user corresponding to the client;

[0436] Process the touch screen data to obtain the touch force, touch duration, and touch position change rate of the current user corresponding to the client;

[0437] Predict the user intention and user emotion intensity according to the blowing intensity, touch force, touch duration, and touch position change rate.

[0438] Optionally, the microphone data includes the sound signal generated when the current user blows air collected by the microphone of the client at a preset sampling frequency;

[0439] When the device is used to process microphone data to obtain the blowing intensity of the current user corresponding to the user terminal, it is specifically used for:

[0440] The sound signal is segmented into multiple sound frames according to a preset length, and each sound frame is processed with a Hamming window function to obtain the sound frame after window function processing;

[0441] Perform a fast Fourier transform on the sound frame after window function processing to obtain the spectral components corresponding to each sound frame, and calculate the complex power of each spectral component;

[0442] According to the complex power of each spectral component and the total number of frequency components, calculate the power spectral density of each spectral component;

[0443] Determine the spectral component with the highest calculated power spectral density as the blowing intensity of the current user corresponding to the user terminal, and perform normalization processing on the blowing intensity to obtain the normalized blowing intensity.

[0444] Optionally, the touch screen data includes capturing the touch signal corresponding to the touch event generated when the current user acts on the touch screen through the sensor of the touch screen of the user terminal;

[0445] When the device is used to process touch screen data to obtain the touch force, touch duration, and touch position change rate of the current user corresponding to the user terminal, it is specifically used for:

[0446] Determine the touch force of the current user corresponding to the user terminal according to the touch signal, and perform normalization processing on the touch force to obtain the normalized touch force;

[0447] Calculate the touch duration of the current user corresponding to the user terminal according to the touch start timestamp and touch end timestamp of the touch event, and perform normalization processing on the touch duration to obtain the normalized touch duration;

[0448] Determine the touch position change rate of the current user corresponding to the user terminal according to the distance change amount and time difference between adjacent touch points among the continuous touch points of the touch event, and perform normalization processing on the touch position change rate to obtain the normalized touch position change rate, where the distance change amount between adjacent touch points is obtained by calculating the Euclidean distance between adjacent touch points according to the effective position coordinates of the continuous touch points.

[0449] Each unit in the above interaction device can be implemented in whole or in part by software, hardware, and their combination. Each of the above units can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above units.

[0450] The interaction device 300 or the interaction device 400 can be integrated into a terminal or a server that has a memory and is installed with a processor and has computing capabilities, or the above devices can be a terminal or a server.

[0451] Optionally, the present application further provides a computer device, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0452] Figure 10 It is a schematic structural diagram of the computer device provided by the embodiment of the present application. The computer device can be a terminal or a server. As Figure 8 shown, the computer device 500 may include: a communication interface 501, a memory 502, a processor 503, and a communication bus 504. The communication interface 501, the memory 502, and the processor 503 communicate with each other through the communication bus 504. The communication interface 501 is used for the computer device 500 to perform data communication with external devices. The memory 502 can be used to store software programs and modules. The processor 503 runs the software programs and modules stored in the memory 502, such as the software programs for the corresponding operations in the foregoing method embodiments.

[0453] Optionally, the processor 503 can call the software programs and modules stored in the memory 502 to perform the following operations: display a target page on the user side, and at least one virtual character is displayed on the target page; obtain multimodal input data, and the multimodal input data includes at least one of position information data, microphone data, and touch screen data; send the multimodal input data to the server side so that the server side generates interaction feedback information for the target page or the virtual character according to the multimodal input data; receive the interaction feedback information sent from the server side and display the interaction feedback information on the target page.

[0454] Optionally, the processor 503 can call the software programs and modules stored in the memory 502 to perform the following operations: receive multimodal input data sent from the user side, and the multimodal input data includes at least one of position information data, microphone data, and touch screen data; generate interaction feedback information for the target page or the virtual character according to the multimodal input data, where the target page is the page displayed on the user side, and at least one virtual character is displayed on the target page; send the interaction feedback information to the user side so that the user side displays the interaction feedback information on the target page.

[0455] The present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the interaction method in the embodiments of the present application. For the sake of brevity, details are not described herein again.

[0456] Figure 11 is a schematic diagram of the storage medium provided by the embodiments of the present application. As Figure 11 shown, it depicts a program product 600 for implementing the above method according to an exemplary embodiment of the present application. It can be a portable compact disc read-only memory (CDROM) and includes program code, and can run on a computer device, such as a mobile phone. However, the program product of the present application is not limited thereto. In the present application, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0457] The present application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to execute the corresponding processes in the interaction method in the embodiments of the present application. For the sake of brevity, details are not described herein again.

[0458] The present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to execute the corresponding processes in the interaction method in the present application. For the sake of brevity, details are not described herein again.

[0459] It should be understood that the processor of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0460] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0461] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0462] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0463] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of that module or unit.

[0464] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be in an electrical, mechanical, or other form.

[0465] As mentioned above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An interactive method, characterized in that: The method comprises: Displaying a target page through a user terminal, wherein the target page displays at least one virtual character; Acquire multimodal input data, wherein the multimodal input data includes at least one of position information data, microphone data, and touch screen data; Sending the multimodal input data to a server, so that the server generates interactive feedback information for the target page or the virtual character according to the multimodal input data; The interactive feedback information sent from the server is received, and the interactive feedback information is displayed on the target page.

2. The interactive method according to claim 1, characterized in that: The multimodal input data includes location information data of the user terminal, and the acquiring of the multimodal input data includes: Acquiring location information data of the user terminal; The interactive feedback information includes target activity information, and the receiving the interactive feedback information sent from the server and displaying the interactive feedback information on the target page includes: Receiving the target activity information sent from the server, the target activity information is determined by the server according to the location information data of the user, and the target activity information at least includes the activity location, activity content and participation method of the target activity; The target activity information is displayed on the target page.

3. The interactive method according to claim 2, characterized in that: The interactive feedback information also includes a city heat map, and displaying the interactive feedback information on the target page includes: Displaying the city heat map on the target page, and marking the lighted city logo of the city to which the activity location of the target activity belongs on the city heat map, wherein the city heat map is used to reflect the participation activity of each city participating in the activity through the target platform in the current cycle; The target page displays the activity content and participation method of the target activity.

4. An interactive method, characterized in that: The method comprises: Receiving multimodal input data sent from a user terminal, wherein the multimodal input data includes at least one of position information data, microphone data, and touch screen data; Generate interactive feedback information for a target page or a virtual character according to the multimodal input data, the target page being a page displayed on the user terminal, and the target page displays at least one virtual character; The interactive feedback information is sent to the user terminal, so that the user terminal displays the interactive feedback information on the target page.

5. The interactive method according to claim 4, characterized in that: The step of generating interactive feedback information for a target page or a virtual character according to the multimodal input data includes: Predicting user intention and user emotion intensity based on the multimodal input data; Determine the user interaction behavior type according to the user intention and the user emotion intensity; Determine interactive feedback information for the virtual character according to the user interactive behavior type, wherein the interactive feedback information includes at least one of interactive virtual animation, tactile feedback information, temperature feedback information, ambient light change information, audio feedback information, character expression change information, and vibration feedback information of the virtual character.

6. An interactive device, characterized in that: The device comprises: A display unit, used for displaying a target page through a user terminal, wherein the target page displays at least one virtual character; an acquisition unit, configured to acquire multimodal input data, wherein the multimodal input data includes at least one of position information data, microphone data, and touch screen data; A first sending unit, configured to send the multimodal input data to a server, so that the server generates interactive feedback information for the target page or the virtual character according to the multimodal input data; The interactive unit is used to receive the interactive feedback information sent from the server and display the interactive feedback information on the target page.

7. An interactive device, characterized in that: The device comprises: A receiving unit, configured to receive multimodal input data sent from a user terminal, wherein the multimodal input data includes at least one of position information data, microphone data and touch screen data; A generating unit, configured to generate interactive feedback information for a target page or a virtual character according to the multimodal input data, wherein the target page is a page displayed on the user terminal, and the target page displays at least one virtual character; The second sending unit is used to send the interactive feedback information to the user terminal, so that the user terminal displays the interactive feedback information on the target page.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the interaction method according to any one of claims 1 to 5.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the interaction method according to any one of claims 1 to 5 by calling the computer program stored in the memory.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the interaction method described in any one of claims 1 to 5 is implemented.

Images