Interaction method and device based on virtual object, electronic equipment and medium
The user's operation location information is obtained through intelligent devices, and virtual objects are generated to provide shopping decision support in the virtual world, solving the problem that users find it difficult to make decisions about products in unfamiliar fields in online shopping, simplifying the shopping process and improving the user experience.
Patent Information
- Application Number
- CN202411035061.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-12
AI Technical Summary
In online shopping scenarios, it is difficult for users to quickly and accurately search for required information for unfamiliar fields such as home improvement and other products, and they need expert explanation or self-study to make shopping decisions, resulting in the complexity of the shopping process.
Through the interaction method based on virtual objects, the user's operation position information is obtained using intelligent devices, the interaction detection results are determined, the user's demand information is obtained, and the reply information matching the demand is generated, showing the virtual object to provide shopping decision support in the virtual world , avoid screen restrictions and real shopping guides.
It provides reliable and warm user shopping decision-making basis, simplifies the shopping process, improves user experience, and accompanys users in full-link shopping scenarios through virtual objects, providing near-real auxiliary services.
Smart Images

Figure CN118939121A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to technical fields such as artificial intelligence, smart devices, and smart customer service, and more specifically, to an interaction method, device, electronic device, and medium based on virtual objects. Background Art
[0002] With the continuous development of computer technology, users can use online engines to learn knowledge and use online shopping platforms to shop online. In such search scenarios, if users know what they want, they can find the information they want to search through content browsing, searching and other operations.
[0003] However, for areas that users are not familiar with, it is difficult for users to quickly and accurately search for the desired knowledge, find the desired functions, buy the desired products, etc. For example, in online shopping scenarios, for shopping categories that users are not familiar with, such as home furnishings and medicines, it is difficult for users to quickly and clearly understand the products they intend to buy and find products that meet their shopping needs. Summary of the invention
[0004] In view of this, the present disclosure provides a virtual object-based interaction method, device, electronic device and medium.
[0005] One aspect of the present disclosure provides an interaction method based on virtual objects, including: obtaining posture information of an operating part of a user in the real world, wherein the posture information is determined based on an image taken by a smart device, and the smart device is also used to display a virtual world to the user, wherein the virtual world includes a virtual screen and virtual objects belonging to the same application software as the virtual screen; based on the posture information, determining an interaction detection result for the virtual object; in response to determining that the interaction detection result represents a trigger for an operation on the virtual object, obtaining user demand information, wherein the user demand information includes demand information for the virtual screen; based on the user demand information, generating reply information matching the user demand information; and displaying the reply information.
[0006] According to an embodiment of the present disclosure, the posture information includes first posture information; based on the posture information, determining the interaction detection result for the virtual object includes: when it is determined that the first posture information matches the preset posture information, determining that the interaction detection result is triggering an operation on the virtual object; when it is determined that the first posture information does not match the preset posture information, determining that the interaction detection result is not triggering an operation on the virtual object.
[0007] According to an embodiment of the present disclosure, the posture information also includes first position information, and based on the posture information, determining an interaction detection result for the virtual object includes: determining a position detection result based on the first position information and second position information of the virtual object, wherein the position detection result represents whether an operating part in the real world is in contact with the virtual object in the virtual world, and the second position information represents the position information of the virtual object in the real world; when the position detection result represents that the operating part is in contact with the virtual object and the first posture information matches the preset posture information, determining that the interaction detection result is a trigger for an operation on the virtual object.
[0008] According to an embodiment of the present disclosure, the method also includes: determining the second position information of the virtual object based on the third position information of the virtual screen; or, obtaining the posture information of the smart device, wherein the posture information includes fourth position information and second posture information; determining the second position information based on the fourth position information, the second posture information and the user arm span distance; the user arm span distance represents the farthest distance when the user spreads his arms.
[0009] According to an embodiment of the present disclosure, based on the user demand information, reply information related to the user demand information is generated, including: based on the user demand information, determining the user intention; and generating reply information matching the user demand information according to the user intention.
[0010] According to an embodiment of the present disclosure, user intent is determined based on user demand information, including: obtaining historical demand information and current screen content of a virtual screen, wherein the current screen content includes at least one screen content corresponding to each of multiple shopping functions in the shopping platform; and utilizing a large language model to determine user intent based on user demand information, historical demand information and current screen content.
[0011] According to an embodiment of the present disclosure, reply information matching the user demand information is generated according to the user intention, including: determining a reply strategy matching the user intention; and generating reply information according to the user demand information and the reply strategy using a large language model.
[0012] According to an embodiment of the present disclosure, displaying reply information includes: displaying the reply information in text form to the user through a display window in the virtual world, wherein the position information of the display window is determined according to the second position information of the virtual object; or, converting the reply information into voice data, and generating a lip shape change sequence matching the voice data; generating a virtual object voice reply animation according to the lip shape change sequence and the model parameters of the virtual object; outputting the voice data through an intelligent device, and synchronously displaying the virtual object voice reply animation in the virtual world.
[0013] According to an embodiment of the present disclosure, the method also includes: determining the user's emotional information based on the user demand information and / or historical demand information; determining a virtual object display animation that matches the emotional information, wherein the virtual object display animation indicates changes in display actions and / or display expressions of the virtual object; and simultaneously displaying reply information and the virtual object display animation.
[0014] Another aspect of the present disclosure provides an interactive device based on virtual objects, including: a first acquisition module, used to obtain posture information of a user's operating part in the real world, wherein the posture information is determined based on an image taken by a smart device, and the smart device is also used to display a virtual world to the user, wherein the virtual world includes a virtual screen and virtual objects belonging to the same application software as the virtual screen; an interaction detection module, used to determine an interaction detection result for the virtual object based on the posture information; a second acquisition module, used to obtain user demand information in response to determining that the interaction detection result represents a trigger for an operation on the virtual object, wherein the user demand information includes demand information for the virtual screen; a generation module, used to generate reply information matching the user demand information based on the user demand information; and a display module, used to display the reply information.
[0015] Another aspect of the present disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.
[0016] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.
[0017] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the above method when executed.
[0018] Through the embodiments of the present disclosure, a virtual screen and a virtual object belonging to the same application software as the virtual screen are displayed to the user by means of a smart device, so that the user can view the virtual screen and the virtual object in an immersive way. When the virtual screen and the virtual object are displayed to the user by means of a smart device, the operation of the virtual object is triggered by the posture information of the user's operating part, thereby obtaining the user's demand information; thereafter, by generating and displaying reply information matching the user's demand information, the user's needs can be met at any time, such as providing the user with a reliable and warm basis for the user's shopping decision or the user's shopping decision. In addition, since the virtual object generates reply information matching the user's demand information in real time in the virtual world, the embodiments of the present disclosure can not only provide the user with close-to-real auxiliary services, but also avoid the virtual object blocking the virtual screen due to the screen limitation of the mobile phone or terminal device, thereby improving the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0020] Figure 1 The application scenario diagram schematically shows the application of the virtual object-based interaction method and device disclosed in the present invention;
[0021] Figure 2 A flowchart of a virtual object-based interaction method according to an embodiment of the present disclosure is schematically shown;
[0022] Figure 3 A schematic diagram schematically shows a scenario of a method for determining an interaction detection result for the virtual object according to an embodiment of the present disclosure;
[0023] Figure 4 A flowchart of a method for determining reply information according to an embodiment of the present disclosure is schematically shown;
[0024] FIG5 schematically shows a flow chart of displaying a virtual object display animation and replying information according to another embodiment of the present disclosure;
[0025] Figure 6 A block diagram schematically shows a virtual object-based interaction device according to an embodiment of the present disclosure; and
[0026] Figure 7 A block diagram of an electronic device suitable for implementing a virtual object-based interaction method and apparatus according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0031] The existing online shopping model can satisfy users' rapid search and purchase of high-standard and low-cognitive threshold categories, such as fast-moving consumer goods, clothing, information appliances 3C (a combination of computers, communications and consumer electronics), etc. For high-standard and low-cognitive threshold categories, users know exactly what they want and what they should want. However, for non-standard and high-cognitive threshold categories, such as home furnishings, users need a process of demand decoding before completing shopping, that is, users need to go through expert explanations or self-study (usually up to 3 months) to gradually know what they should want, and then make reliable shopping decisions.
[0032] Therefore, in order to provide users with reliable and warm user shopping decision basis or user shopping decision in the full-link shopping process, the embodiment of the present disclosure creates a virtual object in the virtual world that belongs to the same application software as the virtual object. The virtual object can accompany the user in the full-link shopping scenario and provide the user with user shopping decision basis or user shopping decision when the user triggers an operation on the virtual object.
[0033] Specifically, an embodiment of the present disclosure provides an interaction method based on virtual objects, including: obtaining posture information of a user's operating part in the real world, wherein the posture information is determined based on an image taken by a smart device, and the smart device is also used to display a virtual world to the user, the virtual world including a virtual screen and virtual objects belonging to the same application software as the virtual screen; based on the posture information, determining an interaction detection result for the virtual object; in response to determining that the interaction detection result represents a trigger for an operation on the virtual object, obtaining user demand information, the user demand information including demand information for the virtual screen; based on the user demand information, generating reply information related to the user demand information; and displaying the reply information.
[0034] Figure 1 The following schematically shows an application scenario diagram to which the method and device for virtual object-based interaction disclosed in the present invention can be applied. It should be noted that: Figure 1 What is shown are merely examples of application scenarios to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0035] like Figure 1 As shown, in the embodiment 100 of the interaction method based on virtual objects, the user 101 can wear a smart device 102, and a plurality of virtual images 104 in the virtual world are presented to the user 101. It can be understood that when the user 101 wears the smart device 102, the virtual world can be viewed through the smart device 102, and when the user 101 does not wear the smart device 102, the real world can be viewed through the eyes.
[0036] Each virtual screen 104 may be a screen displayed by the smart device 102 after superimposing pictures, videos, application software pages, etc. on the real world using virtual technology. For example, virtual technology includes but is not limited to the following implementations: Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Extended Reality (XR).
[0037] For example, the virtual screen 104 may be a virtual page displayed to the user 101 after the smart device 102 projects the home page, search page, product browsing page, product details page, payment page, etc. of the shopping platform.
[0038] The embodiment 100 also includes a virtual object 103. The virtual object 103 may be a virtual object belonging to the same application software as the virtual screen, for example, a virtual person or a virtual animal; or, when the virtual screen is a screen of a shopping platform, the virtual object may be a logo object of the shopping platform.
[0039] After the user 101 wears the smart device 102, he can view the virtual screen 104 and the virtual object 103 on the objects or background displayed in the real world. The user 101 can interact with the virtual screen 104 and the virtual object 103 and control the virtual screen 104 and the virtual object 103 through gestures and other operations.
[0040] It should be noted that the virtual object-based interaction method provided in the embodiments of the present disclosure can generally be implemented by a server. For example, the server can be a background server that supports the application software to which the virtual objects and virtual images belong, such as a background server of a shopping platform. Accordingly, the virtual object-based interaction device provided in the embodiments of the present disclosure can generally be set in the server.
[0041] Figure 2 The flowchart of the virtual object-based interaction method according to an embodiment of the present disclosure is schematically shown.
[0042] like Figure 2 As shown, the method includes operations S210 to S250.
[0043] In operation S210, the posture information of the user's operating part in the real world is obtained, wherein the posture information is determined based on the image taken by the smart device. The smart device is also used to display a virtual world to the user, and the virtual world includes a virtual screen and virtual objects belonging to the same application software as the virtual screen.
[0044] In operation S220 , an interaction detection result for the virtual object is determined based on the posture information.
[0045] In operation S230 , in response to determining that the interaction detection result represents a triggering operation on the virtual object, user demand information is acquired, wherein the user demand information includes demand information for the virtual screen.
[0046] In operation S240, based on the user demand information, reply information matching the user demand information is generated.
[0047] In operation S250, reply information is displayed.
[0048] According to the embodiments of the present disclosure, the real world and the virtual world can be understood as the world that the user actually sees and the world that is presented to the user through the virtual technology of the smart device. The user can see the virtual world, as well as the virtual images and virtual objects that exist in the virtual world through the smart equipment he wears. In addition, although the user and his operating parts exist in the real world, the smart device can also display the user's operating parts in the virtual world.
[0049] For example, user A wears smart equipment, such as VR glasses, and user B is beside user A and does not wear a smart device. User A can see the virtual screen, virtual objects, and user B through the VR glasses. User B can only see user A, but cannot see the virtual screen and virtual objects.
[0050] In an illustrative embodiment, after the user wears the smart device, he can see the virtual object at any place in the field of vision. As the user turns his head, lowers his head, or performs other actions that cause the field of vision to change, the virtual object can also change with the change of the user's visible field of vision.
[0051] According to an embodiment of the present disclosure, the smart device includes but is not limited to: VR device, AR device, MR device and XR device. The smart device includes a camera unit for capturing images from at least one perspective, for example, the camera unit can capture the real-world environment of the user, the user's operating part, etc.
[0052] According to an embodiment of the present disclosure, a user can interact with a virtual screen and a virtual object in a virtual world through the posture information of an operating part. The posture information of an operating part can indicate the position and posture of the operating part in the real world. For example, when the operating part is a hand, the posture information can include the position and posture of the hand, such as a gesture.
[0053] In a specific embodiment, the smart device is equipped with a target detection algorithm, which can perform target detection on the captured image to identify the posture information of the operating part. For example, the target detection algorithm can be a gesture recognition algorithm. The smart device can also analyze the captured image according to the position of the smart device and the position of the camera unit to determine the relative position of the operating part and the smart device / camera unit; then, combined with the position of the smart device, determine the position information of the operating part.
[0054] In an embodiment of the present disclosure, for operation S210, the server of the application software of the virtual screen and the virtual object can obtain the posture information of the user's operating part through interactive operation with the smart device. For example, the server can directly obtain the above posture information from the smart device, or the server can also interact with the background server of the smart device and obtain the above posture information from the background server.
[0055] According to an embodiment of the present disclosure, the virtual object displayed to the user belongs to the same application software as the virtual screen. After the user opens the application software, the virtual screen and virtual object are rendered and displayed to the user through the smart device. It can be understood that the virtual screen of any page in the application software after being projected by the smart device is associated with the virtual object. When the application software is not opened, the virtual screen and virtual object of the application software are not displayed.
[0056] For example, the virtual screen can be a virtual screen projected on a real scene from any shopping page in the shopping platform, and the virtual object can be a virtual shopping guide image of the shopping platform. After the user opens the shopping platform, when switching the virtual screen projected from any shopping page in the shopping platform, the user can always see the virtual object. When the user exits the shopping platform, the virtual screen displayed by the smart device has nothing to do with the shopping platform, and accordingly, the user can no longer see the virtual object.
[0057] According to an embodiment of the present disclosure, for operation S220, the interaction detection result for the virtual object can be determined based on the posture information. Since the virtual object only exists in the virtual world, the user and the operating part cannot actually touch the virtual object. Therefore, the user can perform various operations on the virtual object through the posture information of the operating part to achieve interaction with the virtual object. At this time, the virtual object can also achieve corresponding functions according to the user's operation. For example, the function of obtaining user demand information can be achieved.
[0058] The interaction detection result for the virtual object includes: whether an operation on the virtual object is triggered. In some embodiments, the interaction detection result may also include a specific operation of the user on the virtual object. For example, the user's operation on the virtual object includes: selecting the virtual object, touching the virtual object, clicking the virtual object, etc.
[0059] According to an embodiment of the present disclosure, for operation S230, when it is determined that the user triggers an operation on a virtual object, user demand information is obtained. The user demand information includes demand information for a virtual screen, for example, demand information related to the content displayed on the virtual screen. When the virtual screen displays home improvement products, the user demand information may be "what does the AA mark of home improvement products mean" or "how to distinguish the quality of different seat materials".
[0060] Since the virtual object and the virtual screen belong to the same application software, as long as the user has user demand information for the virtual screen, reply information related to the user demand information can be displayed to the user through the virtual object.
[0061] For operation S240, reply information matching the user demand information may be generated by natural language processing (NLP). For example, NLP may be used to segment and extract features of the user demand information, and then, the reply information having a correlation may be matched based on the feature-extracted information.
[0062] According to an embodiment of the present disclosure, the reply information may be a variety of information such as product recommendations, function indexes, scene introductions, chat replies, etc.
[0063] With respect to operation S250, since the user demand information is obtained in response to the operation on the virtual object, in order to improve the user experience, the generated reply information can be displayed through the virtual object, for example, in a certain area near the virtual object.
[0064] Through the embodiments of the present disclosure, a virtual screen and a virtual object belonging to the same application software as the virtual screen are displayed to the user by means of a smart device, so that the user can view the virtual screen and the virtual object in an immersive way. When the virtual screen and the virtual object are displayed to the user by means of a smart device, the operation of the virtual object is triggered by the posture information of the user's operating part, thereby obtaining the user's demand information; thereafter, by generating and displaying reply information matching the user's demand information, the user's needs can be met at any time, such as providing the user with a reliable and warm basis for the user's shopping decision or the user's shopping decision. In addition, since the virtual object generates reply information matching the user's demand information in real time in the virtual world, the embodiments of the present disclosure can not only provide the user with close-to-real auxiliary services, but also avoid the virtual object blocking the virtual screen due to the screen limitation of the mobile phone or terminal device, thereby improving the user's experience.
[0065] According to an embodiment of the present disclosure, the position and posture information of the operating part includes first posture information representing the posture of the operating part.
[0066] According to an embodiment of the present disclosure, for operation S220, based on the posture information, determining the interaction detection result for the virtual object includes: when it is determined that the first posture information matches the preset posture information, determining that the interaction detection result is triggering an operation on the virtual object; when it is determined that the first posture information does not match the preset posture information, determining that the interaction detection result is not triggering an operation on the virtual object.
[0067] The preset posture information may be user-defined posture information of the operating part, which is used to implement a specific function, for example, to obtain user demand information.
[0068] The first posture information and the preset posture information may both include at least one key point of the operation part, and the matching operation of the first posture information and the preset posture information may include: matching at least one key point in the first posture information with at least one key point of the preset posture information.
[0069] Alternatively, at least one key point in the first posture information is connected in a preset order to form a first polygon, and at least one key point in the preset posture information is connected in a preset order to form a second polygon; the shapes of the first polygon and the second polygon are matched.
[0070] In the embodiments of the present disclosure, the operation of the virtual object in the virtual world is triggered in a non-contact manner through the first posture information of the operating part, so that the user can interact with the virtual object in the virtual world in a simple and convenient way, triggering the subsequent acquisition of user demand information.
[0071] According to an embodiment of the present disclosure, the posture information of the operating part includes first posture information representing the posture of the operating part and first position information of the position of the operating part in the real world.
[0072] For operation S220, based on the posture information, determining the interaction detection result for the virtual object also includes: determining the position detection result according to the first position information and the second position information of the virtual object, wherein the position detection result represents whether the operating part in the real world is in contact with the virtual object in the virtual world, and the second position information represents the position information of the virtual object in the real world; when the position detection result represents that the operating part is in contact with the virtual object and the first posture information matches the preset posture information, determining that the interaction detection result is triggering an operation on the virtual object.
[0073] Although virtual objects and virtual images only exist in the virtual world, their display positions in the virtual world are in the same coordinate system as the smart device, such as the world coordinate system.
[0074] For example, there are no other objects placed on object A in the real world. After the user wears the smart device, he can see object A and the virtual object B above object A. At this time, regardless of whether object A can be viewed in the smart device, with the origin of the world coordinate system fixed, the coordinates of object A in the world coordinate system are (x1, y1, z1); similarly, even if the virtual object B cannot be viewed in the real world, the coordinates of the virtual object B in the world coordinate system are (x1, y1, z2).
[0075] In a specific embodiment, the second position information of the virtual object may be position information of a key point in the virtual object, for example, the key point may be a model center point, a highest point, etc. of the virtual object; or the key point may also be a target part of the virtual object, such as the top of the head, an ear, a face, etc. of the virtual object. Determining the position detection result according to the first position information and the second position information includes: determining the position detection result according to whether the first position information and the second position information are the same.
[0076] In another specific embodiment, the second position information of the virtual object may be three-dimensional area range information where the virtual object is located. Determining the position detection result according to the first position information and the second position information includes: determining the position detection result according to whether the first position information falls within the three-dimensional area range information.
[0077] The position detection result indicates that the operation part is in contact with the virtual object and the first posture information matches the preset posture information, indicating that a contact interaction is triggered between the user and the virtual object. For example, the user can pinch the face of the virtual object with his hand to trigger the operation of the virtual object so as to obtain the user's required information later.
[0078] The embodiments of the present disclosure can determine whether contact occurs between the operating part and the virtual object based on the first position information and the second position information, and at the same time determine whether to trigger an operation on the operating object through the first posture information based on the first posture information and the preset posture information of the operating part, thereby supporting the user to trigger interaction with the virtual object through contact interaction, allowing the user to have a more realistic interaction with the virtual object, thereby improving the user's usage experience.
[0079] According to an embodiment of the present disclosure, when the position detection result indicates that the operating part is in contact with the virtual object and the first posture information matches the preset posture information, the user's operating part is displayed in the virtual world, so that the user can watch the interaction between the user and the virtual object in the virtual world in real time through the smart device.
[0080] Figure 3 A schematic diagram of a scenario of a method for determining an interaction detection result for the virtual object according to an embodiment of the present disclosure is schematically shown.
[0081] like Figure 3 As shown, the posture information 301 of the operating part includes first posture information 302 and first position information 303, and the position information of the virtual object in the real space is second position information 304. According to the first position information 303 of the operating part and the second position information 304 of the virtual object, a position detection result 305 can be determined.
[0082] In operation S310, it is determined whether the first posture information matches the preset posture information.
[0083] In operation S320 , it is determined whether the operation part is in contact with the virtual object.
[0084] In operation S330 , an operation on the virtual object is triggered.
[0085] In operation S340 , an operation on the virtual object is not triggered.
[0086] In one embodiment, for operation S310, when it is determined that the first posture information matches the preset posture information, the operation can be directly entered into operation S330 to trigger the operation on the virtual object; when it is determined that the first posture information does not match the preset posture information, the operation can be directly entered into operation S340 without triggering the operation on the virtual object.
[0087] In another embodiment, for operation S310, when it is determined that the first posture information matches the preset posture information and operation S320 determines that the operating part is in contact with the virtual object, operation S330 is entered to trigger the operation on the virtual object; otherwise, operation S340 is entered without triggering the operation on the virtual object.
[0088] In addition, for operation S320, when it is determined that the operating part is not in contact with the virtual object, the process directly proceeds to operation S340 without triggering the operation on the virtual object.
[0089] According to the embodiments of the present disclosure, smart devices can collect voice data input by users. However, smart devices are currently unable to recognize the voice content of the collected voice data. Therefore, it is necessary to use the voice input function of virtual objects to recognize the collected voice data.
[0090] In a specific embodiment, in response to detecting a user's operation on a virtual object, a voice input function of the virtual object is awakened, and then, after the voice input function of the virtual object is awakened, the application software of the virtual object can obtain voice data collected by a smart device; and determine user demand information based on the voice data.
[0091] According to an embodiment of the present disclosure, voice data may be converted into textual user demand information through speech recognition technology (Automatic Speech Recognition, ASR).
[0092] In the embodiments of the present disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.
[0093] For example, in an embodiment of the present disclosure, before collecting voice data, the application software of the virtual object and the virtual screen will request voice permission from the user, and collect voice data with the authorization or consent of the user.
[0094] According to an embodiment of the present disclosure, if a sound wave exceeding a certain threshold is not collected for a preset time, the voice data collection is stopped, and the voice data input this time is automatically converted into user demand information in text form. For example, the voice data collection may be stopped when no sound wave is detected for 0.5 seconds.
[0095] Alternatively, in response to detecting a preset operation of the user on the virtual object, the voice data collection stops, and the voice data input this time is automatically converted into user demand information in text form. For example, the user manually clicks on the virtual object to start collecting voice data, and clicks on the virtual object again to stop collecting voice data.
[0096] According to an embodiment of the present disclosure, after determining the user demand information, the user demand information after text conversion can be displayed to the user to avoid affecting the generation of subsequent reply information. For example, the user demand information can be displayed in text in the area associated with the virtual object.
[0097] In the embodiments of the present disclosure, the virtual object belonging to the same application software as the virtual screen also supports a voice interaction function, which determines user demand information through voice interaction between the user and the virtual object, so that the virtual object can obtain user demand information in a more intelligent and more shopping guide-friendly manner, providing users with a more intelligent and humanized virtual object.
[0098] In related technologies, due to the limited screen size of mobile phones or PCs, virtual objects only appear on specific pages to solve a specific problem, such as customer service digital people. For online shopping scenarios, if virtual objects are displayed on mobile phones or PCs, their essence is still to display virtual objects on a plane, which will block a large area of the shopping page, thereby affecting the user experience and making it difficult for virtual objects to run through the entire chain of the user's shopping experience.
[0099] On the contrary, the embodiments of the present disclosure display virtual pictures and virtual objects in a three-dimensional virtual space. At this time, the virtual objects can accompany the users throughout the entire process and provide services to them at any time, just like a "shopping guide" in an offline shopping scene.
[0100] According to an embodiment of the present disclosure, the method also includes: determining the second position information of the virtual object based on the third position information of the virtual screen; or, obtaining the posture information of the smart device, wherein the posture information of the smart device includes fourth position information and second posture information; determining the second position information based on the fourth position information, the second posture information and the user's arm span distance; the user's arm span distance represents the farthest distance when the user spreads his arms.
[0101] According to an embodiment of the present disclosure, the smart device may also simultaneously display virtual images projected from multiple application software, i.e., the first display state. For example, if a user runs multiple application software at the same time, the smart device may simultaneously display the virtual images of the multiple application software. If the application software to which the multiple virtual images belong includes virtual objects, the virtual objects may be displayed simultaneously. For example, when the smart device is in semi-immersive mode, the virtual image may be in the first display state.
[0102] According to an embodiment of the present disclosure, the smart device may also only display the virtual screen after projecting a single application software, that is, the second display state. For a virtual object that belongs to the same application software as the virtual screen of a certain application software, when the virtual screen is in the second display state, the smart device may also display the virtual object at the same time. On the contrary, when there is no virtual object, only the virtual screen is displayed. For example, when the smart device is in full immersive mode, the virtual screen can be in the second display state, and the virtual space only includes one virtual screen and the virtual object that belongs to the same application software.
[0103] After opening the application software with a virtual screen, the virtual screen and virtual objects can be displayed simultaneously in the virtual space; when the application software is in the running or background running state, the virtual objects all exist in the virtual space.
[0104] In a specific embodiment, the positions of the virtual object and the virtual screen may be bound to each other. Thus, the second position information of the virtual object is determined according to the third position information of the virtual screen. Specifically, based on the predefined position relationship, the second position information of the virtual object is determined according to the third position information of the virtual screen in the world coordinate system. It should be noted that the third position information refers to the position information of the virtual screen in the world coordinate system.
[0105] Specifically, the predefined position relationship may be: the virtual object is within a preset distance range near the virtual screen so that the virtual object does not block the virtual screen; or the virtual object is at the lower right corner of the virtual screen. In this embodiment, when the user drags the virtual screen, the virtual object may move with the movement of the virtual screen.
[0106] For example, in the first display state, the user wearing the smart device can view the virtual screen in the virtual world and the virtual object in the lower right corner of the virtual screen. That is, in the user's perspective, the virtual object is in a certain posture in the virtual world.
[0107] The second position information of the virtual object is determined according to the third position information of the virtual screen.
[0108] In another specific embodiment, the position of the virtual object may be related to the position and posture information of the smart device. The position and posture information of the smart device includes fourth position information and second posture information, and the second posture information may be, for example, azimuth information.
[0109] In this embodiment, the second posture information of the smart device may represent the viewing angle of the user wearing the smart device. For example, when the smart device is tilted downward, it indicates that the user may be in a head-down state.
[0110] In order to ensure that users can interact with virtual objects at any time, virtual objects need to be in a position where users can see them, that is, within the user's field of view. In addition, if the virtual object is too close to the user, it may scare the user, and if it is too far away, the user may not be able to "touch" the virtual object at any time, thus affecting the interactive experience. Therefore, the position of the virtual object is not only related to the posture of the smart device, but also to the user's arm span distance. The user's arm span distance can be the farthest distance when the user spreads his arms according to expert experience statistics.
[0111] Specifically, determining the second position information based on the fourth position information, the second posture information and the user's arm span distance includes: determining the user's field of view based on the second posture information; within the above-mentioned field of view, determining the second position information of the virtual object based on the fourth position information and the user's arm span distance.
[0112] For example, the direction offset 5° from the central axis of the field of view can be used as the field of view direction, and the fourth position information can be used as the starting point. The position point whose distance from the starting point in the field of view direction is less than or equal to the user's arm span distance can be used as the position point of the virtual object, thereby obtaining the second position information.
[0113] According to an embodiment of the present disclosure, when a user switches the spatial anchor point where the virtual screen is located, the second position information of the virtual object can also be relocated, and based on the switched spatial anchor point, the display position of the virtual object can be adjusted so that the user can always find the virtual object in a relatively comfortable position during the shopping experience. The second position information of the virtual object can be relocated according to the third position information of the virtual screen or the posture information of the smart device. The spatial anchor point here can be understood as a real scene or a virtual scene that carries the virtual screen. For example, the real scene can be a restoration of a bedroom or living room in reality, and the virtual scene can be a modeled bedroom or living room.
[0114] The embodiments of the present disclosure determine the display status of the virtual screen and adjust the display position of the virtual object based on the display status, so that in the first display state or the second display state, the user can find the virtual object in a relatively comfortable position during the full-link shopping experience, thereby improving the user experience.
[0115] According to an embodiment of the present disclosure, based on the user demand information, reply information matching the user demand information is generated, including: based on the user demand information, determining the user intention; and generating reply information matching the user demand information according to the user intention.
[0116] Based on user demand information, determining user intent may include: pre-setting multiple user intents and using the pre-set intents as categories; using a classification algorithm to classify user demand information; and determining user intent based on the classification results. For example, for user demand information (query) input by a user, the probability of each intent is calculated based on the classification model, and finally the user intent matching the user demand information is determined.
[0117] Alternatively, multiple intent recognition algorithms can be used to determine user intent. Intent recognition algorithms include but are not limited to at least one of the following: machine learning algorithms, such as support vector machines (SVM), deep learning algorithms, such as long short-term memory networks (LSTM), bidirectional long short-term memory networks (Bi-RNN), Bi-LSTM-CRF, etc.
[0118] For example, for online shopping scenarios, user intentions may include: product recommendations, shopping suggestions, product category background supplements, and voice communication.
[0119] Generating reply information matching the user's demand information according to the user's intention includes: determining a question-and-answer library matching the user's intention; searching the question-and-answer library for the most similar question-and-answer pair to the user's demand information, and using the answer to the question-and-answer pair as the reply information. In other embodiments, the reply information may also be determined in combination with the user's demand information and the historical demand information.
[0120] The embodiments of the present disclosure determine user intent based on user demand information, and generate reply information matching the user demand information according to the user intent. It is able to divert strategies for generating reply information based on user intent, which not only makes the generated reply information more in line with the user's actual intent needs, but also enables the reply information to be generated quickly based on the user's intent.
[0121] Due to the numerous functions involved in the application software, the virtual objects need to face a variety of virtual screens and provide users with a variety of response information for the above virtual screens. Therefore, large language models (LLM) can be integrated with virtual objects so that virtual objects can provide users with comprehensive response information in real time.
[0122] For example, a large language model is used to determine the user's intention based on the user's demand information. Alternatively, the user's historical demand information is used as context information and used together with the user's demand information as input to the large language model to determine the user's intention. The historical demand information may refer to the demand information input by the user through interaction with the virtual object after the virtual object appears this time.
[0123] According to an embodiment of the present disclosure, user intent is determined based on user demand information, including: obtaining historical demand information and current screen content of a virtual screen, wherein the current screen content includes at least one screen content corresponding to each of multiple shopping functions in the shopping platform; and utilizing a large language model to determine user intent based on user demand information, historical demand information and current screen content.
[0124] In the embodiments of the present disclosure, the application software generally includes various function pages, and each function generally includes multiple display pages, and even includes a page adapted to the personalized needs of the user. Therefore, when the above pages are projected as virtual images through the smart device, the virtual objects will be accompanied by various virtual images.
[0125] Based on the records of the above embodiments, the user can see the virtual object in the lower right corner of the virtual screen and within a preset distance range near the virtual screen, and the existence of the virtual object will not block the virtual screen; or the user can also see the virtual object at the distance of the user's arm span from the center of the field of vision. It can be understood that when the user uses the application software, the virtual object will accompany the user at a comfortable position throughout the process.
[0126] User demand information includes demand information for a virtual screen, and the virtual screen may be a projection of pages of various formats. Therefore, the embodiments of the present disclosure take into account the current screen content of the virtual screen, that is, locate the user's position in the full link, so as to provide more accurate reply information.
[0127] Specifically, user demand information, historical demand information and current screen content can be used as inputs of the large language model, such as prompt, and the large language model can be used to output user intent. When constructing the input of the large language model, prompts such as "output user intent" can also be added.
[0128] According to an embodiment of the present disclosure, historical demand information can be obtained through the historical interaction log between the user and the virtual object. The historical demand information may refer to the demand information input by the user through the interaction between the virtual object and the virtual object after the virtual object appears this time, and the historical demand information can be displayed to the user in the form of a dialogue.
[0129] According to the embodiments of the present disclosure, since the virtual screen is obtained by projection of a smart device, the display page before projection is generated by application software, and thus the application software page before projection can be used as the current screen content of the virtual screen.
[0130] For example, taking the application software as a shopping platform, the virtual object can be regarded as a "smart shopping guide" that accompanies users throughout the entire process of shopping. Users can view the shopping homepage of the shopping platform through smart devices, that is, virtual screen 1. When the user opens the shopping platform, the virtual object can appear in front of the user, such as welcoming the user with a door opening animation. After the user clicks on a function in the shopping page, the function area can be viewed through the smart device, such as the discount area product page, that is, virtual screen 2. Virtual screen 1, virtual screen 2 and virtual objects all belong to the shopping platform.
[0131] For virtual screen 1, user demand information can be "Where is XX function? I can't find it", and historical demand information can be "I have never used this shopping platform, how to use it" and the current screen content, such as the shopping homepage. Therefore, the large language model can determine the user's intention as a function index based on the user demand information, historical demand information and the current screen content. Alternatively, the user demand information can be "What is fun to do", combined with the current screen content "shopping homepage", it can be determined that the user's intention is a function recommendation or a product recommendation.
[0132] For virtual screen 2, the user demand information may be "Any recommendations?", and combined with the current screen content "discount area product page", it can be determined that the user intends to recommend products.
[0133] The embodiments of the present disclosure utilize a large language model, combined with user demand information, historical demand information, and current screen content, to generate more accurate response information from three perspectives: real-time demand, historical demand, and current visual perception, thereby improving user experience.
[0134] According to an embodiment of the present disclosure, reply information matching the user demand information is generated according to the user intention, including: determining a reply strategy matching the user intention; and generating reply information based on the user demand information and the reply strategy using a large language model.
[0135] Multiple user intentions can match multiple reply strategies, and multiple reply strategies can include different question and answer libraries. The matching relationship between user intention and reply strategy can be pre-set, so that after determining the user intention according to the user demand information, a reply strategy matching the user intention can be determined based on the pre-set matching relationship.
[0136] Alternatively, the reply strategy may be a pre-set expert rule or reply template. Thus, after determining the user intention according to the user demand information, the expert rule or reply template matching the user intention may be determined based on a pre-set matching relationship.
[0137] In an embodiment of the present disclosure, the input of the large language model can be generated according to the user demand information and the reply strategy, for example, "generate an answer that meets the user demand information with reference to the reply strategy", so that the large language model outputs the reply information.
[0138] In an embodiment of the present disclosure, the large language model may call a toolbox corresponding to a reply strategy to generate reply information that matches the user's demand information.
[0139] The embodiments of the present disclosure utilize user intention to determine the reply strategy, which can achieve strategy diversion for generating reply information, while improving the generation speed and generation accuracy of the reply information.
[0140] According to an embodiment of the present disclosure, the method further includes: utilizing a large language model to determine reply information based on user demand information, historical demand information, and a reply strategy.
[0141] According to an embodiment of the present disclosure, a large language model is used to determine reply information based on user demand information, historical demand information, current screen content of a virtual screen, and a reply strategy. Similar to determining user intent, since a virtual screen can be a projection of pages of various formats, taking the current screen content of the virtual screen into account can locate the current position of the user in the full link, so as to provide more accurate reply information.
[0142] Figure 4 The flowchart of the method for determining reply information according to an embodiment of the present disclosure is schematically shown.
[0143] like Figure 4 As shown, in embodiment 400, the large language model M10 may be used to determine the user's intention and generate reply information. In other embodiments, only the large language model may be used to determine the user's intention or generate reply information, which is not limited.
[0144] The large language model M10 can integrate multiple model components to achieve multiple tasks. In the embodiment of the present disclosure, the large language model M10 includes at least a first model component M11 and a second model component M12, which are respectively used to determine user intent and generate reply information.
[0145] The user demand information 401, the historical demand information 402 and the current screen content 403 of the virtual screen are input into the first model component M11 of the large language model M10, and the user intention is output. After the reply strategy 404 is determined based on the user intention, the reply strategy 404 can be input into the second model component M12 of the large language model M10. The second model component M12 can generate reply information 405 according to the reply strategy 404, the user demand information 401, the historical demand information 402 and the current screen content 403 of the virtual screen.
[0146] According to an embodiment of the present disclosure, displaying reply information includes: displaying the reply information in text form to the user through a display window in the virtual world, wherein the position information of the display window is determined according to the second position information of the virtual object; or, converting the reply information into voice data, and generating a lip shape change sequence matching the voice data; generating a virtual object voice reply animation according to the lip shape change sequence and the model parameters of the virtual object; outputting the voice data through an intelligent device, and synchronously displaying the virtual object voice reply animation in the virtual world. .
[0147] According to an embodiment of the present disclosure, a display window in a virtual world may be a display area associated with a virtual object, and the display window may be in the style of a simulated chat dialog box, and the virtual object may display user demand information and reply information matching the user demand information to the user in the form of a dialogue bubble. In addition, the display area also supports further interaction by the user through clicks and the like.
[0148] There is a positional association between the display window and the virtual object. For example, the display window is usually located above and behind the virtual object.
[0149] In another specific embodiment, the virtual object can also output voice messages to the user in a manner that simulates a real conversation scenario.
[0150] Specifically, after the reply information is generated, the reply information can be converted into voice data using text-to-speech technology, such as deep learning-based text-to-speech (TTS).
[0151] At the same time, a lip shape change sequence can be generated based on the reply information. The change speeds of the two lip shapes in the lip shape change sequence match the change speeds of the two syllables in the voice data to avoid the problem of audio and video being out of sync. In order to create a real interaction scene with the virtual object, the reply information can also be "spoken" by the virtual object. Specifically, the converted voice data can be output through the voice playback unit of the smart device, and the virtual object voice reply animation can be displayed through the smart device, thereby showing the user a virtual object that "speaks" synchronously.
[0152] In the embodiment of the present disclosure, a lip response animation can be generated according to the lip shape change sequence and the model lip shape parameters; then, the lip response animation can be integrated with other model parameters of the virtual object to generate a virtual object voice response animation.
[0153] In the embodiments of the present disclosure, since the reply information is generated based on the user's operation on the virtual object, the reply information can be fed back to the user through a display method associated with the virtual object, so that the interaction process between the user and the virtual object is similar to the conversation method of a real shopping guide, thereby improving the user experience.
[0154] FIG5 schematically shows a flow chart of displaying virtual object display animation and reply information according to another embodiment of the present disclosure.
[0155] As shown in FIG. 5 , the method 500 for displaying a virtual object display animation and replying information includes operations S510 to S530 .
[0156] In operation S510, emotion information of the user is determined according to the user demand information and / or historical demand information.
[0157] In operation S520, a virtual object display animation matching the emotion information is determined, wherein the virtual object display animation indicates a display action change and / or a display expression change of the virtual object.
[0158] In operation S530, the reply message and the virtual object display animation are displayed simultaneously.
[0159] According to an embodiment of the present disclosure, in order to enable a virtual object to provide reply information to a user in a more humane manner, the virtual object may also have corresponding action changes and expression changes while displaying the reply information.
[0160] In a specific embodiment, a virtual object display animation, such as a smiling or waving animation, may be preset, and the reply message and the virtual object display animation may be displayed to the user at the same time.
[0161] In another specific embodiment, the user's emotional information can also be determined based on the user demand information and / or historical demand information, and a virtual object display animation matching the emotional information can be determined, so that the generated virtual object display animation is more in line with the user's current emotional needs.
[0162] Specifically, the emotion information of the user can be determined from the user demand information by using emotion recognition technology, or the emotion information of the user can be determined by combining historical demand information and user demand information. There is a pre-set matching relationship between the emotion information and the virtual object display animation. For example, when the emotion information is happy, the virtual object display animation can be jumping or laughing. When the emotion information is unhappy, the virtual object display animation can be smiling.
[0163] The embodiments of the present disclosure determine the user's current emotional information through user demand information and / or historical demand information, and match the appropriate virtual object display animation and reply information to present them to the user, which can make the virtual object more humane to meet the user's emotional needs.
[0164] When the embodiments of the present disclosure are applied to an online shopping platform, the virtual object can serve as an intelligent shopping guide to accompany the user throughout the entire shopping process. Through the interaction between the user and the intelligent shopping guide, and the underlying large model and toolbox for question analysis and response, the user's online shopping mode can be upgraded from only being able to purchase low-threshold and highly standardized goods on the platform to having the platform's experienced intelligent shopping guide analyze and disassemble their own needs, and then support the user's non-standardized high-threshold goods, and provide auxiliary shopping decisions for the user at any time.
[0165] In addition, virtual objects can interact with users as smart shopping guides under spatial computing logic. Unlike ordinary flat rendering displays, virtual objects based on spatial computing will have depth of field information, which can create a feeling that the virtual objects are really around the user.
[0166] Figure 6 A block diagram of a virtual object-based interaction device according to an embodiment of the present disclosure is schematically shown.
[0167] like Figure 6 As shown, the virtual object-based interaction device 600 includes a first acquisition module 610 , an interaction detection module 620 , a second acquisition module 630 , a generation module 640 and a display module 650 .
[0168] The first acquisition module 610 is used to obtain the posture information of the user's operating part in the real world, wherein the posture information is determined based on the image taken by the smart device. The smart device is also used to display the virtual world to the user, and the virtual world includes a virtual screen and virtual objects belonging to the same application software as the virtual screen.
[0169] The interaction detection module 620 is used to determine the interaction detection result for the virtual object based on the posture information.
[0170] The second acquisition module 630 is configured to acquire user demand information in response to determining that the interaction detection result represents a triggering operation on the virtual object, wherein the user demand information includes demand information for the virtual screen.
[0171] The generating module 640 is used to generate reply information matching the user demand information based on the user demand information.
[0172] The display module 650 is used to display the reply information.
[0173] According to an embodiment of the present disclosure, the interaction detection module 620 includes a first posture detection unit and a second posture detection unit. The posture information includes first posture information.
[0174] The first posture detection unit is used to determine that the interaction detection result is triggering an operation on the virtual object when it is determined that the first posture information matches the preset posture information.
[0175] The second posture detection unit determines that the interaction detection result is that the operation on the virtual object is not triggered when it is determined that the first posture information does not match the preset posture information.
[0176] According to an embodiment of the present disclosure, the interaction detection module 620 further includes a position detection unit and an interaction detection unit. The posture information also includes the first position information
[0177] The position detection unit is used to determine a position detection result based on the first position information and the second position information of the virtual object, wherein the position detection result represents whether the operating part in the real world contacts the virtual object in the virtual world, and the second position information represents the position information of the virtual object in the real world.
[0178] The interaction detection unit is used to determine that the interaction detection result is a trigger for operating the virtual object when the position detection result indicates that the operation part is in contact with the virtual object and the first posture information matches the preset posture information.
[0179] According to an embodiment of the present disclosure, the virtual object-based interaction device 600 further includes: a first position determination module and / or a second position determination module.
[0180] The first position determination module is used to determine the second position information of the virtual object according to the third position information of the virtual picture.
[0181] The second position determination module is used to obtain the position and posture information of the smart device, wherein the position and posture information of the smart device includes fourth position information and second posture information; the second position information is determined according to the fourth position information, the second posture information and the user's arm span distance; the user's arm span distance represents the farthest distance when the user spreads his arms.
[0182] According to an embodiment of the present disclosure, the generation module 620 includes an intention determination unit and a generation unit. The intention determination unit is used to determine the user intention based on the user demand information. The generation unit is used to generate reply information matching the user demand information according to the user intention.
[0183] According to an embodiment of the present disclosure, the intention determination unit includes an acquisition subunit and a first determination subunit.
[0184] The acquisition subunit is used to acquire historical demand information and current screen content of the virtual screen, wherein the current screen content includes at least one screen content corresponding to each of multiple shopping functions in the shopping platform.
[0185] The first determination subunit is used to determine the user intention based on the user demand information, historical demand information and current screen content by using a large language model.
[0186] According to an embodiment of the present disclosure, the generation unit includes a second determination subunit and a generation subunit. The second determination subunit is used to determine a reply strategy that matches the user's intention. The generation subunit is used to generate reply information based on the user's demand information and the reply strategy using a large language model.
[0187] According to an embodiment of the present disclosure, the display module 630 includes a first display unit or a second display unit. The first display unit is used to display the reply information in text form to the user through a display window in the virtual world, wherein the position information of the display window is determined according to the second position information of the virtual object. The second display unit is used to convert the reply information into voice data and generate a lip shape change sequence matching the voice data; generate a virtual object voice reply animation according to the lip shape change sequence and the model parameters of the virtual object; output the voice data through the smart device, and synchronously display the virtual object voice reply animation in the virtual world.
[0188] According to an embodiment of the present disclosure, the virtual object-based interaction device 600 includes an emotion determination module, an animation determination module and an animation display module.
[0189] The emotion determination module is used to determine the user's emotion information based on the user demand information and / or historical demand information. The animation determination module is used to determine the virtual object display animation that matches the emotion information, wherein the virtual object display animation indicates the display action change and / or display expression change of the virtual object. The animation display module is used to display the reply information and the virtual object display animation at the same time.
[0190] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.
[0191] For example, any multiple of the acquisition module 610, the generation module 620, and the display module 630 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the generation module 620, and the display module 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 610 , the generation module 620 , and the presentation module 630 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.
[0192] It should be noted that the description of the device part in the embodiments of the present disclosure specifically refers to the method part, which will not be repeated here.
[0193] Figure 7 A block diagram of an electronic device suitable for implementing a virtual object-based interaction method and apparatus according to an embodiment of the present disclosure is schematically shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0194] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0195] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0196] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.
[0197] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.
[0198] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0199] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.
[0200] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .
[0201] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains a program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the virtual object-based interaction method provided by the embodiment of the present disclosure.
[0202] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0203] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0204] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0206] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A virtual object-based interaction method, comprising: Acquiring position information of an operating part of a user in the real world, wherein the position information is determined based on an image captured by a smart device, and the smart device is further used to display a virtual world to the user, wherein the virtual world includes a virtual screen and a virtual object belonging to the same application software as the virtual screen; Determining an interaction detection result for the virtual object based on the posture information; In response to determining that the interaction detection result represents triggering an operation on the virtual object, obtaining user demand information, the user demand information including demand information for the virtual screen; Based on the user demand information, generating reply information matching the user demand information; and The reply information is displayed.
2. The method according to claim 1, wherein: The posture information includes first posture information; and determining the interaction detection result for the virtual object based on the posture information includes: In a case where it is determined that the first posture information matches the preset posture information, determining that the interaction detection result is triggering an operation on the virtual object; When it is determined that the first posture information does not match the preset posture information, it is determined that the interaction detection result is that the operation on the virtual object is not triggered.
3. The method according to claim 2, wherein: The posture information further includes first position information, and determining the interaction detection result for the virtual object based on the posture information further includes: Determine a position detection result according to the first position information and the second position information of the virtual object, wherein the position detection result indicates whether the operation part in the real world contacts the virtual object in the virtual world, and the second position information indicates the position information of the virtual object in the real world; When the position detection result indicates that the operation part is in contact with the virtual object and the first posture information matches the preset posture information, it is determined that the interaction detection result triggers an operation on the virtual object.
4. The method according to claim 3, further comprising: Determining the second position information of the virtual object according to the third position information of the virtual screen; or, Obtaining posture information of the smart device, wherein the posture information of the smart device includes fourth position information and second posture information; determining the second position information according to the fourth position information, the second posture information and the user arm span distance; the user arm span distance represents the farthest distance when the user spreads his arms.
5. The method according to claim 1, wherein: The generating, based on the user demand information, reply information matching the user demand information comprises: Determining user intent based on the user demand information; and According to the user intention, reply information matching the user demand information is generated.
6. The method according to claim 5, wherein: The determining the user intention based on the user demand information includes: Acquire historical demand information and current screen content of the virtual screen, wherein the current screen content includes at least one screen content corresponding to each of a plurality of shopping functions in the shopping platform; The user intention is determined by using a large language model according to the user demand information, historical demand information and the current screen content.
7. The method according to claim 5, wherein: The generating, according to the user intention, reply information matching the user demand information comprises: Determining a response strategy that matches the user's intent; The reply information is generated by utilizing the large language model based on the user demand information and the reply strategy.
8. The method according to claim 1, wherein: The displaying of the reply information includes: displaying the reply information in text form to the user through a display window in the virtual world, wherein the position information of the display window is determined according to the second position information of the virtual object; or The reply information is converted into voice data, and a lip shape change sequence matching the voice data is generated; a virtual object voice reply animation is generated according to the lip shape change sequence and the model parameters of the virtual object; the voice data is output through the smart device, and the virtual object voice reply animation is synchronously displayed in the virtual world.
9. The method according to claim 1 or 8, further comprising: Determining the user's emotion information according to the user demand information and / or historical demand information; Determining a virtual object display animation that matches the emotion information, wherein the virtual object display animation indicates a display action change and / or a display expression change of the virtual object; The reply information and the virtual object display animation are displayed simultaneously.
10. An interactive device based on a virtual object, comprising: A first acquisition module is used to acquire the posture information of the user's operating part in the real world, wherein the posture information is determined according to the image taken by the smart device, and the smart device is also used to display a virtual world to the user, wherein the virtual world includes a virtual screen and a virtual object belonging to the same application software as the virtual screen; An interaction detection module, configured to determine an interaction detection result for the virtual object based on the posture information; A second acquisition module is configured to acquire user demand information in response to determining that the interaction detection result represents triggering an operation on the virtual object, wherein the user demand information includes demand information for a virtual screen; A generating module, configured to generate, based on the user demand information, reply information matching the user demand information; and A display module is used to display the reply information.
11. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.
12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 9.
13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.
Citation Information
Cited By
Method and apparatus for performing interaction on basis of virtual object, and electronic device and medium
WO2026026336A1
