Interaction method and apparatus

Abstract

According to embodiments of the present disclosure, an interaction method, device, electronic device and computer storage medium are provided. The method described herein includes: presenting a decision curve associated with target audio in an interaction interface, the shape of the decision curve being determined based on the waveform of the target audio, and the decision curve moving with the playing of the target audio; determining a set of interaction positions corresponding to a set of predetermined time instants based on user interaction; and generating an evaluation for the user interaction based on the comparison of the set of interaction positions and the decision curve. In this way, the interactive form of the application can be enriched, and the user experience can be improved.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to an interaction method, an interaction apparatus, an electronic device, and a computer-readable storage medium. BACKGROUND

[0002] With the development of computer technology, various types of interaction applications have emerged. For example, in many fields such as games, simulations, virtual reality, etc., an electronic device can provide a user with a rich virtual scene, and the user can interact with the virtual scene. For example, some music interactive applications (e.g., rhythm games) can determine the user's interaction performance based on user interaction, and then feed back the corresponding score. SUMMARY

[0003] In a first aspect of the present disclosure, an interaction method is provided. The method comprises: presenting, at an interaction interface, a decision curve associated with target audio, the shape of the decision curve being determined based on a waveform of the target audio, and the decision curve moving with the playing of the target audio; determining, based on user interaction, a set of interaction positions corresponding to a set of predetermined time instants; and generating, based on a comparison of the set of interaction positions and the decision curve, an evaluation for the user interaction.

[0004] In a second aspect of the present disclosure, an interaction apparatus is provided. The apparatus comprises: a presentation module configured to present, at an interaction interface, a decision curve associated with target audio, the shape of the decision curve being determined based on a waveform of the target audio, and the decision curve moving with the playing of the target audio; an interaction position determination module configured to determine, based on user interaction, a set of interaction positions corresponding to a set of predetermined time instants; and an evaluation generation module configured to generate, based on a comparison of the set of interaction positions and the decision curve, an evaluation for the user interaction.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program, which is executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the description in the SUMMARY is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other features, aspects, and advantages of various embodiments of the present disclosure will become more apparent with reference to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0009] FIG. 1 A block diagram illustrating an example environment in which some embodiments of the present disclosure can be implemented is shown;

[0010] FIG. 2 A flow diagram illustrating an interaction process according to some embodiments of the present disclosure is shown;

[0011] FIGS. 3A-3D A schematic diagram illustrating an interaction interface example according to some embodiments of the present disclosure is shown;

[0012] FIG. 4 A schematic diagram illustrating an interaction interface example according to some embodiments of the present disclosure is shown;

[0013] FIG. 5 A schematic diagram illustrating an interaction in an extended reality scenario according to some embodiments of the present disclosure is shown;

[0014] FIG. 6 A schematic structural block diagram of an interaction device according to some embodiments of the present disclosure is shown; and

[0015] FIG. 7 A block diagram of a computing device capable of implementing a number of embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of embodiments of the present disclosure, the term "including" and its derivatives, shall be understood as "including, but not limited to". The term "based on" shall be understood as "based at least in part on". The term "one embodiment" or "an embodiment" shall be understood as "at least one embodiment". The term "first", "second", and the like can refer to different or identical objects. Other explicit or implicit definitions can also be included below.

[0018] Currently, in many fields such as games, simulations, virtual reality, etc., the interaction between the user and the virtual scene mainly relies on visual presentation. For example, in a game scene, the user completes the corresponding action or task according to the prompt information, and drives the game process. In the above-mentioned scene, audio is only used as background music or to carry prompt information. Such interaction form is relatively single, which affects the scene immersion.

[0019] According to various embodiments of the present disclosure, an interaction scheme is proposed. For example, in the scheme, a determination region associated with audio is presented in an interaction interface. The determination region can be determined based on the waveform of the audio, and the presented determination region moves with the playing of the audio. In this way, the control of the audio can be realized through user interaction in a virtual environment. Further, the evaluation of the user interaction can be generated based on the interaction position in the interaction interface and the determination region.

[0020] According to the implementation of the present disclosure, the interaction form related to audio can be supported in a virtual environment, and the interaction evaluation can be provided. Additionally, the interaction difficulty and the interaction evaluation can be adapted to the attributes (e.g., melody fluctuation, pitch, rhythm) of the audio. Thus, rich interaction forms can be provided, the user experience can be improved, and the scene immersion can be enhanced.

[0021] In order to illustrate the principles and ideas of the embodiments of the present disclosure, some of the following descriptions will refer to the field of music interaction application. However, it will be understood that this is only exemplary and is not intended to limit the scope of the present disclosure in any way. Embodiments of the present disclosure can be applied to various fields of simulation, simulation, virtual reality (VR), augmented reality (AR), etc.

[0022] Example Environment

[0023] Firstly, refer to FIG. 1 which schematically shows a schematic diagram of an example environment 100 in which an example implementation according to the present disclosure can be implemented. As FIG. 1 shown, the example environment 100 can include an electronic device 10.

[0024] In some embodiments, the electronic device 10 can include a portable device of a suitable type. For example, the electronic device 10 can support the user's two hands to hold or wear for various interactive operations. Such electronic device 10 can include, but is not limited to, a smartphone, a tablet computer, a palm computer, a portable game terminal, a VR or AR device, etc.

[0025] In some embodiments, the electronic device 10 can include appropriate types of sensors for detecting user interactions. For example, the electronic device 10 can include a touch screen for detecting various gestures made by the user on the touch screen. Such gestures include, but are not limited to, tapping, swiping, scrolling, rotating, pinching, and the like. Additionally or alternatively, the electronic device 10 can also include any other types of sensing devices for detecting user interactions, such as proximity sensors, acceleration sensors, inertial measurement systems, and the like.

[0026] It should be appreciated that the electronic device 10 is shown as a portable device in FIG. 1 for illustrative purposes only. In other embodiments, the electronic device 10 can also be other appropriate forms of devices. For example, the electronic device 10 can include a display device for display and a computing device for computation, and the display device and the computing device can be physically coupled or can also be physically separated. Thus, the scope of the present disclosure is not limited in this respect.

[0027] As an example implementation, the electronic device 10 can include a display screen for presenting an interactive interface and a game console for screen rendering and game control. In such a scenario, the electronic device 10 can implement the interaction, for example, with other appropriate input devices. For example, the electronic device 10 can implement the interaction through a communicatively coupled keyboard, mouse, joystick, gamepad, and the like interaction devices.

[0028] With continued reference to FIG. 1 , the electronic device 10 can be installed with an application 104, such as a game application, an AR application, a VR application, and the like. In such embodiments, the electronic device 10 can present an interactive interface 110 of the application 104 to the user 102. For example, the interactive interface 110 can be a virtual scene provided by the application 104.

[0029] For illustrative purposes, in FIG. 1 example, the interactive interface 110 is shown as a music interactive application scenario. For example, the music interactive application can involve music performance, band rehearsal, and the like. In such a scenario, a decision curve 120 associated with the audio can be presented and the decision curve 120 can move in the interactive interface 110 as the audio is played. The interactive interface 110 can include an interaction control 130. The user 102 can manipulate the interaction control 130 (e.g., activate the control 130 by a swiping gesture) to control the position of a decision point 132 in the virtual scene. Additionally, the electronic device 10 can also generate an evaluation, such as a score, for the user interaction. The process regarding the user interaction and the interaction evaluation will be described in detail below.

[0030] It should be appreciated that the layout, functionality, and number of elements in environment 100 are described for illustrative purposes only, and various designs can exist in practice. Embodiments of the present disclosure can also be applied to environments having different structures and / or functionality. The scope of the present disclosure is not limited in this regard.

[0031] The interactive processes and details will be described in greater detail below with continued reference to the accompanying drawings. In addition, embodiments will also be described below with reference to example states of several UIs. It should be appreciated that these UIs and interactions are merely illustrative, and various interface designs and ways of interaction can exist in practice. In addition, the controls included in these UIs can be implemented with any currently known or future developed UI elements and technologies. Furthermore, the types, forms, ways of operation, layouts, arrangements, etc. of these controls are illustrative, and are not intended to limit the scope of the present disclosure in any way.

[0032] Example Process

[0033] FIG. 2 A flowchart of an interactive process 200 according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 120 of FIG. 1 For ease of discussion, process 200 will be described in conjunction with FIG. 1

[0034] For example, process 200 can be a process in which user 102 interacts with music interactive application 104 via electronic device 10 and obtains an interactive evaluation. However, it should be appreciated that the music interactive scenario is merely one of the application scenarios of process 200, and embodiments of the present disclosure can be applied to various fields of simulation, simulation, virtual reality, augmented reality, etc. Therefore, the scope of the present disclosure is not limited in this regard.

[0035] At block 210, electronic device 10 presents a decision curve associated with the target audio at interactive interface 110. The shape of the decision curve is determined based on the waveform of the target audio, and the decision curve moves with the playing of the target audio. For example, the decision curve moves from right to left while the audio is playing. The target audio can include, but is not limited to, a song, a piece of music, a melody, a rhythm, etc.

[0036] The interactive interface of electronic device 10 will be described below with reference to example interface 300 shown in FIG. 3A FIG. 3A ​​As shown, in the interactive interface 300, a judgment curve 310 is presented as the target audio is played. In some embodiments, the judgment curve 310 may be a band-shaped area with width. Furthermore, the degree of fluctuation of the judgment curve may correspond to the melody fluctuations, pitch, and tempo of the target audio. Additionally or alternatively, the degree of fluctuation and width of the judgment curve 310 may be associated with the difficulty of the interaction. Therefore, in some embodiments, the degree of fluctuation and width of the judgment curve 310 may be adaptively modified according to the difficulty.

[0037] The interactive interface may include an interface presented by a display screen, an augmented reality (AR) interface, a virtual reality (VR) interface, and so on. In some embodiments, the electronic device 10 may receive user interactions with the interactive interface. For example, in an embodiment where the interface is presented by a display screen, the user 102 may interact by clicking, swiping, or touching the display screen with a finger or stylus.

[0038] Additionally or alternatively, in embodiments where the interactive interface is an AR or VR interface, user 102 can interact with electronic device 10 by manipulating electronic device 10 or a wearable device associated with electronic device 10. For example, user 102 can interact by waving a handle or moving their head equipped with a head-mounted display.

[0039] like FIG. 3A As shown, the electronic device 10 can detect user interaction via the interactive control 320. For example, user 102 can manipulate the interactive control 320 to slide in any direction within the interactive interface 300. The interactive interface 300 also presents a decision point 322, whose position within the interactive interface 300 changes with user interaction. For example, in FIG. 3A In the example, decision point 322 is aligned with the position of control 320 in the horizontal direction, and in response to user interaction in the vertical direction (e.g., swiping up or down), decision point 322 also moves accordingly in the vertical direction.

[0040] Alternatively or additionally, the interactive interface 300 may also display interaction points 330 to 335 corresponding to predetermined times T0 to T5. In some embodiments, interaction points 330 to 335 may be determined based on the melody fluctuations, pitch, tempo, etc., of the target audio. Therefore, interaction points 330 to 335 may correspond to the fluctuations of the decision curve 310 over time. To enhance the interactive experience, interaction points 330 to 335 may be presented as indicator elements, such as avatars corresponding to virtual characters.

[0041] Alternatively or additionally, the interface 300 may also present a decision line 340, which may be located at a predetermined position within the interface 340. As an example implementation, the decision point 322 may be fixed horizontally at the decision line 340 and move synchronously in the vertical direction with user interaction.

[0042] In box 220, electronic device 10 determines a set of interactive positions corresponding to a set of predetermined times based on user interaction.

[0043] For example, in music-related interactive applications, the electronic device 10 can determine the corresponding interaction position at times T0 to T5 when the interaction points 330 to 335 pass through the decision line 340 as the audio is played. Furthermore, the determined interaction position can be used to indicate the position of the decision point 322 at that predetermined time.

[0044] In box 230, electronic device 10 generates an evaluation of user interaction based on a comparison of a set of interaction locations with judgment curves. For example, the evaluation of user interaction may include, but is not limited to, scores, ratings, etc. FIG. 3A In the example, the rating is the current score of the user interaction, and the rating 350 is displayed as "Current score: 200" in the interaction interface 300.

[0045] In music-related interactive applications, at a predetermined time, if the judgment point 322 is determined to be within the range of the judgment curve 310 at that predetermined time based on a comparison between the interaction position and the judgment curve 310, a first preset evaluation can be provided to the user 102, such as a score indication, a positive indication based on text, audio, or special effects, etc. If the judgment point 322 is determined to be outside the range of the judgment curve 310 at that predetermined time based on a comparison between the interaction position and the judgment curve 310, a second preset evaluation can be provided to the user 102, such as a deduction indication, a negative indication based on text, audio, or special effects, etc.

[0046] The following is for reference FIG. 3B The example interface 301 shown illustrates the process of generating evaluations based on user interactions. FIG. 3B In the example, the first interaction position in a set of interaction positions can correspond to the interaction position at the first time T0. Regarding the first interaction position, the electronic device 10 can determine a first offset of the first interaction position at the first time T0 relative to the interactive interface 301. For example... FIG. 3B As shown, the first offset can be the offset d0 of the first interaction position in the vertical direction from the top of the interface. Of course, in other embodiments, the first offset can be the offset of the first interaction position relative to other reference lines or reference points on the interaction interface.

[0047] Additionally, the electronic device 10 can determine the target offset range in the determination curve 310 corresponding to the first time point T0. For example... FIG. 3B As shown, the target offset range can be the height range D0 of the decision curve 310 relative to the decision line 340 at the first time T0. Subsequently, the electronic device 10 can generate a first interaction evaluation for the first interaction position based on a comparison between the first offset and the target offset range. For example, the electronic device 10 can compare d0 with the endpoint of D0 to determine whether the first interaction position falls within the target offset range.

[0048] Depending on the comparison results, the first interaction evaluation may differ. For example, if the first offset falls within the target offset range, the electronic device 10 can determine the first interaction evaluation as the first preset evaluation. FIG. 3B As shown, at time T0, if the first offset falls within the target offset range, then the first interaction evaluation for the first interaction position is generated as the first preset evaluation, i.e., the score indication 341 "+10". Additionally or alternatively, in this case, the evaluation 350 for the user interaction can be changed; for example, the current score may be changed from... FIG. 3A The "200" shown has been increased to FIG. 3B "210" shown.

[0049] Conversely, if the first offset does not fall within the target offset range, the electronic device 10 can determine the first interaction evaluation as the second preset evaluation. For example... FIG. 3C As shown, at time T0, the first offset of the first interaction position 324 relative to the interaction interface 302 is d0', and the target offset range is D0. Based on the endpoints of the first offset d0' and the target offset range, the first offset d0' is determined not to fall within the target offset range. Accordingly, the decision point 326 is outside the range of the decision curve 310 at time T0. In this case, the first interaction evaluation for the first interaction position 324 is generated as a second preset evaluation, i.e., a negative indication 342 "missed". Additionally or alternatively, in this case, the evaluation 350 for the user interaction can be changed, for example, the current score can be changed from... FIG. 3A The "200" shown is reduced to FIG. 3C "190" shown.

[0050] In some embodiments, if the offsets corresponding to a predetermined number of consecutive interaction positions in a set of interaction positions all fall within the corresponding offset range, the electronic device 10 can generate a motivational evaluation. Furthermore, the electronic device 10 can generate an evaluation of the user interaction based at least on the motivational evaluation and a set of interaction evaluations for the set of interaction positions.

[0051] For example, incentive evaluations can include bonus points, special effects, etc. The predetermined number can be adaptively modified according to the difficulty of the interaction.FIG. 3D For example, a set of interaction positions includes at least interaction positions 331 to 339 corresponding to predetermined times T1 to T9, and the predetermined number is 4. At times T1 to T4, the decision point 322 corresponding to interaction position 320 falls within the target offset range of the decision curve 310 relative to the decision line 340. In this case, the electronic device 10 can generate an incentive evaluation 343, for example, adding a score indication "Continuous Correct: 40". Based on the incentive evaluation 343 and the interaction evaluations of interaction positions 331 to 334, a user interaction evaluation 350 "Current Score 280" can be generated.

[0052] It should be understood that as the number of consecutive interactive positions corresponding to the offsets falling within the corresponding offset range increases, the generated incentive evaluation can also change accordingly; for example, the additional score increases proportionally or exponentially.

[0053] In some embodiments, an indicator element indicating that a determination will be made at a predetermined time may also be displayed in the interactive interface. The electronic device 10 may change the display style of the indicator element based on a comparison between the first offset and the target offset range.

[0054] like FIG. 3A As shown, at time T0, interaction point 330 reaches decision line 340, and the indicator element of interaction point 330 jumps from its initial position to a predetermined position. This prompts user 102 to perform an operation. Furthermore, the indicator element of interaction point 330 is presented as a blank human head. In response to the first offset falling within the target offset range at time T0, as... FIG. 3B As shown, the display style of the indicator element at interaction point 330 changes from an expressionless character portrait to a smiling character portrait. Conversely, in response to the first offset not falling within the target offset range at time T0, as... FIG. 3C As shown, the display style of the indicator element at interaction point 330 changes from an expressionless character avatar to a surprised character avatar.

[0055] In some embodiments, the electronic device 10 can control the playing of the target audio at the first time instance based on a comparison of the first offset with the target offset range. For example, if the first offset falls within the target offset range, the electronic device 10 can play the audio signal of the target audio corresponding to the first time instance. For example, in the case that the operation of the user 102 at the interaction point is correct, the audio can be played normally. In contrast, if the first offset does not fall within the target offset range, the electronic device 10 can adjust the pitch of the audio signal for playing based on the distance between the first offset and the target offset range. For example, in the case that the operation of the user 102 at the interaction point is incorrect, the pitch can be adjusted to indicate the operation is inaccurate. As an example, if the endpoint of the first offset exceeds the upper limit of the target offset range, the pitch of the audio signal can be raised. If the endpoint of the first offset is lower than the lower limit of the target offset range, the pitch of the audio signal can be lowered.

[0056] Additionally or alternatively, more than one user interaction can be included. As an example, the more than one user interaction can include user interactions from multiple users. For example, in the music interactive application scenario of band rehearsal, a first user interaction can be from a first user and correspond to drum sound, a second user interaction can be from a second user and correspond to guitar, and so on.

[0057] Additionally or alternatively, the more than one user interaction can include multiple user interactions from the same user, which can correspond to different audio effects. As an example, in the music interactive application scenario of band rehearsal, a first user interaction can correspond to drum sound, a second user interaction can correspond to applause, and so on. Further, the multiple user interactions can be triggered by different fingers or different body parts of the same user.

[0058] In some embodiments, the user interaction can be a first user interaction, and the electronic device 10 can further receive a second user interaction. The electronic device 10 can determine a set of interaction time instances corresponding to the second user interaction. In turn, the electronic device 10 can generate an evaluation for the user interaction based on a first comparison of the set of interaction time instances with the set of predetermined time instances and a second comparison of the set of interaction locations with the decision curve. Additionally or alternatively, in some embodiments, the evaluation for the user interaction can be generated based on one of the first comparison and the second comparison.

[0059] It should be appreciated that the interaction time instances can be different from the predetermined time instances, or the two can completely or partially overlap.

[0060] In FIG. 4In the example, a set of predetermined times includes T0 to T5. The first user interaction 420 may correspond to the user's right-hand interaction and is used to control the decision point 422. The second user interaction 450 may correspond to the user's left-hand interaction, and the interaction times corresponding to the second user interaction 450 include T0, T2, and T4, which correspond to interaction points 452, 454, and 456. As an example implementation, the electronic device 10 may generate an evaluation, for example, based on whether the interaction time of the second user interaction is close to the predetermined time, and / or whether the interaction position of the second user interaction falls within the target offset range of the decision curve. For example, in response to the second user interaction's interaction time being close to time T0, the electronic device 10 may generate an incentive evaluation 460. In this case, the electronic device 10 may generate a user interaction evaluation 470 based at least on the interaction evaluation 440 and the incentive evaluation 460.

[0061] In some embodiments, the electronic device 10 may also determine difficulty information associated with the evaluation. Furthermore, based on the difficulty information, the electronic device 10 may determine at least one of the following: the number of preset moments, the distribution of the preset moments, the degree of fluctuation of the judgment curve, and the width of the judgment curve.

[0062] In some embodiments, users can also implement the interaction processes discussed above in XR (Extended Reality) scenarios such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

[0063] FIG. 5 Schematic diagrams illustrating interactions in an XR scene according to some embodiments of the present disclosure are shown. FIG. 5 As shown, user 520 may wear XR device 520. XR device 520 can be a head-mounted or wearable near-eye display device, such as a head-mounted display, smart glasses, etc., supporting technologies such as VR, AR, and MR. XR device 520 may include image generation components and optical display components for providing images in a monocular or binocular field of view, such as… FIG. 5 The interactive interface shown is 540.

[0064] Such an interactive interface 540 can, for example, be part of an immersive virtual scene. For instance, such an interactive interface 540 can be a window within a virtual reality scene.

[0065] In some embodiments, similar to the above process based on the touch screen, the XR device 510 can detect the position change of the object 530 in the air, and detect the user interaction according to the position change. Accordingly, the XR hand 510 can determine the interaction position 550 in the interface 540 corresponding to the user interaction. Such object 530 may, for example, include a finger of the user, or a suitable type of manipulator (e.g., a manipulation joystick, etc.) held by the hand of the user. It should be understood that any suitable technology such as image recognition, sensor sensing, etc. can be employed to locate the object 530, and the present disclosure is not intended to be limited in this regard.

[0066] Taking the example that the user 520 interacts with the hand, the user 520 may, for example, make a gesture action of waving up and down in the air following the decision curve 560 shown in the interface 540. Such gesture action may be similar to the conducting action of a conductor, thereby bringing a more realistic interaction experience to the user.

[0067] Accordingly, the interaction position 550 in the interface 540 corresponding to the gesture action of the user 520 will move up and down, and can be compared with the decision curve 560 in the manner introduced above to obtain the interaction evaluation of the interaction of the user 520. The process of comparison with the decision curve 560 and determination of the interaction evaluation can refer to the part discussed above, and will not be repeated here.

[0068] In some embodiments, the XR device 510 can also provide multiple interaction interfaces in the virtual scene. Each interaction interface may, for example, have a similar arrangement as the interaction interface 540. In some embodiments, such multiple interaction interfaces may correspond to different tracks of the target audio, for example, the tracks corresponding to different musical instruments in the target audio.

[0069] In some embodiments, the user 520 can utilize different objects to respectively interact with different target interfaces. For example, the user 520 can utilize the left hand or a manipulator held by the left hand to interact with the interaction interface corresponding to the guitar track in the target audio, and utilize the right hand or another manipulator held by the right hand to interact with the interaction interface corresponding to the piano track in the target audio. In this way, the embodiments of the present disclosure can further enrich the interaction experience of the user.

[0070] In some embodiments, the interaction interfaces corresponding to different tracks in the virtual scene may, for example, also have different depths of field relative to the user 520. The user 520 may, for example, select to switch between multiple interaction interfaces having different depths of field to select the interaction interface currently desired to interact with.

[0071] For example, some audio has different main sound-producing instruments at different time periods. Such an interactive interface can also automatically switch, for example, according to the distribution of audio tracks in the target audio. For example, user 510 can need to interact with the interactive interface corresponding to the piano at the first time portion of the target audio, at which time the interactive interface corresponding to the guitar, for example, can have a greater depth of field and be in an inactive state. Further, when proceeding to the second time portion of the target audio, which can have the guitar as the main sound-producing instrument, the interactive interface corresponding to the guitar can be automatically switched to have a smaller depth of field to receive the interaction of user 520. Accordingly, the interactive interface corresponding to the piano, for example, can be automatically adjusted to have a greater depth of field.

[0072] In some embodiments, embodiments of the present disclosure can also support multiple users to interact in the same virtual scene, for example. For example, another user can also wear an XR device that can enable the user to join the same virtual scene as user 520.

[0073] In a multi-user scenario, different users can have interaction permissions for different interfaces in the virtual scene, for example. For example, one user can have interaction permissions for the interactive interfaces corresponding to the piano and the guitar in the target audio, and another user can have interaction permissions for the interactive interfaces corresponding to the trumpet and the drum in the target audio. Thus, multiple users can coordinate in the virtual scene to complete the interaction of the same target audio.

[0074] In some embodiments, multiple users can also cooperatively interact with the same interactive interface, for example. For example, in addition to user 520, another user can also join the virtual scene through the network and interact with interface 540. In this case, separate interaction evaluations can be generated for individual users to achieve a scenario similar to a music competition. Alternatively, a final unified evaluation can also be generated for the interactions of multiple users respectively. FIG. 5

[0075] It should be understood that for multi-user interaction, multiple users can be in the same or different physical environments, and the present disclosure is not intended to be limited in this regard.

[0076] Based on the processes discussed above, embodiments of the present disclosure can support audio-related interaction forms in a virtual environment and provide interaction evaluations. Additionally, the interaction difficulty and the interaction evaluation can be adapted to the properties of the audio (e.g., melody fluctuation, pitch, rhythm). This can enrich the interactive forms of the application and improve the user experience.

[0077] Example Devices and Apparatus

[0078] ​Embodiments of the present disclosure also provide a corresponding apparatus for implementing the above method or process. FIG. 6 A schematic structural block diagram of an interaction apparatus 600 is shown according to some embodiments of the present disclosure.

[0079] As shown in FIG. 6 The apparatus 600 includes a presentation module 610 configured to present a decision curve associated with the target audio at an interaction interface, a shape of the decision curve being determined based on a waveform of the target audio, and the decision curve moving with a playing of the target audio. The apparatus 600 further includes an interaction position determination module 620 configured to determine a set of interaction positions corresponding to a set of predetermined time instants based on a user interaction. In addition, the apparatus 600 further includes an evaluation generation module 630 configured to generate an evaluation for the user interaction based on a comparison of the set of interaction positions with the decision curve.

[0080] In some embodiments, the interaction evaluation generation module 630 is further configured to, for a first interaction position in the set of interaction positions: determine a first offset of the first interaction position relative to the interaction interface at a first time instant; determine a target offset range in the decision curve corresponding to the first time instant; and generate a first interaction evaluation for the first interaction position based on a comparison of the first offset with the target offset range.

[0081] In some embodiments, generating the first interaction evaluation for the first interaction position comprises: determining the first interaction evaluation as a first preset evaluation in response to the first offset falling within the target offset range; or determining the first interaction evaluation as a second preset evaluation in response to the first offset not falling within the target offset range.

[0082] In some embodiments, the interaction evaluation generation module 630 is further configured to: generate an encouragement evaluation in response to offsets corresponding to a predetermined number of consecutive interaction positions in the set of interaction positions all falling within corresponding offset ranges; and generate the evaluation for the user interaction based on at least the encouragement evaluation and a set of interaction evaluations for the set of interaction positions.

[0083] In some embodiments, the apparatus 600 further includes an indication element presentation module configured to present an indication element for making a decision at the first time instant in the interaction interface; and a display style changing module configured to change a display style of the indication element based on the comparison of the first offset with the target offset range.

[0084] In some embodiments, the apparatus 600 further includes a playing control module configured to control a playing of the target audio at the first time instant based on the comparison of the first offset with the target offset range.

[0085] In some embodiments, the playback control module is configured to: in response to the first offset falling within the target offset range, play the audio signal corresponding to the first time instant in the target audio; or in response to the first offset not falling within the target offset range, adjust the pitch of the audio signal for playback based on a distance between the first offset and the target offset range.

[0086] In some embodiments, the user interaction is a first user interaction, and the evaluation generation module 630 is configured to: receive a second user interaction; determine a set of interaction time instants corresponding to the second user interaction; generate the evaluation for the user interaction based on a first comparison of the set of interaction time instants and the set of predetermined time instants and a second comparison of the set of interaction positions and the decision curve.

[0087] In some embodiments, the apparatus 600 further includes: a difficulty information determination module configured to determine difficulty information associated with the evaluation; and an attribute determination module configured to determine, based on the difficulty information, at least one of: a number of the set of preset time instants, a distribution of the set of preset time instants, a fluctuation degree of the decision curve, and a width of the decision curve.

[0088] In some embodiments, the interaction interface includes: an interface presented by a display screen; an augmented reality (AR) interface; or a virtual reality (VR) interface.

[0089] In some embodiments, the interaction interface includes a first interaction interface in a virtual scene constructed by an extended reality device.

[0090] In some embodiments, the virtual scene further includes a second interaction interface, the first interaction interface and the second interaction interface corresponding to different audio tracks of the target audio.

[0091] In some embodiments, the first interaction interface and the second interaction interface have different depths of field in the virtual scene.

[0092] In some embodiments, the first interaction interface is configured to receive the user interaction from a first user, and the second interaction interface is configured to receive the user interaction from a second user different from the first user.

[0093] In some embodiments, further including an interaction detection module configured to: detect a position change of a target object, the target object including a hand of the user and / or a manipulator held by the user; and determine the user interaction based on the position change.

[0094] The units included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 600 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0095] FIG. 7 A block diagram of an electronic device / server 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... FIG. 7 The electronic device / server 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0096] like FIG. 7 As shown, the electronic device / server 700 is in the form of a general-purpose computing device. Components of the electronic device / server 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 760, and one or more output devices 760. The processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the computing device / server 700.

[0097] The computing device / server 700 typically includes multiple computer storage media. Such media can be any available media accessible to the computing device / server 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within the computing device / server 700.

[0098] The electronic device / server 700 can further include additional detachable / non-detachable, volatile / non-volatile storage media. Although not shown in FIG. 7

[0099] The communication unit 740 enables communication with other computing devices over communication media. Additionally, the functionality of the components of the computing device / server 700 can be implemented in a single computing cluster or multiple computer machines that are capable of communicating over a communication connection. Thus, the computing device / server 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0100] The input device 750 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 760 can be one or more output devices, such as a display, a speaker, a printer, etc. The computing device / server 700 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc. through the communication unit 740, as needed, one or more devices that enable a user to interact with the computing device / server 700, or any devices (e.g., a network card, a modem, etc.) that enable the computing device / server 700 to communicate with one or more other computing devices. Such communication can be enabled by an input / output (I / O) interface (not shown).

[0101] According to an exemplary implementation of the present disclosure, a computer readable storage medium is provided, having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method described above.

[0102] Various aspects of the disclosure are now described with reference to the drawings. In general, the drawings are for illustration only and, as such, are not necessarily drawn to scale. Like numbers used in the figures refer to like components. The following description is ordered in three sections: (1) a description of the various aspects of the disclosure, (2) a description of the various aspects of the disclosure, and (3) a description of the various aspects of the disclosure.

[0103] ​These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all. The computer readable storage medium can also have other meanings inhered therein by those skilled in the art.

[0104] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0105] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various implementations of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (‘instructions’). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0106] implementations have been described above, the description is intended to be illustrative, and not restrictive, of the implementations disclosed. Many modifications and variations of the implementations described herein are possible and come within the scope of the implementations described. It is therefore to be understood that there is no intention to limit the disclosure to the specific implementations disclosed and that any one or more of the features can be implemented alone or in any combination thereof. The scope of the disclosure is to be determined by the following claims.

Claims

1. An interaction method, comprising: presenting, in an interaction interface, a decision curve associated with a target audio, a shape of the decision curve being determined based on a waveform of the target audio, and the decision curve moving with a play of the target audio, the decision curve being a band-shaped region having a width, the width being associated with a property of the target audio; determining, based on a user interaction, a set of interaction positions corresponding to a set of predetermined time instants; and generating, based on a comparison of the set of interaction positions with the band-shaped region of the decision curve corresponding to the set of predetermined time instants, an evaluation for the user interaction.

2. The method of claim 1, wherein generating the evaluation for the user interaction comprises: for a first interaction position in the set of interaction positions: determining a first offset of the first interaction position at a first time instant with respect to the interaction interface; determining a target offset range in the decision curve corresponding to the first time instant; generating a first interaction evaluation for the first interaction position based on a comparison of the first offset with the target offset range.

3. The method of claim 2, wherein generating the first interaction evaluation for the first interaction position comprises: in response to the first offset falling within the target offset range, determining the first interaction evaluation as a first preset evaluation; or in response to the first offset not falling within the target offset range, determining the first interaction evaluation as a second preset evaluation.

4. The method of claim 2, wherein generating the evaluation for the user interaction comprises: in response to offsets corresponding to a predetermined number of consecutive interaction positions in the set of interaction positions all falling within corresponding offset ranges, generating an engagement evaluation; and generating the evaluation for the user interaction based on at least the engagement evaluation and a set of interaction evaluations for the set of interaction positions.

5. The method of claim 2, further comprising: presenting, in the interaction interface, an indicating element for making a decision at the first time instant; and changing a display style of the indicating element based on the comparison of the first offset with the target offset range.

6. The method of claim 2, further comprising: controlling a play of the target audio at the first time instant based on the comparison of the first offset with the target offset range.

7. The method of claim 6, wherein controlling the play of the target audio at the first time instant comprises: in response to the first offset falling within the target offset range, playing an audio signal in the target audio corresponding to the first time instant; or in response to the first offset not falling within the target offset range, adjusting a pitch of the audio signal for playing based on a distance between the first offset and the target offset range.

8. The method of claim 1, wherein the user interaction is a first user interaction, and generating the evaluation for the user interaction comprises: receiving a second user interaction; determining a set of interaction time instants corresponding to the second user interaction; ​ ​ ​ ​ generating the evaluation for the user interaction based on a first comparison of the set of interaction time instants and the set of predetermined time instants and a second comparison of the set of interaction positions and the decision curve.

9. The method of claim 1, further comprising: determining difficulty information associated with the evaluation; and determining, based on the difficulty information, at least one of a number of the set of predetermined time instants, a distribution of the set of predetermined time instants, a degree of fluctuation of the decision curve, and a width of the decision curve.

10. The method of claim 1, wherein the interaction interface comprises: an interface presented by a display screen; an augmented reality (AR) interface; or a virtual reality (VR) interface.

11. The method of claim 1, wherein the interaction interface comprises a first interaction interface in a virtual scene constructed by an extended reality device.

12. The method of claim 11, wherein the virtual scene further comprises a second interaction interface, the first interaction interface and the second interaction interface corresponding to different audio tracks of the target audio.

13. The method of claim 12, wherein the first interaction interface and the second interaction interface have different depths of field in the virtual scene.

14. The method of claim 12, wherein the first interaction interface is configured to receive user interaction from a first user, and the second interaction interface is configured to receive user interaction from a second user different from the first user.

15. The method of claim 11, further comprising: detecting a position change of a target object, the target object comprising a hand of a user and / or a manipulator held by the user; and determining the user interaction based on the position change.

16. An interaction apparatus, comprising: a presentation module configured to present, at an interaction interface, a decision curve associated with target audio, a shape of the decision curve being determined based on a waveform of the target audio, and the decision curve moving with a play of the target audio, the decision curve being a band-shaped region having a width, the width being associated with a property of the target audio; an interaction position determination module configured to determine, based on a user interaction, a set of interaction positions corresponding to a set of predetermined time instants; and an evaluation generation module configured to generate, based on a comparison of the set of interaction positions and the band-shaped region corresponding to the decision curve, an evaluation for the user interaction.

17. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit causing the electronic device to perform the method according to any one of claims 1-15.

18. A computer-readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the method according to any one of claims 1-15. ​ ​

Images