Interaction method, electronic device, storage medium and computer program product

Abstract

The invention relates to an interaction method, electronic equipment, a storage medium and a computer program product, and relates to the technical field of computers. The interaction method comprises the following steps: displaying first multimedia content from a user and request information of the user based on the first multimedia content, wherein the first multimedia content comprises one or more actions; processing one or more actions in the first multimedia content based on the request information to determine second multimedia content and response information of the request information; and displaying the second multimedia content and the response information.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an interaction method, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the development of Internet technology, users can use mobile phones, computers and other electronic devices to learn and practice dance, yoga and other sports. For example, users can imitate the movements in other people's sports videos. Or, users can also participate in some online courses and connect with teachers through video, and the teachers can remotely guide the user's movements in real time. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] According to some embodiments of the present disclosure, an interaction method is provided, comprising: displaying first multimedia content from a user and request information of the user based on the first multimedia content, the first multimedia content including one or more actions; based on the request information, processing one or more actions in the first multimedia content to determine second multimedia content and response information of the request information; and displaying the second multimedia content and the response information.

[0005] According to some embodiments of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the interaction method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0006] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the interactive method of any embodiment described in the present disclosure is performed.

[0007] According to some embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, cause the processor to perform the interactive method of any embodiment described in the present disclosure.

[0008] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure. Each of the drawings together with the following detailed description is included in this specification and forms a part of the specification, and is used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure. In the drawings:

[0010] Figure 1 A flowchart of an interaction method according to some embodiments of the present disclosure is shown.

[0011] Figure 2 A schematic flow chart of a method for displaying second multimedia content and response information according to some embodiments of the present disclosure is shown.

[0012] Figure 3 A flowchart of a method for displaying second multimedia content and response information according to some other embodiments of the present disclosure is shown.

[0013] Figure 4 A flowchart of a method for displaying second multimedia content and response information according to some further embodiments of the present disclosure is shown.

[0014] Figure 5 A schematic structural diagram of an interaction device according to some embodiments of the present disclosure is shown.

[0015] Figure 6 A schematic structural diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0016] Figure 7 A schematic diagram of the structure of a computer system according to some embodiments of the present disclosure is shown.

[0017] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure, but it is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of the embodiments is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein.

[0019] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0020] The term "include" and its variations used in the present disclosure mean an open term that includes at least the following elements / features but does not exclude other elements / features, that is, "including but not limited to". In addition, the term "include" and its variations used in the present disclosure mean an open term that includes at least the following elements / features but does not exclude other elements / features, that is, "including but not limited to". Therefore, include is synonymous with include. The term "based on" means "based at least in part on".

[0021] References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, but may do so.

[0022] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0026] It should be understood that the present disclosure does not limit how to obtain the image to be applied / processed. In one embodiment of the present disclosure, it can be obtained from a storage device, such as an internal memory or an external storage device. In another embodiment of the present disclosure, a photographic component can be mobilized to shoot. It should be noted that the acquired image can be a captured image or a frame of an image in a captured video, and is not particularly limited to this.

[0027] In the context of the present disclosure, an image may refer to any of a variety of images, such as a color image, a grayscale image, etc. It should be noted that in the context of the present specification, the type of image is not specifically limited. In addition, the image may be any appropriate image, such as an original image obtained by a camera device, or an image that has been subjected to specific processing, such as preliminary filtering, anti-aliasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that the preprocessing operation may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0028] When users learn dance, yoga and other movements by themselves by referring to videos on the Internet, there may be non-standard movements, but it is difficult for users to detect and correct them by themselves, and the movements that beginners can learn are also relatively limited. Online teaching is difficult to initiate anytime and anywhere, and it is difficult for users to get guidance anytime and anywhere, such as correction of non-standard movements, or inspiration for adapting or expanding movements. Therefore, in the related art, it is difficult for users to efficiently obtain feedback on multimedia content containing movements.

[0029] In order to solve the above technical problems, the present disclosure provides an interactive method, an electronic device, a storage medium and a computer program product. Figure 1 An embodiment of the disclosed interactive method is described.

[0030] Figure 1 FIG. 1 is a flow chart showing an interactive method according to some embodiments of the present disclosure. Figure 1 As shown, the interaction method of this embodiment includes steps S102 to S106.

[0031] In step S102, first multimedia content from a user and user request information based on the first multimedia content are displayed, where the first multimedia content includes one or more actions.

[0032] The first multimedia content is image-based multimedia content, for example, it may be one or more images, or one or more videos. The first multimedia content may carry various actions, such as yoga actions, dance actions, fitness actions, martial arts actions, sports actions, and the like. These actions may be demonstrated by the user in step S102, or by other objects. That is, the first multimedia content may be an image or video of the user performing an action or actions in step S102, or an image or video of other objects (such as other objects, or virtual objects) performing an action or actions. The first multimedia content may be taken or uploaded locally by the user from the user device, or stored by the user in the cloud or server and specified by the user.

[0033] The content of the request information is associated with the first multimedia content, and can be text or voice. The user can input the request information through a text input control, a voice input control, or a selection control. The request information is, for example, an evaluation request, which indicates that the first multimedia content is evaluated in its entirety or in part; another example is a generation request, which indicates that new multimedia content is generated based on the first multimedia content. The generated multimedia content can be a continuation or extension of the first multimedia content. The request information can also include other content as needed, which will not be described here.

[0034] The action in the first multimedia content can be determined based on the posture of the object in the first multimedia content. In some embodiments, the object in the first multimedia content is identified; based on the posture of the object, one or more actions of the object in the first multimedia content are determined. For example, the object and the posture of the object can be determined by performing image recognition on the image in the first multimedia content. For example, the posture can be identified by using target detection and tracking algorithms, convolutional neural networks (CNN), etc., and then classified based on the posture. In addition, when the first multimedia content is a video or includes multiple images, the first multimedia content can be regarded as serialized data, and a model such as a recurrent neural network (RNN) that processes serialized data can be used to obtain a determination result of the action.

[0035] In step S104, based on the request information, one or more actions in the first multimedia content are processed to determine the second multimedia content and response information to the request information.

[0036] Based on the request information, a processing method for one or more actions in the first multimedia content may be determined. In some embodiments, it may also be determined based on the request information which action of the one or more actions in the first multimedia content is to be processed. The processing method may be to evaluate; or it may be to predict, for example, predicting other actions to generate new multimedia content.

[0037] The second multimedia content is a video or one or more images. The second multimedia content is related to the first multimedia content and also includes one or more actions. For example, it includes the same actions as the first multimedia content, or related actions. The type of the second multimedia content can be the same as the type of the first multimedia content. For example, when the first multimedia content is a video, the second multimedia content is a video; when the first multimedia content is an image, the second multimedia content is one or more images. Of course, as needed, the type of the second multimedia content and the first multimedia content may also be different.

[0038] The response information is used to explain or describe at least one of the first multimedia content and the second multimedia content, and can respond to the request information. In other words, the request information and the response information are based on the same subject. For example, if the request information indicates an evaluation of an action in the first multimedia content, the response information may include evaluation content combined with at least one of the first multimedia content and the second multimedia content; for another example, if the request information indicates generating the second multimedia content based on the action in the first multimedia content, the response information may include an interpretation of the generated second multimedia content. The response information may be text or voice.

[0039] In some embodiments, when the request information is voice, the text in the voice can be obtained by voice recognition first, and then processed using the text. Of course, when the model for processing the request information supports voice input, voice-to-text conversion may not be performed in advance.

[0040] In the process of determining the second multimedia content, a generative model can be used. For example, based on the request information, one or more generative models are used to process one or more actions in the first multimedia content to determine the second multimedia content and the response information of the request information. The generative model is used to output the target content based on the input information. The input information includes the processing basis of the generative model in the generation process, such as the request information (or the text corresponding to the request information), the requirements for the output content, etc. The generative model used in this application includes, for example, a model that generates text (or voice) and image (or video) based on text (or voice) and image (or video), that is, it can be a generative model that supports multimodal input and multimodal output. Of course, a generative model with multimodal input and single-modal output can also be used, or multiple types of models can be used. For example, the request information (or the text corresponding to the request information) and the first multimedia content are first processed using the first generative model to generate the second multimedia content; and then the second generative model is used to process the second multimedia content to generate the response information. In some embodiments, the second multimedia content may be generated using a generative adversarial network (GAN), a variational autoencoder (VAE), a policy gradient algorithm, a Q-learning algorithm, or the like.

[0041] In some embodiments, a deep learning model is used to construct an action prediction model to determine the action included in the second multimedia content. The model includes an input layer, a hidden layer, and an output layer. The input layer is used to receive feature vectors of music (optional) and action, which can be an action in the first multimedia content or a determined action in the second multimedia content to be generated; the hidden layer is used to learn the characteristics and laws of music (optional) and action; the output layer is used to generate prediction results of the action. During the model training process, a large amount of music and action data can be used as training data, and a back propagation algorithm can be used to optimize the parameters of the model. By continuously adjusting the parameters of the model, the model can better learn the characteristics and laws of music and action, thereby improving the accuracy of action prediction. Then, indicators such as accuracy, recall rate, and F1 value can be used to evaluate the performance of the model. Through the evaluation of the model, the advantages and disadvantages of the model can be understood, and the model can be further optimized and improved.

[0042] In step S106, the second multimedia content and the response information are displayed.

[0043] Steps S102 to S106 may all be executed by the user device, or steps S102 and S106 may be executed on the user device, and step S104 may be executed on other devices different from the user device and the execution result may be sent to the user device. The user device may be a tablet computer, a mobile phone (such as a folding screen mobile phone, a large screen mobile phone, etc.), a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, a high-definition TV, a 4K TV, a smart projector, and other Internet of Things (IOT) devices. The present disclosure does not impose any restrictions on the specific types of user devices.

[0044] In the above embodiment, the multimodal input from the user is processed to parse the first multimedia content based on the request information, generate a multimodal output and display it to the user, so as to respond to the user's request in two forms of second multimedia content and response information. Thus, the embodiment of the present disclosure can automatically parse the multimedia content including the action, and generate the corresponding multimedia content and response information, so that the user can more efficiently obtain the parsing results of the action in the first multimedia content, thereby improving the user's learning efficiency of yoga, dance and other actions.

[0045] In some embodiments, the processing method can be determined based on the subject of the request information. Figure 2 An embodiment of the present disclosure determining second multimedia content and response information is described.

[0046] Figure 2 FIG. 1 is a flow chart showing a method for displaying second multimedia content and response information according to some embodiments of the present disclosure. Figure 2 As shown, the method of this embodiment includes steps S202 to S204.

[0047] In step S202, the request information is subjected to intent recognition to determine the subject of the request information.

[0048] The subject of the request information reflects the key content of the request information. For example, the subject can be the category to which the request information belongs, the keywords in the request information, the refinement of the content of the request information, etc. The subject of the request information can be determined from several preset subjects or can be freely generated.

[0049] In some embodiments, the request information may be processed using a topic analysis model to obtain its topic; or, the request information may be processed using a classification model to obtain the category to which the request information belongs as the topic. In some embodiments, the topic of the request information may also be determined using a generative model. For example, the request information and the processing instructions are output to a generative model for determining the topic, and the processing instructions are used to instruct the generative model to determine the topic of the request information, and the result returned by the generative model is the topic.

[0050] In step S204, based on the subject of the request information, one or more actions in the first multimedia content are processed to determine the second multimedia content and response information of the request information.

[0051] Different themes may correspond to the same processing method, but may affect the processing basis in the process of generating the second multimedia content and the response information. For example, in the process of using one or more generative models for generating the second multimedia content and the response information, the theme (or the processing instructions determined based on the theme) and one or more actions are input into the generative model, so that different themes may affect the generation result.

[0052] Different topics may also correspond to different processing methods. For example, for several known topics, generative models corresponding to each topic may be set respectively. When generating the second multimedia content and the response information, the generative model corresponding to the topic of the request information is selected to process one or more actions.

[0053] Through the above embodiments, based on the subject of the request information, one or an action can be processed corresponding to the subject, so that the generated second multimedia content and response information can accurately respond to the user, thereby improving information acquisition efficiency and learning efficiency.

[0054] The present disclosure describes the two scenarios of "evaluation" and "arranging actions" in more detail. Of course, it should be clear to those skilled in the art that the present disclosure is also applicable to other application scenarios, which will not be described in detail here.

[0055] Figure 3 The flowchart of the method of requesting the second multimedia content and the response information according to some other embodiments of the present disclosure is shown. In this embodiment, the request information indicates that the user wants to evaluate the action in the first multimedia content. For example, the subject of the request information is evaluation. Figure 3 As shown, the method of this embodiment includes steps S302 to S306.

[0056] In step S302, a target action among one or more actions of the first multimedia content is determined.

[0057] In some embodiments, the target action may be one that the user has not specified in the request information. The target action may be an action with problems. For example, if the request information is "Help me see if there is any problem with this dance", it does not include instructions for a certain action, and the machine needs to process it to identify the action with problems.

[0058] To determine whether the target action needs to be improved, it can be determined whether the target action itself is standard, or whether the target action matches the music in the first multimedia content.

[0059] In some embodiments, a matching result between one or more actions in the first multimedia content and a sound in the first multimedia content is determined; and an action that does not match the sound is determined as a target action.

[0060] For example, the rhythm of the action and the rhythm of the sound in the first multimedia content (such as the background music of the video) can be referenced for matching. An action that produces an inconsistent rhythm is an action that does not match the sound. The rhythm of the action can be determined based on the changes between different actions and the amplitude of the action, for example, by processing one or more actions and sub-actions thereof through a video or image recognition model, and the generation time of each action (or sub-action) is determined. The rhythm of the sound can be determined based on the beat or accent of the sound, for example, by processing the audio signal through a sound processing model.

[0061] For another example, the target action can be determined by referring to whether the emotion or language of the action and the emotion or semantics of the sound match. If the action is passionate and unrestrained, but the background music during the action is soothing, then the action can also be determined as the target action. When analyzing the emotion or semantics of an action or sound, the action or sound can be first converted into a feature vector, and then the feature vector is processed using a semantic analysis model to obtain a semantic analysis result.

[0062] In cases where the action does not match the sound, it can be determined as a problematic target action even though the action itself may be standard.

[0063] In some embodiments, reference multimedia content corresponding to one or more actions is determined; the reference multimedia content is compared with the first multimedia content to determine the target action with problems. The reference multimedia content can be multimedia content including correct actions. It can be read from a database, searched through a search engine, or generated based on the description information of the stored or searched correct actions. When making a comparison, the features of the action in the first multimedia content (such as strength, angle, amplitude, etc.) and the features of the action in the reference multimedia content can be extracted respectively, and it is determined whether there is a problem with the action based on the comparison result of the features. Of course, if the features of the correct action are pre-stored, the features of the action in the first multimedia content can also be directly compared with them without the reference multimedia content.

[0064] The target action may also be an action specified in the request information. In some embodiments, the action that matches the action indicated by the request information among one or more actions is determined as the target action. For example, if the request information is "Help me see if the cartwheel is standard" and "Is there anything wrong with the second action?", the target actions are "cartwheel" and "second action" clearly indicated in the request information, respectively. The machine can determine the position of the target action in the second multimedia content according to the indication.

[0065] In some embodiments, the target action can be identified based on the dance action evaluation model, or the problem of the target action can be determined. The dance action evaluation model may include an input layer, a hidden layer, and an output layer. The input layer is used to receive the feature vector of the action, such as the type, speed, and strength of the action. The hidden layer is used to learn the characteristics and laws of the action, such as the fluency, coordination, and rhythm of the action. The output layer is used to evaluate the standard degree of the action and provide guidance suggestions, such as the correctness, aesthetics, and improvement suggestions of the dance action.

[0066] In step S304, second multimedia content is determined based on the target action.

[0067] The second multimedia content includes the target action. The second multimedia content may include the part of the first multimedia content involved in the target action, such as the involved image or video segment; it may also include the multimedia content after the corrected action; or the above two contents may be both.

[0068] In the case where the action and the sound do not match, in some embodiments, the target action in the first multimedia content is adjusted so that the target action matches the sound to generate the second multimedia content. That is, the second multimedia content is the result of adjusting the first multimedia content, which makes it easier for users to compare and learn, and improves the efficiency of information acquisition. For example, if the first multimedia content is a video of object A practicing dancing in a classroom, the second multimedia content can still be a video of object A practicing dancing in a classroom, except that some non-standard actions have been corrected. Of course, the second multimedia content may include or exclude the part corresponding to the non-target action in the first multimedia content, that is, the part where the action has no problem.

[0069] In some embodiments, a second multimedia content including the target action may also be generated or searched. For example, the features of the correct action corresponding to the target action may be determined based on the type, name, or features of the target action, and the second multimedia content including the correct action may be generated using these features. For another example, the type, name, or features of the target action may be used to search for an existing second multimedia content including the correct action.

[0070] In step S306, at least one of evaluation information on the target action in the first multimedia content and explanation information on the second multimedia content is generated as response information.

[0071] The evaluation information is used to describe at least one of the problems existing in the target action, the causes of the problems, and the correction methods, and the explanation information is used to describe at least one of the key points of the correct action corresponding to the target action and the correction method of the target action. For example, the evaluation information or explanation information can be generated through the difference information of the target action in the first multimedia content and the correct action corresponding to the target action in the second multimedia content.

[0072] As required, the evaluation information and explanation information may also appear in the second multimedia content in the form of sound, subtitles, indication marks, etc.

[0073] Through the above embodiments, the target action in the first multimedia content can be accurately located and analyzed based on the action in the first multimedia content, thereby improving the information acquisition efficiency and learning efficiency of the user.

[0074] Figure 4 A flowchart of a method for requesting second multimedia content and response information according to some other embodiments of the present disclosure is shown. In this embodiment, the request information indicates that the user wants to evaluate the action in the first multimedia content. For example, the subject of the request information is evaluation. Figure 4 As shown, the method of this embodiment includes steps S402 to S406.

[0075] In step S402, in response to the subject of the request information being a choreographed action, one or more target actions are predicted based on one or more actions.

[0076] Arranging an action refers to generating a combination of one or more target actions, and the generated one or more target actions may include part or all of one or more actions in the first multimedia content, or may not include any action in the first multimedia content. In the prediction process, in addition to referring to one or more actions, reference may also be made to the content in the request information and at least one item in the first multimedia content, for example, the description of the arranged action in the request information, the background music desired to be used in the arranged target action, the music in the first multimedia content, and the like.

[0077] In some embodiments, one or more target actions after one or more actions are predicted. Thus, the first multimedia content can be continued. For example, a user can give the beginning of a dance or a series of actions, and the machine can perform subsequent action choreography.

[0078] In some embodiments, one or more target actions including one or more actions are generated. Thus, the first multimedia content can be expanded. For example, a user can give a key action in a dance or a series of actions, and the machine can expand it.

[0079] When predicting the target action, a prediction model, such as a deep learning model, etc., can be used. For example, input data can be generated according to one or more actions in the first multimedia content; the input data can be input into the prediction model; and one or more target actions output by the prediction model can be obtained. The prediction model can include, for example, an input layer, a hidden layer, and an output layer. The input layer is used to receive feature vectors of one or more actions, the hidden layer is used to learn the characteristics and laws of music and dance actions, and the output layer is used to generate prediction results of dance actions.

[0080] The generated one or more target actions may refer to the name, description information, specific features (such as the position, angle, amplitude, duration, etc. of the action) or information represented by a vector of the generated target action, or may generate a video or image of each action in the one or more target actions.

[0081] In step S404, second multimedia content is generated based on the target action. For example, the target action may be processed using a generative model capable of generating multimedia content to obtain the generated second multimedia content.

[0082] For example, a model capable of generating multimedia content may be used to process the name, description information, or image of the target action to generate a video or image corresponding to the target action.

[0083] In step S406, explanation information of the action in the second multimedia content is generated as response information.

[0084] The explanation information may be generated based on the target action as a unit, that is, the explanation information may be generated for each action in part or all of the target action, or may be generated based on the entire second multimedia content, or may be a combination of the two. For example, the description information of the target action may be generated or searched as the explanation information, or the description information of the second multimedia content may be generated based on the description information of the target action as the explanation information.

[0085] Through the above embodiments, new actions can be arranged based on the first multimedia content to generate new second multimedia content. Thus, users can efficiently arrange actions such as choreography, thereby improving the efficiency of information acquisition and action learning.

[0086] In some embodiments, the subject of the request information may also be "recommendation". For example, the user may provide one or more personal photos as the first multimedia content, and the request information indicates to recommend a suitable dance or exercise based on the photos, and may also include some specific descriptions or requirements. The second multimedia content may be the second multimedia content recommended to the user and suitable for the physical condition of the subject in the photo, and the response information may be the reason for recommending the second multimedia, etc.

[0087] Some application scenarios of the embodiments of the present disclosure are described below by way of example.

[0088] In some embodiments, the user can send request information and the first multimedia content through the dialogue interface with the robot. After determining the second multimedia content and the response information, the robot can send the second multimedia content and the response information in the dialogue interface. The robot can generate corresponding content based on the dialogue sent by other subjects in the dialogue scene (such as other users or robots participating in the dialogue). It can be implemented in software, hardware, or a combination of software and hardware. The robot can also be called a virtual agent of a digital person or a machine learning model. The robot can be implemented based on a machine learning model, for example, based on a large language model (Large Language Model, referred to as: LLM) or a foundation model. The machine learning model can be a generative model, which is used to output target content based on input information.

[0089] In some embodiments, the user may also input request information through a form in the application interface and upload or specify the first multimedia content. Then, in the same interface or a new interface, the second multimedia content and the response information may be displayed to the user.

[0090] The embodiments of the present disclosure can be applied to dialogue applications with robots, sports applications, dance applications, etc. Of course, they can also be applied to other applications as needed, which will not be described in detail here.

[0091] The above are embodiments of the interactive method of the present disclosure. The following introduces the apparatus and device for executing the interactive method of the present disclosure.

[0092] Figure 5 FIG. 2 shows a schematic diagram of the structure of an interactive device according to some embodiments of the present disclosure. Figure 5 As shown, the interactive device 5 of this embodiment includes: a first display module 501, configured to display first multimedia content from a user and request information of the user based on the first multimedia content, the first multimedia content including one or more actions; a processing module 502, configured to process one or more actions in the first multimedia content based on the request information to determine second multimedia content and response information of the request information; a second display module 503, configured to display the second multimedia content and the response information.

[0093] In some embodiments, the processing module 502 is further configured to perform intent recognition on the request information to determine the subject of the request information; based on the subject of the request information, process one or more actions in the first multimedia content to determine the second multimedia content, and response information of the request information.

[0094] In some embodiments, the processing module 502 is further configured to determine a target action among one or more actions of the first multimedia content in response to the subject of the request information being evaluation, wherein the target action is an action with a problem or an action indicated by the request information; determine the second multimedia content based on the target action; and generate at least one of evaluation information on the target action in the first multimedia content and explanation information on the second multimedia content as response information.

[0095] In some embodiments, the processing module 502 is further configured to determine a matching result between one or more actions in the first multimedia content and a sound in the first multimedia content; and determine an action that does not match the sound as a target action.

[0096] In some embodiments, the processing module 502 is further configured to generate the second multimedia content by adjusting the target action in the first multimedia content so that the target action matches the sound.

[0097] In some embodiments, the processing module 502 is further configured to determine reference multimedia content corresponding to one or more actions; and compare the reference multimedia content with the first multimedia content to determine the problematic target action.

[0098] In some embodiments, the processing module 502 is further configured to determine an action among the one or more actions that matches the action indicated by the request information as a target action.

[0099] In some embodiments, the processing module 502 is further configured to generate or search for second multimedia content including the target action.

[0100] In some embodiments, the processing module 502 is further configured to predict one or more target actions based on one or more actions in response to the subject of the request information being choreographed actions; generate second multimedia content based on the target actions; and generate explanation information for the actions in the second multimedia content as response information.

[0101] In some embodiments, the processing module 502 is further configured to predict one or more target actions following the one or more actions; or generate one or more target actions including the one or more actions.

[0102] In some embodiments, the interaction device 5 further includes: a determination module 504 configured to identify an object in the first multimedia content; and determine one or more actions of the object in the first multimedia content according to the posture of the object.

[0103] In some embodiments, the request information is text or voice.

[0104] In some embodiments, the first multimedia content is a video or one or more images; the second multimedia content is a video or one or more images.

[0105] It should be noted that the above-mentioned units are only logical modules divided according to the specific functions implemented by them, and are not used to limit the specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the above-mentioned units are shown with dotted lines in the drawings to indicate that these units may not actually exist, and the operations / functions implemented by them can be implemented by the processing circuit itself.

[0106] In addition, although not shown, the device may also include a memory, which may store various information generated by the device, various units contained in the device during operation, programs and data for operation, data to be sent by the communication unit, etc. The memory may be a volatile memory and / or a non-volatile memory. For example, the memory may include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), and a flash memory. Of course, the memory may also be located outside the device. Optionally, although not shown, the device may also include a communication unit, which may be used to communicate with other devices. In one example, the communication unit may be implemented in a suitable manner known in the art, such as including communication components such as an antenna array and / or a radio frequency link, various types of interfaces, communication units, etc. It will not be described in detail here. In addition, the device may also include other components not shown, such as a radio frequency link, a baseband processing unit, a network interface, a processor, a controller, etc. It will not be described in detail here.

[0107] Some embodiments of the present disclosure also provide an electronic device. Figure 6 A schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure is shown. For example, in some embodiments, the electronic device 6 can be various types of devices, for example, can include but not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. For example, the electronic device 6 can include a display panel for displaying data and / or execution results used in the scheme according to the present disclosure. For example, the display panel can be in various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel. In addition, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.

[0108] like Figure 6 As shown, the electronic device 6 of this embodiment includes: a memory 61 and a processor 62 coupled to the memory 61. It should be noted that Figure 6 The components of the electronic device 6 shown are only exemplary and non-restrictive. The electronic device 6 may also have other components according to actual application requirements. The processor 62 may control other components in the electronic device 6 to perform desired functions.

[0109] In some embodiments, the memory 61 is used to store one or more computer-readable instructions. When the processor 62 is used to execute the computer-readable instructions, the computer-readable instructions are executed by the processor 62 to implement the method according to any of the above embodiments. For the specific implementation of each step of the method and the related explanation content, please refer to the above embodiments, and the repeated parts will not be repeated here.

[0110] For example, the processor 62 and the memory 61 may communicate with each other directly or indirectly. For example, the processor 62 and the memory 61 may communicate with each other through a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 62 and the memory 61 may also communicate with each other through a system bus, which is not limited by the present disclosure.

[0111] For example, the processor 62 may be embodied as various appropriate processors, processing devices, etc., such as a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc. For example, the memory 61 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The memory 61 may include, for example, a system memory, which may store, for example, an operating system, an application, a boot loader (Boot Loader), a database, and other programs. Various applications and various data may also be stored in the storage medium.

[0112] In addition, according to some embodiments of the present disclosure, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 7 The computer system 70 shown installs the programs constituting the software, and when the various programs are installed, the computer system can perform various functions, including the functions described above and the like. Figure 7 A schematic diagram of the structure of a computer system according to some embodiments of the present disclosure is shown.

[0113] exist Figure 7In the embodiment, a central processing unit (CPU) 701 performs various processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 707 to a random access memory (RAM) 703. In RAM 703, data required when CPU 701 performs various processes, etc., is also stored as needed. The central processing unit is merely exemplary, and it may also be other types of processors, such as the various processors described above. ROM 702, RAM 703, and storage section 707 may be various forms of computer-readable storage media, as described below. It should be noted that although Figure 7 ROM 702, RAM 703 and storage device 707 are shown separately in the figure, but one or more of them may be combined or located in the same or different memory or storage modules.

[0114] The CPU 701, the ROM 702, and the RAM 703 are connected to one another via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0115] The following components are connected to the input / output interface 705: an input portion 706, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 707, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 707, including a hard disk, a magnetic tape, etc.; and a communication portion 709, including a network interface card such as a LAN card, a modem, etc. The communication portion 709 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 7 The various devices or modules in the computer system 70 are shown to communicate via a bus 704, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless networks and wired networks.

[0116] A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is mounted on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 707 as needed.

[0117] In the case where the above-described series of processing is implemented by software, a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 711 .

[0118] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 707, or installed from a ROM 702. When the computer program is executed by the CPU 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0119] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0120] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0121] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute any of the methods of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0122] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0125] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0126] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0127] In the description provided herein, many specific details are set forth. However, it is understood that embodiments of the present invention can be implemented without these specific details. In other cases, well-known methods, structures, and techniques are not presented in detail in order not to obscure the understanding of the description.

[0128] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0129] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An interactive method, comprising: displaying first multimedia content from a user and request information of the user based on the first multimedia content, wherein the first multimedia content includes one or more actions; Based on the request information, processing the one or more actions in the first multimedia content to determine second multimedia content and response information to the request information; The second multimedia content and the response information are displayed.

2. The interactive method according to claim 1, wherein: The processing of the one or more actions in the first multimedia content based on the request information to determine the second multimedia content and the response information to the request information includes: Performing intent recognition on the request information to determine the subject of the request information; Based on the subject of the request information, the one or more actions in the first multimedia content are processed to determine second multimedia content and response information of the request information.

3. The interactive method according to claim 2, wherein: The processing of the one or more actions in the first multimedia content based on the subject of the request information to determine the second multimedia content and the response information of the request information includes: In response to the subject of the request information being evaluation, determining a target action among the one or more actions of the first multimedia content, wherein the target action is an action with a problem or an action indicated by the request information; determining the second multimedia content based on the target action; At least one of evaluation information on the target action in the first multimedia content and explanation information on the second multimedia content is generated as the response information.

4. The interactive method according to claim 3, wherein: In response to the subject of the request information being evaluation, determining a target action among the one or more actions of the first multimedia content includes: Determine a matching result between the one or more actions in the first multimedia content and the sounds in the first multimedia content; An action not matching the sound is determined as the target action.

5. The interactive method according to claim 4, wherein: The determining the second multimedia content based on the target action comprises: The second multimedia content is generated by adjusting the target action in the first multimedia content so that the target action matches the sound.

6. The interactive method according to claim 3, wherein: In response to the subject of the request information being evaluation, determining a target action among the one or more actions of the first multimedia content includes: determining reference multimedia content corresponding to the one or more actions; The reference multimedia content is compared with the first multimedia content to determine problematic target actions.

7. The interactive method according to claim 3, wherein: In response to the subject of the request information being evaluation, determining a target action among the one or more actions of the first multimedia content includes: An action among the one or more actions that matches the action indicated by the request information is determined as the target action.

8. The interactive method according to claim 6 or 7, wherein: The determining the second multimedia content based on the target action comprises: Generate or search for second multimedia content including the target action.

9. The interactive method according to claim 2, wherein: The processing of the one or more actions in the first multimedia content based on the subject of the request information to determine the second multimedia content and the response information of the request information includes: In response to the subject of the request information being a choreographed action, predicting one or more target actions based on the one or more actions; generating second multimedia content based on the target action; Generate explanation information of the action in the second multimedia content as the response information.

10. The interactive method according to claim 9, wherein: The predicting one or more target actions based on the one or more actions comprises: predicting one or more target actions following the one or more actions; or One or more target actions are generated that include the one or more actions.

11. The interactive method according to any one of claims 1 to 10, further comprising: identifying an object in the first multimedia content; One or more actions of the object in the first multimedia content are determined according to the posture of the object.

12. The interactive method according to any one of claims 1 to 11, wherein: The request information is text or voice.

13. The interactive method according to any one of claims 1 to 12, wherein: The first multimedia content is a video or one or more images; The second multimedia content is a video or one or more images.

14. An electronic device comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the interaction method according to any one of claims 1 to 13 based on instructions stored in the memory.

15. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the interactive method according to any one of claims 1 to 13.

16. A computer program product, when the computer program product is run on a computer, enables the computer to implement the interactive method according to any one of claims 1 to 13.

17. A computer program comprising: Instructions, which, when executed by a processor, cause the processor to perform the interaction method according to any one of claims 1 to 13.

Images