User interaction method and device, equipment, storage medium and program product

Abstract

The invention provides a user interaction method and device, equipment, a storage medium and a program product, and relates to the technical field of user interaction. The method comprises the following steps: acquiring a user action image and motion data of a user in a motion process; determining a motion part and a motion standard degree of the user based on the user motion image; and adjusting a virtual image corresponding to the user based on the motion data, the motion part and the motion standard degree, and feeding back the adjusted virtual image. According to the method and the device, interaction with the user is enhanced through the virtual image, and the exercise effect is shown to the user in a mode of changing the image, so that the user can see the exercise effect of the user more clearly, and the user has more interest and motivation of exercise.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, specifically to the field of user interaction technology, and in particular to a user interaction method, device, equipment, storage medium and program product. Background Art

[0002] As people's material living standards improve, their demand for fitness is gradually increasing. Home fitness, as a convenient and fast way of fitness, is gradually being sought after by people. Unlike fitness in the gym, home fitness can only be practiced alone by watching TV videos, which lacks interactivity and is boring. Therefore, how to combine the characteristics of TV to make home fitness more effective and interesting is an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present disclosure provide a user interaction method, apparatus, device, storage medium, and program product.

[0004] In a first aspect, an embodiment of the present disclosure proposes a user interaction method, including: obtaining a user motion image and motion data of the user during exercise; determining the user's motion part and motion standard based on the user motion image; adjusting the user's corresponding virtual image based on the motion data, motion part and motion standard, and providing feedback on the adjusted virtual image.

[0005] In the second aspect, an embodiment of the present disclosure proposes a user interaction device, including: an information acquisition module, configured to acquire user motion images and motion data of the user during exercise; a motion information determination module, configured to determine the user's motion part and motion standard based on the user motion image; a virtual image adjustment module, configured to adjust the user's corresponding virtual image based on the motion data, motion part and motion standard, and provide feedback on the adjusted virtual image.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the user interaction method described in any implementation method in the first aspect when executing.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the user interaction method described in any implementation manner in the first aspect when executed.

[0008] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the user interaction method described in any implementation manner in the first aspect.

[0009] The user interaction method, apparatus, device, storage medium and program product provided by the embodiments of the present disclosure can enhance the interaction with the user through the virtual image, and can provide more feedback information to the user when exercising. In addition, the virtual image can track the user's exercise status and show the effect of exercise to the user by changing the image, so that the user can see the effect of his or her exercise more clearly, which makes the user more interested in and motivated to exercise.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0012] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0013] Figure 2 A flowchart of a user interaction method provided by an embodiment of the present disclosure;

[0014] Figure 3 A flowchart of an embodiment of determining a user's action standard based on a user action image provided by an embodiment of the present disclosure;

[0015] Figure 4 A flowchart of another embodiment of determining a user's action standard based on a user action image provided by an embodiment of the present disclosure;

[0016] Figure 5 A flowchart for determining a user's moving parts based on a user's motion image provided by an embodiment of the present disclosure;

[0017] Figure 6 A flowchart of another user interaction method provided by an embodiment of the present disclosure;

[0018] Figure 7 A schematic diagram of a model display effect of adjusting a corresponding position of a virtual image provided by an embodiment of the present disclosure;

[0019] Figure 8 A schematic diagram of adjusting the display effect of facial features of a virtual image provided by an embodiment of the present disclosure;

[0020] Fig. 9 A flowchart of another user interaction method provided by an embodiment of the present disclosure;

[0021] Fig.10 A structural block diagram of a user interaction device provided in an embodiment of the present disclosure;

[0022] Fig.11 A schematic diagram of the structure of an electronic device suitable for executing a user interaction method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the user interaction method, apparatus, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0026] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 may be installed with various applications for realizing information communication between the two, such as video social applications, instant messaging applications, etc.

[0028] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0029] The server 105 can provide various services through various built-in applications. The video images and the like involved in the various applications used by the user can be obtained from the terminal devices 101, 102, 103 through the network 104, or can be pre-stored in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally, it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0030] Since the processing of video images requires more computing resources and stronger computing power, the user interaction methods provided in the subsequent embodiments of the present disclosure are generally performed by the server 105 with stronger computing power and more computing resources. Accordingly, the user interaction device is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations handed over to the server 105 through the video social application installed thereon, and then output the same result as the server 105. Especially in the case where there are multiple terminal devices with different computing powers at the same time, but when the video social application determines that the terminal device where it is located has stronger computing power and more remaining computing resources, it can let the terminal device perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Accordingly, the user interaction device can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0031] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0032] Please refer to Figure 2 , Figure 2 This is a flow chart of a user interaction method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0033] Step 201: Obtain user motion images and motion data of the user during exercise.

[0034] This step is intended to be performed by the execution subject of the user interaction method (such as Figure 1 The server 105 shown, or the terminal devices 101, 102, 103) obtains the user's motion image and motion data during the user's exercise. The user motion image refers to a video image of the user during the exercise that can be collected by a camera device that is connected to the execution subject through communication. The motion data is the user's body index related data, such as heart rate, calories, stress index, body temperature, etc., collected by motion sensors (such as wearable devices such as smart bracelets and smart watches), sports applications (using mobile phones or tablets GPS positioning and acceleration sensors to record motion data) or fitness equipment (such as treadmills, elliptical machines, etc.) during the user's exercise.

[0035] Step 202: Determine the user's movement parts and movement standard based on the user's movement image.

[0036] This step is intended to allow the above-mentioned execution subject to analyze the user's motion image to obtain the user's moving parts during exercise and the user's motion standard. In this embodiment, the user's motion image can be processed and analyzed by, for example, computer vision technology and machine learning algorithms to identify the user's moving parts. For the motion standard, the user's motion image can be combined with a preset standard motion image or motion parameters to compare and analyze the standard degree of the user's motion during exercise, thereby obtaining the motion standard.

[0037] Step 203: adjusting the virtual image corresponding to the user based on the motion data, motion parts and motion standard, and providing feedback on the adjusted virtual image.

[0038] In this embodiment, the interaction between the execution subject and the user can be realized through the constructed virtual image, and the user can be given real-time feedback on the user's movement process by displaying the virtual image to the user. The virtual image can be displayed through a display window created on the display device viewed by the user. Specifically, the execution subject can create a virtual image display window at the top level of the current display window hierarchy, and display the virtual image in the virtual image display window. If the virtual image is adjusted according to the user's movement, the adjusted virtual image can also be displayed in the virtual image display window.

[0039] In specific implementations, the specific presentation appearance of the virtual image may include multiple types, such as cartoon characters, comic characters, personified animals, personified plants, etc. The specific presentation appearance to be used can be determined according to the user's choice. For example, when it is detected that the user starts to exercise, several alternative virtual images are displayed to the user for selection through the display device, and the specific presentation appearance of the virtual image is determined in response to the user's clicking operation. In other optional implementations, several virtual images that are closer to the user's body image can be generated based on the user's body image identified in the user's motion image (for example, for a user with a thinner figure, a virtual image of a cute little monkey can be generated; for a user with a stronger figure, a virtual image of a panda or a little bear can be generated with a cute image) for the user to choose.

[0040] For the virtual image, the appearance of the virtual image can be adjusted accordingly in combination with the user's exercise data, exercise parts and action standard. For example, if the user's exercise data and exercise parts are combined to determine that the user's exercise process is mainly to exercise the upper limb strength, the upper limb image of the virtual image can be highlighted to highlight that the upper limb muscles have been fully exercised; if the user's action standard is combined to determine that the user's actions during this exercise are very standard, the virtual image can cheer, applaud and other actions to give the user positive feedback and encourage the user to continue.

[0041] In some optional implementations of this embodiment, in addition to providing feedback to the user through the virtual image during the user's exercise, when the user starts exercising, the execution subject can detect the time interval between the user's current exercise and the previous exercise, and adjust the overall image of the virtual image at the beginning of this exercise according to the time interval. For example, when it is detected that the time interval between two exercises of the user is greater than five days, it indicates that the interval between the two exercises is long. The execution subject can adjust the virtual image to a more robust image, indicating that the virtual image has lacked exercise during this period, so as to motivate the user to exercise more diligently; when it is detected that the time interval between two exercises of the user is less than three days, it indicates that the interval between the two exercises is short. The execution subject can adjust the virtual image to a spirited appearance, indicating that the virtual image has maintained a good exercise frequency during this period, so as to motivate the user to continue to maintain it. In addition, the execution subject's adjustment of the virtual image can also be different according to different settings. For example, the execution subject can establish a setting: the virtual image relies on the energy generated by the user's exercise to replenish itself, that is, the more energy the user generates from the exercise, the stronger the overall virtual image, and vice versa. Correspondingly, if the interval between two exercises of the user is long, the virtual image has no energy replenishment for a long time, then the execution entity may adjust the overall image of the virtual image to be thin; if the interval between two exercises of the user is short, the energy replenishment of the virtual image is sufficient, then the execution entity may adjust the overall image of the virtual image to be strong. This embodiment is not limited to this and can be set and adjusted according to needs.

[0042] The user interaction method provided by the disclosed embodiment can enhance the interaction with the user through the virtual image, and can provide more feedback information to the user when exercising. In addition, the virtual image can track the user's exercise status and show the effect of exercise to the user by changing the image, so that the user can see the effect of his or her exercise more clearly, making the user more interested in and motivated to exercise.

[0043] In some optional implementations of this embodiment, such as Figure 3 As shown, in the above step 202, the process of determining the user's action standard based on the user action image mainly includes:

[0044] Step 301: binarizing a preset reference motion image and a user motion image to obtain a reference motion binary image and a user motion binary image; the preset reference motion image is a video image that the user refers to during exercise.

[0045] In this embodiment, the user's action standard can be determined by comparing the user's action image with the reference action image. The reference action image refers to the action image that the user refers to when exercising. For example, when the user is watching a fitness program on TV, in which a coach leads the audience to practice some fitness movements, the action image of the coach can be used as the reference image. Alternatively, when the user subscribes to some special exercise courses, the course can provide a demonstration video of a real coach, or a demonstration action of a three-dimensional model, and the demonstration video or the image of the demonstration action can be used as the reference image.

[0046] For the reference action image and the user action image, a mature image processing algorithm (such as the OTSU algorithm (maximum between-class variance method)) can be used for processing to obtain a reference action binary image and a user action binary image respectively, and the pixel value of the character area is set to black and the background is set to white.

[0047] Step 302: superimpose and calculate the pixel value ratio of the repeated area of ​​the reference action binary image and the user action binary image in the reference action binary image.

[0048] The reference action binary image and the user action binary image are superimposed and calculated to determine the pixel values ​​of the repeated area of ​​the reference action binary image and the user action binary image, and calculate the proportion of the pixel values ​​of the repeated area in the reference action binary image.

[0049] Step 303: Determine the action standard based on the pixel value ratio.

[0050] In this embodiment, the standard of the user's action is determined by the proportion of the pixel values ​​in the repeated area. That is, the more standard the user's action is, the more repeated areas there should be between the user's action binary image and the reference action binary image.

[0051] Correspondingly, when the pixel value ratio is higher, the degree of positive feedback given to the user through the virtual image is higher; alternatively, the pixel value ratio can be set to be in different intervals, and the degree of feedback given to the user through the virtual image is also different. For example, when the pixel value ratio is in the interval [0, 60%), [60%, 75%), [75%, 90%), [90%, 100%], the virtual image can respectively provide different content to the user through text and / or voice feedback: "You need to work harder!", "Not bad, keep it up!", "Great, hold on!", "Perfect! Keep it up!", etc.

[0052] In some optional implementations of this embodiment, the above step 202, the process of determining the user's action standard based on the user action image, is not limited to the method described in the above embodiment, such as Figure 4 As shown, the process mainly includes:

[0053] Step 401: binarize the user action image to obtain a user action binary image.

[0054] The process of binarizing the user action image in step 401 is similar to the process described in Figure 3 The step 301 shown is consistent, and the same content can be found in the corresponding part of the previous embodiment, which will not be repeated here.

[0055] Step 402: Determine the user's motion parameters based on the user motion binary image.

[0056] By using the user action binary image, the motion parameters such as the position and angle reached by the user when performing some actions during exercise can be directly determined.

[0057] Step 403: Calculate the difference between the motion parameter and the preset motion standard parameter.

[0058] When users subscribe to some special exercise courses, the course can provide demonstration movements of the 3D model, or when users use AR (Augmented Reality) devices to exercise, they can directly obtain the standard movement parameters of the standard movement that the user refers to through the 3D model or AR model. Calculate the difference between the movement parameters and the preset standard movement parameters.

[0059] Step 404: Determine the action standard based on the difference.

[0060] In this embodiment, the standard of the user's movements is determined by the difference. That is, the more standard the user's movements are, the smaller the difference between the movement parameters and the movement standard parameters should be.

[0061] Correspondingly, when the difference is smaller, the positive feedback level that can be given to the user through the virtual image is higher; or, when the difference is in different intervals, the feedback level given to the user through the virtual image is also different. For example, when the difference is in the intervals of [0, 5], (5, 10], (10, 15], (15, 20], the virtual image can respectively give different feedback to the user through text and / or voice: "Perfect! Keep it up!", "Great, hold on!", "Not bad, keep up the good work!", "You need to work harder!", etc.

[0062] Through the above process, the degree of effort of the user during exercise can be accurately analyzed. According to the degree of effort, corresponding feedback can be further given to the user through the virtual image to enhance the user's sense of participation and enthusiasm in exercise.

[0063] In some optional implementations of this embodiment, such as Figure 5 As shown, the above step 202, the process of determining the user's moving part based on the user's motion image, mainly includes:

[0064] Step 501: Perform a spatiotemporal graph convolution operation on the user action image to generate a feature graph; wherein the spatiotemporal graph used for the spatiotemporal graph convolution operation is a spatiotemporal graph constructed with joints as graph nodes and natural connections of human body structure and time as graph edges.

[0065] Through Spatial Temporal Graph Convolutional Networks (ST-GCN), the user's behavior and actions are analyzed, and then the user's current movement part is analyzed. A spatiotemporal graph is constructed with joints as graph nodes and natural connections of human body structure and time as graph edges. The input is the joint coordinate vector on the graph node, which is composed of pixel intensity vectors residing on a two-dimensional image grid. Multi-level spatiotemporal graph convolution operations are performed on the input data to generate feature maps on the graph.

[0066] Step 502: Classify the feature map and determine the moving parts.

[0067] The user's moving part can be determined by performing classification processing based on the feature map through a standard classifier (such as a support vector machine, a convolutional neural network (CNN), etc.).

[0068] Through the above process, the moving parts of the user during exercise can be accurately analyzed, thereby providing a basis for the subsequent corresponding adjustment of the virtual image, and providing timely feedback to the user to enhance the user's exercise experience.

[0069] Please refer to Figure 6 , Figure 6 A flowchart of another user interaction method provided by an embodiment of the present disclosure, wherein process 600 includes the following steps:

[0070] Step 601: Obtain user motion images and motion data of the user during exercise.

[0071] Step 602: Determine the user's movement parts and movement standard based on the user's movement image.

[0072] The above steps 601-602 are similar to Figure 2 Steps 201-202 shown are consistent, and for the same contents, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0073] Step 603: adjusting the virtual image corresponding to the user based on the motion data, motion parts and motion standard, and providing feedback on the adjusted virtual image.

[0074] In this embodiment, the process of adjusting the virtual image corresponding to the user based on motion data, motion parts and motion standards mainly includes: adjusting the model display effect of the corresponding position of the virtual image based on the motion data, motion parts and motion standards.

[0075] The execution subject can adjust the model display effect of the corresponding position of the virtual image based on the motion data, motion parts and action standards. The ST-GCN network recognizes that the main parts of the user's movement are the right arm and left leg. Combined with the motion data and action standards, the right arm and left leg of the virtual image panda are also adjusted accordingly, for example, they can be adjusted to be more robust, or the muscle lines are more obvious. Exemplarily, based on the user's motion data and motion parts, it is determined that the user's exercise process is mainly to exercise the upper limb strength. The upper limb part of the virtual image can be highlighted to highlight that the muscles of the upper limbs have been fully exercised, giving the user positive feedback and encouraging the user to continue.

[0076] It should be noted that the right arm and left leg involved in the above adjustment process are only examples. The parts of the virtual image that can be adjusted include: expression, arm muscles, leg muscles, waist line, chest muscles, etc. Correspondingly, for the model of the virtual image, four variables can be set for adjustment: arm muscles A, leg muscles L, waist line M, chest muscles B.

[0077] According to the moving part, it is determined that one or more of the above four variables need to be adjusted accordingly. Then, the image adjustment result for the virtual image model can be determined by the following formula (1):

[0078] W=K·X i +AS·X j +m (1)

[0079] Among them, W is the adjustment result of the virtual image model; K is calories; AS is the standard of action; X represents the variable characteristics, that is, the weights of arm muscles A, leg muscles L, waist lines M, chest muscles B, etc.; i and j represent the indexes under different K and AS coefficients; m is the correction difference. The specific values ​​of i, j, and m can be adjusted through multiple sets of data sets, and finally determine the values ​​of i, j, and m corresponding to different variable characteristics and coefficients.

[0080] For example, it is determined through the exercise part that the virtual image needs to adjust the arm muscle A, and the corresponding calories K and the action standard AS are input into formula (1). Correspondingly, formula (1) becomes: W = K·A i +AS·A j +m, where i, j, and m are adjusted through multiple sets of data sets to ultimately determine the value of the corresponding arm muscle A, and ultimately obtain the specific adjustment result for the arm muscle A.

[0081] As for the situation where the overall image of the virtual image needs to be adjusted, for example, according to the time interval between two exercises of the user, the overall image of the virtual image is adjusted to be more sturdy or thinner, etc., in this case, the execution subject can simultaneously adjust the four variables of arm muscles A, leg muscles L, waist line M, and chest muscles B to achieve the adjustment of the overall image of the virtual image.

[0082] In some optional implementations of this embodiment, such as Figure 7 As shown, the execution subject can also be combined with Figure 5 In the embodiment shown, based on the constructed space-time graph, the virtual image model is mapped to adjust the specific position of the virtual image model. Specifically, the execution subject can set the virtual image's body joint sequence according to the user's action image, and map the virtual image's body joint sequence to the space-time graph constructed based on the user's action image. The weights of the bone points in the virtual image's body joint sequence are adjusted in combination with the user's motion data and action standard, thereby adjusting the specific position of the virtual image model.

[0083] In some optional implementations of this embodiment, in addition to adjusting the corresponding part features of the virtual image according to the user's exercise part, the facial display features of the virtual image can also be adjusted according to the user's exercise situation, based on the exercise part and the standard of the movement. For example, if it is detected that the user is exercising, the exercise part is the hand or foot, and the heart rate reaches 90 beats / minute (usually the heart rate is 60-100 beats / minute), it means that the user may be doing some relatively easy stretching or warm-up exercises, and the exercise intensity is at a low level. Correspondingly, the execution subject can adjust the facial features of the virtual image to relatively easy expressions such as blinking, smiling, etc. Figure 8 As shown, the virtual image is more receptive to the current exercise like the user; or, if it is detected that the user is exercising with the arms or legs and the heart rate reaches 120 beats / minute (normally the heart rate is 60-100 beats / minute), it indicates that the user's exercise intensity is at a high level. Correspondingly, the execution subject can adjust the facial features of the virtual image to a more strenuous situation and add some sweating effects to show that the virtual image is working hard for the exercise like the user.

[0084] The user interaction method provided by the embodiment of the present disclosure tracks the user's exercise status through a virtual image during the user's exercise, and changes the appearance of the virtual image in real time to show the user the process and effect of exercise, so that the user can see the effect of his or her exercise more clearly, making the user more interested in and motivated to exercise.

[0085] In actual applications, in addition to adjusting the facial features of the virtual image according to the user's moving parts, movement standard, etc., further adjustments can be made according to other information of the user. In some implementations of this embodiment, the execution subject can obtain the user's voice information through a microphone or other sound receiving device, and recognize at least one of the volume, tone and text information from the voice information, and adjust the facial display features of the virtual image accordingly according to the recognized information.

[0086] Exemplarily, when the executing entity recognizes that the volume of the user's voice information is high, for example, higher than 70 decibels (the normal speaking volume is around 40 to 60 decibels), it indicates that the user is doing some exercise movements with more effort. At this time, the executing entity can adjust the facial features of the virtual image to a more strenuous situation and add some sweating effects to show that the virtual image is making efforts for exercise just like the user.

[0087] Exemplarily, the executing entity recognizes that the pitch of the user's voice information is higher, for example, higher than the pitch that has been sustained for a period of time before, indicating that the user is doing some exercise movements with more effort. At this time, the executing entity can adjust the facial features of the virtual image to a more strenuous situation and add some sweating effects to show that the virtual image is making efforts for exercise just like the user.

[0088] For example, when the execution subject recognizes that the text information of the user's voice information contains keywords or key words such as "Come on", "A few more", "Hold on", etc., it indicates that the user is cheering for himself. At this time, the execution subject can adjust the facial features of the virtual image to a cheering and encouraging expression, and can also add the display of text content such as "Come on" to cheer for the user. In addition, corresponding response information can be generated for the text information, and the virtual image can respond by voice, such as cheering for the user, so as to further improve the user's feeling during exercise, so that the user can feel the motivation personally and improve the user experience.

[0089] It should be noted that, in specific implementation, the execution subject may provide feedback through only one of the above-mentioned volume, tone and text information, or through a combination of any two of the above-mentioned volume, tone and text information, or a combination of all three to provide feedback together. This embodiment is not limited to this.

[0090] Please refer to Fig. 9 , Fig. 9 A flowchart of another user interaction method provided by an embodiment of the present disclosure, wherein process 900 includes the following steps:

[0091] Step 901: Obtain user motion images and motion data of the user during exercise.

[0092] Step 902: Determine the user's movement parts and movement standard based on the user's movement image.

[0093] Step 903: adjusting the virtual image corresponding to the user based on the motion data, motion parts and motion standard, and providing feedback on the adjusted virtual image.

[0094] The above steps 901-903 are similar to Figure 2 Steps 201-203 shown are consistent, and for the same contents, please refer to the corresponding parts of the previous embodiment, which will not be repeated here.

[0095] Step 904: Acquire virtual images of multiple users, and display the multiple virtual images in the same interface.

[0096] In this embodiment, in addition to interacting with a user through a virtual image during exercise for a single user, virtual images of multiple users can also be obtained and displayed in the same interface, thereby enhancing the interest of the overall display. Moreover, since the displayed virtual images correspond to the real-time exercise status of different users, users can also be informed of the exercise status of other users through the virtual images, thereby enhancing the competitiveness of the exercise process.

[0097] Step 905: sorting the exercise status of each user based on the exercise data, exercise part and movement standard of each user, and sorting the multiple virtual images according to the sorting results.

[0098] After displaying multiple virtual images, the exercise status of each user can be further sorted according to the exercise data, exercise parts and movement standards of different users, and the multiple virtual images can be sorted according to the sorting results. In fact, an online exercise mode has been added. Users can turn on the online exercise function and upload virtual images. During exercise, multiple people can exercise at the same time through matching, sharing or invitation, and multiple virtual images will appear on the same interface. In addition, each person's virtual image will also correct the part image according to the user's exercise status, and display the virtual image ranking according to the exercise status.

[0099] The user interaction method provided in this embodiment can improve the fun and competitiveness of the overall display by displaying the ranking of different users in terms of sports content and sports status, and can also enhance the user's sense of participation and motivate the user to exercise actively.

[0100] Further references Fig.10 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a user interaction device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0101] like Fig.10 As shown, the user interaction device 1000 of this embodiment may include: an information acquisition module 1001, a motion information determination module 1002, and a virtual image adjustment module 1003. The information acquisition module 1001 is configured to acquire the user's motion image and motion data during the user's motion process. The motion information determination module 1002 is configured to determine the user's motion part and motion standard based on the user's motion image. The virtual image adjustment module 1003 is configured to adjust the user's corresponding virtual image based on the motion data, motion part and motion standard, and feed back the adjusted virtual image.

[0102] In this embodiment, the specific processing of the information acquisition module 1001, the motion information determination module 1002 and the virtual image adjustment module 1003 and the technical effects thereof can be referred to in the respective Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0103] This embodiment exists as a device embodiment corresponding to the above-mentioned method embodiment. The user interaction device provided by this embodiment strengthens the interaction with the user through the virtual image, and can provide more feedback information for the user when exercising. In addition, the virtual image can track the user's exercise status and show the effect of exercise to the user by changing the image, so that the user can see the effect of his or her exercise more clearly, which makes the user more interested in and motivated to exercise.

[0104] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the user interaction method described in any of the above embodiments can be implemented when the at least one processor executes.

[0105] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the user interaction method described in any of the above embodiments when executed.

[0106] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the user interaction method described in any of the above embodiments.

[0107] Fig.11A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0108] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0109] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as user interaction methods. For example, in some embodiments, the user interaction method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the user interaction method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the user interaction method in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A user interaction method, comprising: Acquire user motion images and motion data of the user during exercise; Determine the user's movement part and movement standard based on the user movement image; The virtual image corresponding to the user is adjusted based on the motion data, the motion part and the motion standard, and the adjusted virtual image is fed back.

2. The user interaction method according to claim 1, wherein: Determining the action standard of the user based on the user action image includes: Binarization is performed on a preset reference motion image and the user motion image to obtain a reference motion binary image and a user motion binary image; the preset reference motion image is a video image that the user refers to during exercise; Superimposing and calculating the pixel value ratio of the repeated area of ​​the reference action binary image and the user action binary image in the reference action binary image; The action standard is determined based on the pixel value ratio.

3. The user interaction method according to claim 1, wherein: Determining the action standard of the user based on the user action image includes: Binarizing the user action image to obtain a user action binary image; Determining the motion parameters of the user based on the user motion binary image; Calculating the difference between the motion parameter and a preset motion standard parameter; The action standard is determined based on the difference.

4. The user interaction method according to claim 1, wherein: Determining the movement part of the user based on the user motion image includes: Performing a spatiotemporal graph convolution operation on the user action image to generate a feature graph; wherein the spatiotemporal graph used for performing the spatiotemporal graph convolution operation is a spatiotemporal graph constructed with joints as graph nodes and natural connections of human body structure and time as graph edges; The feature map is classified and processed to determine the motion part.

5. The user interaction method according to claim 1, wherein: The adjusting the virtual image corresponding to the user based on the motion data, the motion part and the motion standard degree includes: Based on the motion data, motion parts and motion standard, the model display effect of the corresponding position of the virtual image is adjusted.

6. The user interaction method according to claim 1, wherein: Also includes: Acquiring voice information of the user; Identifying at least one of the volume, pitch, and text information of the voice information; The facial display features of the avatar are adjusted based on at least one of the volume, pitch, and text information.

7. The user interaction method according to claim 6, wherein: Also includes: A response message is generated based on the text message, and a voice response is made through the virtual image.

8. The user interaction method according to claim 1, wherein: Also includes: In response to detecting that the user starts exercising, obtaining the interval time from the user's last exercise; The overall image of the avatar is adjusted based on the interval time.

9. The user interaction method according to claim 1, wherein: The initial state of the virtual image is generated according to the user characteristics of the user, or is generated in response to the operation instruction of the user.

10. The user interaction method according to claim 1, wherein: Feedback on the adjusted avatar, including: Create a virtual image display window at the top level of the current display window hierarchy; The adjusted virtual image is displayed in the virtual image display window.

11. The user interaction method according to any one of claims 1 to 10, wherein: Also includes: Acquire virtual images of multiple users, and display the multiple virtual images in the same interface; Based on the exercise data, exercise parts and movement standards of each user, the exercise conditions of each user are sorted, and the multiple virtual images are sorted according to the sorting results.

12. A user interaction device, comprising: An information acquisition module, configured to acquire a user motion image and motion data of a user during a motion process; A motion information determination module, configured to determine the user's motion part and motion standard based on the user motion image; The virtual image adjustment module is configured to adjust the virtual image corresponding to the user based on the motion data, motion parts and action standards, and feed back the adjusted virtual image.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the user interaction method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the user interaction method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the user interaction method according to any one of claims 1 to 11 are implemented.

Images