Action recognition method and device for user virtual image and electronic equipment

Abstract

The invention discloses an action recognition method and device for a user virtual image and electronic equipment, and relates to the field of artificial intelligence, the field of financial science and technology or other related technical fields, and the action recognition method comprises the steps: obtaining video files, collected based on N camera devices, of a target user, and obtaining N video files; for each video file, inputting the video file into a neural network model corresponding to camera equipment through the camera equipment, and outputting action recognition data corresponding to the camera equipment; and fusing all the action recognition data based on the action recognition data corresponding to each camera device and a weight coefficient of the camera device to obtain an action recognition result of the virtual image corresponding to the target user, the weight coefficient being configured based on a multi-dimensional feature of the camera device. According to the method and the device, the technical problem that the accuracy of action recognition is relatively low in a mode of capturing actions of a target user based on a sensor in related technologies is solved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, financial technology and other related technical fields, and in particular to a method and device for recognizing the motion of a user's virtual image, and an electronic device. Background Art

[0002] With the development and application of metaverse technology, a user's virtual image can be constructed based on the user's physical image in various fields, and the virtual image can be controlled to synchronously execute the user's actions. For example, in online live broadcast activities, ordinary users can drive virtual people to perform dance, sports and other activities in the live broadcast room; or in games, the virtual image can avoid obstacles and unlock game levels according to the user's actions.

[0003] Motion capture in related technologies requires the use of professional motion capture equipment, such as wearable sensors to capture the movements of specific parts of the body, thereby identifying the real-time movements of the target user and controlling the target user's virtual image to perform the collected movements to achieve human-computer interaction. However, the method of capturing human body movements through sensors and then generating movements of the user's virtual image has a low accuracy rate in motion recognition.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for motion recognition of a user virtual image, and an electronic device, so as to at least solve the technical problem in the related art that the accuracy of motion recognition is low in the method of capturing the motion of the target user based on sensors.

[0006] According to one aspect of an embodiment of the present invention, a method for motion recognition of a user virtual image is provided, comprising: acquiring a video file of a target user captured by N cameras to obtain N video files, wherein N is a positive integer; for each of the video files, inputting the video file into a neural network model corresponding to the camera device through the camera device, and outputting motion recognition data corresponding to the camera device, wherein the neural network model is a pre-trained model for motion recognition; based on the motion recognition data corresponding to each of the cameras and the weight coefficient of the camera device, all the motion recognition data are merged to obtain a motion recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

[0007] Optionally, before obtaining the video files of the target user captured based on N cameras, the method further includes: obtaining performance parameters of the camera devices; and configuring a first confidence level for the camera devices based on the performance parameters.

[0008] Optionally, after configuring the first confidence level for the camera device based on the performance parameter, the method further includes: acquiring a position relationship between the camera device and a target user; and configuring a second confidence level for the camera device based on the position relationship.

[0009] Optionally, after configuring the second confidence level for the camera device based on the positional relationship, the method further includes: acquiring detail feature data captured by the camera device; and configuring a third confidence level for the camera device based on the detail feature data.

[0010] Optionally, after configuring a third confidence level for the camera device based on the detail feature data, the method further includes: obtaining a situation in which the camera device is blocked by an obstacle; and configuring a fourth confidence level for the camera device based on the blocking situation.

[0011] Optionally, before fusing all the action recognition data based on the action recognition data corresponding to each of the cameras and the weight coefficients of the cameras, the method further includes: calculating the average of the first confidence, the second confidence, the third confidence and the fourth confidence to obtain an average confidence value; and configuring a weight coefficient for the camera based on the average confidence value.

[0012] Optionally, after obtaining the action recognition result of the target user's corresponding virtual image, it also includes: obtaining a target video file of the target user, wherein the target video file is a video file including the target user's face; extracting facial emotion features of the target user from the target video file; and optimizing the action recognition result based on the facial emotion features.

[0013] Optionally, after obtaining the action recognition result of the virtual image corresponding to the target user, it also includes: mapping the action recognition result with the action library of the virtual image to obtain an action instruction; and controlling the virtual image to perform a corresponding action based on the action instruction.

[0014] According to another aspect of an embodiment of the present invention, there is also provided a device for motion recognition of a user virtual image, comprising: an acquisition unit, used to acquire a video file of a target user collected based on N cameras, to obtain N video files, wherein N is a positive integer; an output unit, used to input, for each of the video files, the video file into a neural network model corresponding to the camera device through the camera device, and output motion recognition data corresponding to the camera device, wherein the neural network model is a pre-trained model for motion recognition; a fusion unit, used to fuse all motion recognition data based on the motion recognition data corresponding to each of the cameras and the weight coefficient of the camera device, to obtain a motion recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

[0015] Optionally, the action recognition device for the user virtual image also includes: a first acquisition module, used to obtain performance parameters of the camera device; and a first configuration module, used to configure a first confidence level for the camera device based on the performance parameters.

[0016] Optionally, the action recognition device for the user virtual image also includes: a second acquisition module, used to obtain the position relationship between the camera device and the target user; and a second configuration module, used to configure a second confidence level for the camera device based on the position relationship.

[0017] Optionally, the action recognition device for the user virtual image also includes: a third acquisition module, used to obtain detail feature data captured by the camera device; and a third configuration module, used to configure a third confidence level for the camera device based on the detail feature data.

[0018] Optionally, the action recognition device for the user virtual image also includes: a fourth acquisition module, used to obtain the situation that the camera device is blocked by an obstacle; and a fourth configuration module, used to configure a fourth confidence level for the camera device based on the blocking situation.

[0019] Optionally, the action recognition device for the user virtual image also includes: a first calculation module, used to calculate the average of the first confidence, the second confidence, the third confidence and the fourth confidence to obtain the confidence average value; a fifth configuration module, used to configure a weight coefficient for the camera device based on the confidence average value.

[0020] Optionally, the action recognition device of the user virtual image also includes: a fifth acquisition module, used to obtain the target video file of the target user, wherein the target video file is a video file including the face of the target user; a first extraction module, used to extract the facial emotion features of the target user from the target video file; and a first optimization module, used to optimize the action recognition result based on the facial emotion features.

[0021] Optionally, the action recognition device for the user virtual image also includes: a first mapping module, used to map the action recognition result with the action library of the virtual image to obtain an action instruction; and a first execution module, used to control the virtual image to perform a corresponding action based on the action instruction.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned user virtual image action recognition methods.

[0023] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for recognizing the action of a user virtual image.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is further provided, the computer program product comprising a computer program, wherein the computer program implements any one of the above-mentioned methods for recognizing the action of a user virtual image when executed by a processor.

[0025] In the present application, the following steps are performed: first, a video file of a target user collected based on N cameras is obtained to obtain N video files, wherein N is a positive integer; then, for each video file, the video file is input into a neural network model corresponding to the camera through a camera, and action recognition data corresponding to the camera is output, wherein the neural network model is a pre-trained model for action recognition; finally, all action recognition data are fused based on the action recognition data corresponding to each camera and the weight coefficient of the camera to obtain an action recognition result of a virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera.

[0026] In the present application, multiple cameras are used to collect the target user's movements from different dimensions to obtain video files. The cameras are connected to a cloud platform, and each camera corresponds to a neural network model. The video files are processed based on the neural network model to identify the target user's movements. The results identified by all cameras are fused based on the weight coefficients corresponding to each camera to obtain the final movement recognition result. The video files can be used to more accurately analyze the target user's movement information and improve the accuracy of movement recognition, thereby solving the technical problem of low movement recognition accuracy in the related technology based on the method of capturing the movement of the target user by sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 is a flow chart of an optional method for identifying the action of a user virtual image according to an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of an optional action recognition process according to an embodiment of the present invention;

[0030] Figure 3 is a schematic diagram of an optional device for identifying a user's virtual image's action according to an embodiment of the present invention;

[0031] Figure 4 The present invention is a hardware structure block diagram of an electronic device (or mobile device) for a method for recognizing a user virtual image's actions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the method and device for identifying the action of a user virtual image in the present application can be used in the field of artificial intelligence (when identifying the action of a user virtual image based on artificial intelligence), and can also be used in any field other than the field of artificial intelligence (when identifying the action of a user virtual image based on artificial intelligence). The present application does not limit the application field of the method and device for identifying the action of a user virtual image.

[0035] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0036] It should be noted that in this application, when collecting and analyzing customer information, corresponding operation entrances are provided for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0037] The following embodiments of the present invention can be applied to systems / applications / devices for motion recognition of various user virtual images. The present invention uses multiple camera devices to capture the target user's motion from multiple dimensions and generate video files. The motion is recognized through a neural network model deployed on a cloud platform to obtain motion recognition results. Finally, all recognition results are fused based on the weight coefficient of each device to obtain the final motion recognition result. Based on the final motion recognition result, the virtual image of the target user is controlled to perform related actions. The target user's motion can be analyzed more carefully through the video file to improve the accuracy of the motion recognition results. At the same time, the requirements for motion capture devices are low, and no professional motion capture devices are required, which saves costs while improving the applicability of the solution.

[0038] The present invention is described in detail below in conjunction with various embodiments.

[0039] Embodiment 1

[0040] According to an embodiment of the present invention, an embodiment of a method for motion recognition of a user virtual image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 1 is a flow chart of an optional method for identifying the action of a user virtual image according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0042] Step S101, obtaining video files of a target user collected by N camera devices, and obtaining N video files, where N is a positive integer;

[0043] Step S102, for each video file, input the video file into a neural network model corresponding to the camera device through a camera device, and output action recognition data corresponding to the camera device, wherein the neural network model is a pre-trained model for action recognition;

[0044] Step S103, all the action recognition data are integrated based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device to obtain the action recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

[0045] Through the above steps, first obtain the video file of the target user collected by N cameras to obtain N video files, where N is a positive integer, and then for each video file, input the video file into the neural network model corresponding to the camera through the camera, and output the action recognition data corresponding to the camera, where the neural network model is a pre-trained model for action recognition, and finally fuse all the action recognition data based on the action recognition data corresponding to each camera and the weight coefficient of the camera to obtain the action recognition result of the virtual image corresponding to the target user, where the weight coefficient is configured based on the multi-dimensional features of the camera.

[0046] In this embodiment, the movements of the target user are collected from different dimensions by multiple cameras to obtain video files. The cameras are connected to the cloud platform, and each camera corresponds to a neural network model. The video files are processed based on the neural network model to identify the movements of the target user. The results identified by all cameras are fused based on the weight coefficients corresponding to each camera to obtain the final movement recognition result. The movement information of the target user can be analyzed more accurately through the video files, and the accuracy of movement recognition is improved, thereby solving the technical problem of low accuracy of movement recognition based on sensor-based movement capture of the target user in related technologies.

[0047] The embodiment of the present invention is described in detail below in combination with the above steps.

[0048] With the development and progress of metaverse technology, its application scenarios are gradually expanding. For example, in online live broadcast activities, a virtual image is created based on the target user, and the virtual image is controlled to synchronously execute the relevant actions of the target user, so that the target user can drive the virtual character to perform activities such as dancing and sports. For example, in the game field, by capturing the target user's motion, the virtual image can be controlled to avoid obstacles and complete tasks based on the results of motion recognition, thereby enhancing the user's gaming experience. In short, the existing virtual image technology has been widely used in various fields. How to efficiently and quickly assign the target user's actions to the virtual image and control the virtual image to synchronously execute related actions has become a problem that urgently needs to be solved.

[0049] The embodiment of the present invention is based on artificial intelligence and cloud computing. The motion data of the target user is collected from multiple angles through multiple devices to form a video file. The camera device establishes a connection with the cloud server, and the collected video file can be uploaded to the corresponding neural network model on the cloud server. The models on the cloud server are independent of each other and are trained and used separately. Each model generates the user's motion recognition data based on the video file collected by its corresponding camera device as input data. The motion recognition data is used to characterize the spatial position coordinates and specific actions of key points on the user. Multiple motion recognition results are output based on the neural network model, and then a comprehensive calculation is performed based on the weight coefficients of each camera device to obtain the final motion recognition result, and the motion recognition result is converted into an action instruction to control the virtual image to synchronously execute the target user's action to achieve human-computer interaction. Motion capture and recognition can be achieved through ordinary camera devices, thereby enhancing the adaptability of this embodiment. At the same time, the neural network model based on cloud computing can quickly output the motion recognition result, and a more accurate motion recognition result can be generated based on the weight coefficient, thereby improving the accuracy of motion recognition.

[0050] It should be noted that before performing action recognition, the embodiment of the present invention needs to configure weight coefficients for the camera devices according to the characteristics of each camera device. Due to differences in performance, shooting angles, model algorithm recognition accuracy, etc. of different camera devices, user actions recognized based on the video stream of any camera device are not reliable enough. The recognition results of multiple camera devices need to be integrated to obtain more reliable action recognition results.

[0051] Optionally, before obtaining the video files of the target user captured based on N cameras, the method further includes: obtaining performance parameters of the camera devices; and configuring a first confidence level for the camera devices based on the performance parameters.

[0052] It should be noted that when configuring the weight coefficient for the camera device, comprehensive consideration can be made from multiple dimensions. First, based on the performance parameters of the camera device, a first confidence level is configured for the camera device. This embodiment has no restrictions on the professionalism of the camera device. Ordinary devices with camera functions (for example: cameras on mobile devices, cameras on PCs, tablet cameras, etc.) can be used for shooting. The performance parameters corresponding to different camera devices are definitely different. The performance parameters may include: the pixels of the camera device, the frame rate of the camera device, etc., and a higher confidence level is configured for cameras with better performance parameters.

[0053] Alternatively, when configuring the first confidence level, multiple performance parameters may be selected and importance levels may be configured for the performance parameters. The first confidence level may be configured based on a comprehensive analysis of the numerical values ​​of the performance parameters and the importance levels of the performance parameters.

[0054] Optionally, after configuring the first confidence level for the camera device based on the performance parameter, the method further includes: acquiring a position relationship between the camera device and the target user; and configuring a second confidence level for the camera device based on the position relationship.

[0055] It should be noted that the embodiment of the present invention collects the motion data of the target user through multiple cameras. The cameras are placed at different positions and have different positional relationships with the target users. Therefore, a second confidence level can be configured based on the positional relationship between the cameras and the target user. For example, a camera that shoots video from the front is given a higher confidence level, and a camera that shoots from the side, rear or back is given a lower confidence level.

[0056] Optionally, after configuring the second confidence level for the camera device based on the positional relationship, the method further includes: acquiring detail feature data captured by the camera device; and configuring a third confidence level for the camera device based on the detail feature data.

[0057] It should be noted that based on different shooting angles, the light and shadow conditions of the pictures captured by the camera device are different. The camera device with good light and shadow conditions can capture video pictures with richer details, and then can identify more detail features. The user actions derived from this are also more accurate. Therefore, this embodiment configures a third confidence level based on the detail feature data captured by the camera device, and configures a higher confidence level for the camera device that captures more detail features.

[0058] Optionally, after configuring the third confidence level for the camera device based on the detail feature data, the method further includes: obtaining a situation in which the camera device is blocked by an obstacle; and configuring a fourth confidence level for the camera device based on the blocking situation.

[0059] It should be noted that the camera device may be blocked for some reasons. Therefore, it is necessary to configure the fourth confidence level based on the situation of the camera device being blocked. For example, in some pictures, the user's body may be partially blocked by the table or chair. The larger the blocked area of ​​the picture, the lower the confidence level assigned to the camera device.

[0060] Optionally, before fusing all action recognition data based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device, the method also includes: calculating the average of the first confidence, the second confidence, the third confidence and the fourth confidence to obtain the confidence average; and configuring the weight coefficient for the camera device based on the confidence average.

[0061] In addition, confidence levels can be configured for the camera device based on multi-dimensional information such as the continuity of the motion captured by the camera device, the accuracy of the model's recognition results for the camera device, and the consistency between the emotion labels of the motion captured by the camera device and the emotion labels of the user. The weight coefficient of the camera device is obtained by comprehensive calculation based on the first confidence level, the second confidence level, the third confidence level, the fourth confidence level and multiple other confidence levels.

[0062] Step S101: Obtain video files of a target user collected by N camera devices to obtain N video files.

[0063] It should be noted that the above N is a positive integer. The embodiment of the present invention is based on multiple camera devices prepared by the target user to collect motion. The multiple camera devices are deployed around the target user. The camera device can be an ordinary camera with a camera function, such as an external camera of a computer, a camera built into a smart device such as a mobile phone or a tablet. The camera device can be used to shoot the user's face and body respectively. There can be multiple cameras for shooting the body, and they can shoot from different angles and directions. In addition, the camera devices can be connected to a motion capture application, which analyzes the video image to obtain the user's motion data. The motion capture application can be deployed in a cloud server and connected to each camera device via a wireless network.

[0064] Step S102: for each video file, the video file is input into a neural network model corresponding to the camera device through a camera device, and action recognition data corresponding to the camera device is output.

[0065] It should be noted that the neural network model is a pre-trained model for action recognition. Each camera device corresponds to a neural network model, and the neural network model is deployed in a cloud server. The cloud server includes the same number of neural network models as the cameras. Each neural network model is independent of each other and is trained and used separately. Each model is iteratively trained based on the video files collected by its corresponding camera device as samples. When in use, the video files collected by the camera device are input into the neural network model corresponding to the camera device to generate the user's action recognition data. The action recognition data is used to characterize the spatial position coordinates and action descriptions of key points on the user.

[0066] Step S103 , based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device, all action recognition data are integrated to obtain the action recognition result of the virtual image corresponding to the target user.

[0067] It should be noted that the weight coefficient is configured based on the multi-dimensional features of the camera device. The weight coefficient is configured for the camera device in advance. When performing action recognition, the weight coefficient of each camera device is directly obtained, and the action recognition data output by each neural network model is fused and calculated based on the weight coefficient, so as to obtain the action recognition result of the virtual image corresponding to the target user.

[0068] Optionally, after obtaining the action recognition result of the target user's corresponding virtual image, it also includes: obtaining a target video file of the target user, wherein the target video file is a video file including the target user's face; extracting facial emotion features of the target user from the target video file; and optimizing the action recognition result based on the facial emotion features.

[0069] It should be noted that the user's actions can express the user's emotions. In order to obtain the target user's action recognition results more accurately, the video file corresponding to the target user's face can be obtained by using a camera that captures the user's face, and the target user's facial emotion features can be extracted from the facial video file. Based on the facial emotion features, the fused action recognition results can be optimized, so that the emotions conveyed by the target user's body movements are consistent with the user's facial emotions, thereby improving the accuracy of action recognition.

[0070] Optionally, after obtaining the action recognition result of the target user corresponding to the virtual image, the method further includes: mapping the action recognition result with the action library of the virtual image to obtain an action instruction; and controlling the virtual image to perform a corresponding action based on the action instruction.

[0071] It should be noted that after identifying the target user's action, it is necessary to assign the target user's action to the virtual image corresponding to the target user, and control the virtual image to synchronously execute the action of the target user entity. Therefore, it is necessary to map the action recognition result with the action library of the virtual image to obtain the action instruction. The action library stores specific action packages and action instructions corresponding to the actions, and then control the virtual image of the target user to execute the action instruction and make the same action as the target user entity.

[0072] Another optional specific implementation is described in detail below.

[0073] Figure 2 is a schematic diagram of an optional action recognition process according to an embodiment of the present invention. Figure 2 As shown in the figure, the entire action recognition process includes:

[0074] Step 1, start;

[0075] Step 2: photographing the target user using multiple camera devices;

[0076] Step 3, outputting action recognition data based on the neural network model corresponding to the camera device;

[0077] Step 4, obtaining the weight coefficient of each camera device;

[0078] The weight coefficient of the camera device is calculated based on the pre-configured confidence level. Specifically,

[0079] Configuring a confidence level based on performance parameters of the camera device to obtain a first confidence level;

[0080] Configuring a confidence level based on an angle relationship between the camera device and the user to obtain a second confidence level;

[0081] Based on the light and shadow conditions of the picture captured by the camera device, that is, based on the detail feature data identified from the captured video file, a confidence level is configured to obtain a third confidence level;

[0082] Configuring a confidence level based on an obstacle blocking condition of the camera device to obtain a fourth confidence level;

[0083] Configuring a confidence level based on the action continuity captured by the camera device to obtain a fifth confidence level;

[0084] Confidences are configured based on the differences between the action results recognized by the multiple camera devices to obtain a sixth confidence level, and the user actions recognized by the multiple models at the same time are compared. If the recognition results of most models are similar, and the recognition result of a certain model is significantly different from the recognition results of other models, a lower confidence level is configured for the camera device corresponding to the model;

[0085] Configuring confidence based on consistency between the action emotion labels and the user emotion labels identified by multiple camera devices to obtain a seventh confidence;

[0086] Configuring confidence levels based on the parts photographed by the camera device to obtain an eighth confidence level;

[0087] Finally, the weight coefficient of each camera device is comprehensively calculated based on the above multiple confidence levels.

[0088] Step 5: Based on the action recognition data corresponding to the camera device and the weight coefficient of the camera device, all output action recognition data are integrated to obtain the final action recognition result of the target user;

[0089] Step 6: Mapping the action recognition results into action instructions, driving the virtual person (i.e., the virtual image) to execute the instructions to carry out activities, and controlling the virtual person to synchronously execute the relevant actions of the target user;

[0090] Step seven, end.

[0091] The embodiment of the present invention captures the target user's actions from multiple dimensions through multiple camera devices and generates video files, recognizes the actions through a neural network model deployed on a cloud platform to obtain action recognition results, and finally fuses all recognition results based on the weight coefficient of each device to obtain a final action recognition result, and controls the target user's virtual image to perform related actions based on the final action recognition result. The target user's actions can be analyzed in more detail through the video files to improve the accuracy of the action recognition results. At the same time, the requirements for action capture devices are low, and no professional action capture equipment is required, which saves costs while improving the applicability of the solution.

[0092] The following is a detailed description in conjunction with another embodiment.

[0093] Embodiment 2

[0094] A user virtual image motion recognition device provided in this embodiment includes multiple implementation units, each implementation unit corresponds to each implementation step in the above-mentioned embodiment 1. Its specific implementation method and beneficial effects can refer to the aforementioned method embodiment and will not be repeated here.

[0095] Figure 3 is a schematic diagram of an optional user virtual image action recognition device according to an embodiment of the present invention, such as Figure 3 As shown, the action recognition device of the user virtual image may include: an acquisition unit 31, an output unit 32, and a fusion unit 33, wherein:

[0096] An acquisition unit 31 is used to acquire video files of a target user collected by N camera devices to obtain N video files, where N is a positive integer;

[0097] The output unit 32 is used for, for each video file, inputting the video file into the neural network model corresponding to the camera device through the camera device, and outputting the action recognition data corresponding to the camera device, wherein the neural network model is a pre-trained model for action recognition;

[0098] The fusion unit 33 is used to fuse all the action recognition data based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device to obtain the action recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

[0099] The above-mentioned action recognition device for the user virtual image obtains the video file of the target user collected by N cameras through the acquisition unit 31, and obtains N video files, wherein N is a positive integer; for each video file, the video file is input into the neural network model corresponding to the camera device through the output unit 32, and the action recognition data corresponding to the camera device is output, wherein the neural network model is a pre-trained model for action recognition; all the action recognition data are fused based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device through the fusion unit 33, to obtain the action recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

[0100] In this embodiment, the movements of the target user are collected from different dimensions by multiple cameras to obtain video files. The cameras are connected to the cloud platform, and each camera corresponds to a neural network model. The video files are processed based on the neural network model to identify the movements of the target user. The results identified by all cameras are fused based on the weight coefficients corresponding to each camera to obtain the final movement recognition result. The movement information of the target user can be analyzed more accurately through the video files, and the accuracy of movement recognition is improved, thereby solving the technical problem of low accuracy of movement recognition based on sensor-based movement capture of the target user in related technologies.

[0101] Optionally, the action recognition device for the user virtual image also includes: a first acquisition module, used to obtain performance parameters of the camera device; and a first configuration module, used to configure a first confidence level for the camera device based on the performance parameters.

[0102] Optionally, the motion recognition device for the user virtual image also includes: a second acquisition module, used to obtain the position relationship between the camera device and the target user; and a second configuration module, used to configure a second confidence level for the camera device based on the position relationship.

[0103] Optionally, the motion recognition device for the user virtual image also includes: a third acquisition module, used to acquire detail feature data captured by the camera device; and a third configuration module, used to configure a third confidence level for the camera device based on the detail feature data.

[0104] Optionally, the motion recognition device for the user virtual image also includes: a fourth acquisition module, used to obtain the situation where the camera device is blocked by obstacles; and a fourth configuration module, used to configure a fourth confidence level for the camera device based on the blocking situation.

[0105] Optionally, the action recognition device for the user virtual image also includes: a first calculation module, used to calculate the average of the first confidence, the second confidence, the third confidence and the fourth confidence to obtain the confidence average value; a fifth configuration module, used to configure a weight coefficient for the camera device based on the confidence average value.

[0106] Optionally, the action recognition device of the user virtual image also includes: a fifth acquisition module, used to obtain the target video file of the target user, wherein the target video file is a video file including the face of the target user; a first extraction module, used to extract facial emotion features of the target user from the target video file; and a first optimization module, used to optimize the action recognition results based on the facial emotion features.

[0107] Optionally, the action recognition device for the user virtual image further includes: a first mapping module, used to map the action recognition result with the action library of the virtual image to obtain an action instruction; and a first execution module, used to control the virtual image to execute a corresponding action based on the action instruction.

[0108] The above-mentioned user virtual image action recognition device may also include a processor and a memory. The above-mentioned acquisition unit 31, output unit 32, fusion unit 33, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0109] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the target user's actions can be identified and the virtual image can be controlled to perform related actions by adjusting kernel parameters.

[0110] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0111] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned user virtual image action recognition methods.

[0112] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any one of the above-mentioned methods for recognizing the action of a user virtual image.

[0113] According to another aspect of an embodiment of the present invention, a computer program product is further provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, any of the above-mentioned methods for recognizing the action of a user virtual image is implemented.

[0114] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: obtaining a video file of a target user captured by N cameras to obtain N video files, wherein N is a positive integer; for each video file, inputting the video file into a neural network model corresponding to the camera through a camera, and outputting action recognition data corresponding to the camera, wherein the neural network model is a pre-trained model for action recognition; fusing all action recognition data based on the action recognition data corresponding to each camera and the weight coefficient of the camera to obtain an action recognition result of a virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera.

[0115] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: before obtaining a video file of a target user captured based on N cameras, it also includes: obtaining performance parameters of the camera device; and configuring a first confidence level for the camera device based on the performance parameters.

[0116] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: after configuring a first confidence level for a camera device based on performance parameters, it also includes: obtaining a positional relationship between the camera device and a target user; and configuring a second confidence level for the camera device based on the positional relationship.

[0117] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: after configuring a second confidence level for a camera device based on a positional relationship, it also includes: obtaining detail feature data captured by the camera device; and configuring a third confidence level for the camera device based on the detail feature data.

[0118] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: after configuring a third confidence level for a camera device based on detail feature data, it also includes: obtaining a situation in which the camera device is blocked by an obstacle; and configuring a fourth confidence level for the camera device based on the blocking situation.

[0119] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing an initialization program having the following method steps: before fusing all action recognition data based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device, the method also includes: calculating the average value of the first confidence level, the second confidence level, the third confidence level and the fourth confidence level to obtain the confidence average value; configuring the weight coefficient for the camera device based on the confidence average value.

[0120] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: after obtaining the action recognition result of the target user's corresponding virtual image, it also includes: obtaining a target video file of the target user, wherein the target video file is a video file including the target user's face; extracting the target user's facial emotion features from the target video file; and optimizing the action recognition results based on the facial emotion features.

[0121] The present application also provides a computer program product, which, when executed on a data processing device, is also suitable for executing a program that initializes the following method steps: after obtaining the action recognition result of the target user corresponding to the virtual image, it also includes: mapping the action recognition result with the action library of the virtual image to obtain the action instruction; and controlling the virtual image to perform the corresponding action based on the action instruction.

[0122] Figure 4 1 is a hardware structure block diagram of an electronic device (or mobile device) for a method for recognizing a user virtual image's action according to an embodiment of the present invention. Figure 4 As shown, the electronic device may include one or more ( Figure 4 402a, 402b, ..., 402n are used to illustrate) processor 402 (processor 402 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), memory 404 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0123] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0127] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0129] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying the actions of a user's virtual image, characterized in that: include: Obtaining video files of a target user collected by N camera devices to obtain N video files, where N is a positive integer; For each of the video files, the video file is input into a neural network model corresponding to the camera device through the camera device, and action recognition data corresponding to the camera device is output, wherein the neural network model is a pre-trained model for action recognition; Based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device, all the action recognition data are fused to obtain the action recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

2. The identification method according to claim 1, characterized in that: Before obtaining the video files collected by the target user based on N cameras, the following steps are also included: Obtaining performance parameters of the camera device; A first confidence level is configured for the camera device based on the performance parameter.

3. The identification method according to claim 2, characterized in that: After configuring a first confidence level for the camera device based on the performance parameter, the method further includes: Acquire the position relationship between the camera device and the target user; A second confidence level is configured for the camera device based on the positional relationship.

4. The identification method according to claim 3, characterized in that: After configuring a second confidence level for the camera device based on the positional relationship, the method further includes: Acquire detailed feature data captured by the camera device; A third confidence level is configured for the camera device based on the detail feature data.

5. The identification method according to claim 4, characterized in that: After configuring a third confidence level for the camera device based on the detail feature data, the method further includes: Obtaining a situation in which the camera device is blocked by an obstacle; A fourth confidence level is configured for the camera device based on the occlusion situation.

6. The identification method according to claim 5, characterized in that: Before fusing all the action recognition data based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device, the method further includes: Calculating an average of the first confidence level, the second confidence level, the third confidence level, and the fourth confidence level to obtain an average confidence level; A weight coefficient is configured for the camera device based on the confidence average value.

7. The identification method according to claim 1, characterized in that: After obtaining the action recognition result corresponding to the virtual image of the target user, the method further includes: Acquire a target video file of the target user, wherein the target video file is a video file including the face of the target user; Extracting facial emotion features of the target user from the target video file; The action recognition result is optimized based on the facial emotion feature.

8. The identification method according to claim 1, characterized in that: After obtaining the action recognition result corresponding to the virtual image of the target user, the method further includes: Mapping the action recognition result with the action library of the virtual image to obtain an action instruction; The virtual image is controlled to perform corresponding actions based on the action instruction.

9. A user avatar motion recognition device, characterized in that: include: An acquisition unit, used to acquire video files of a target user collected by N camera devices, to obtain N video files, where N is a positive integer; An output unit, for each of the video files, inputting the video file into a neural network model corresponding to the camera device through the camera device, and outputting action recognition data corresponding to the camera device, wherein the neural network model is a pre-trained model for action recognition; A fusion unit is used to fuse all the action recognition data based on the action recognition data corresponding to each camera device and the weight coefficient of the camera device to obtain the action recognition result of the virtual image corresponding to the target user, wherein the weight coefficient is configured based on the multi-dimensional features of the camera device.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for identifying the action of a user virtual image according to any one of claims 1 to 8.

11. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the user virtual image motion recognition method described in any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program, wherein When the computer program is executed by a processor, the method for recognizing the action of a user virtual image as claimed in any one of claims 1 to 8 is implemented.

Images