Motion recognition method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides an action recognition method, the method comprising: obtaining an image to be recognized; extracting target image features from the image to be recognized, the target image features being associated with key parts of the target object performing an action, and associated parts having an associated relationship with the key parts; based on the target image features, detecting the part of the target object performing an action in the image to be recognized, and obtaining a part detection result; based on the part detection result, recognizing the action performed by the target object in the image to be recognized, and obtaining an action recognition result. The method provided by the present application can effectively solve the problem of low action recognition accuracy in the prior art.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an action recognition method, device, electronic device and storage medium. Background Art

[0002] At present, motion recognition is mainly used in public places, hospitals, security, control and other interactive / monitoring scenarios, in order to provide a more intuitive, natural and friendly and more familiar interactive / monitoring method. It is understandable that, especially in some interactive scenarios, such as home scenarios, in order to better control smart home devices, more complex actions can be designed for home users to distinguish different smart home devices or control the same smart home devices to perform different operations. Of course, there are also some interactive scenarios, such as museums, exhibition halls and other public places, because visitors are untrained users, and the actions designed for them are mostly simple and intuitive.

[0003] Existing action recognition usually adopts the detection + classification method. However, whether it is a more complex action or a simple and intuitive action, when detecting the key parts that perform the action (such as hands), especially in low light or in the presence of some objects similar to the key parts (such as hand toys), it is easy to misdetect the key parts, which makes it easy to misrecognize the action based on the aforementioned misdetection results during action classification, and ultimately cause false triggering in the interactive scene.

[0004] This shows that the existing action recognition still has the defect of low accuracy. Summary of the invention

[0005] The embodiments of the present application provide a motion recognition method, device, electronic device and storage medium, which can solve the problem of low accuracy of motion recognition existing in the related art.

[0006] The technical solutions adopted in this application are as follows:

[0007] According to one aspect of an embodiment of the present application, an action recognition method includes: acquiring an image to be recognized; extracting target image features from the image to be recognized, wherein the target image features are associated with key parts of a target object performing an action, and associated parts that have an associated relationship with the key parts; based on the target image features, detecting the parts of the image to be recognized where the target object performs an action to obtain a part detection result; based on the part detection result, recognizing the action performed by the target object in the image to be recognized to obtain an action recognition result.

[0008] According to one aspect of an embodiment of the present application, an action recognition device includes: an image acquisition module, used to acquire an image to be recognized; a feature extraction module, used to extract target image features from the image to be recognized, wherein the target image features are associated with key parts of a target object performing an action, and associated parts that have an associated relationship with the key parts; a part detection module, used to detect the part of the target object performing an action in the image to be recognized based on the target image features, and obtain a part detection result; and an action recognition module, used to recognize the action performed by the target object in the image to be recognized based on the part detection result, and obtain an action recognition result.

[0009] In an exemplary embodiment, the part detection module includes: a key part detection unit, which is used to detect the key parts of the target object performing the action in the image to be identified according to the target image features, and obtain a key part detection result; an associated part detection unit, which is used to detect the associated parts in the image to be identified that have an associated relationship with the key parts according to the target image features, and obtain an associated part detection result; and a part detection result determination unit, which is used to determine the part detection result based on the key part detection result and the associated part detection result.

[0010] In an exemplary embodiment, the associated part detection unit includes: a position determination subunit, used to determine the relative position of the associated part relative to the key part in the image to be identified based on the target image features; and an associated part determination subunit, used to determine the associated part detection result based on the relative position of the associated part relative to the key part.

[0011] In an exemplary embodiment, the part detection module includes: a position positioning unit, which is used to locate the key part of the target object performing the action in the area of ​​the image to be identified based on the target image features, and obtain the position of the key part of the target object performing the action; a key part determination unit, which is used to determine the key part detection result according to the position of the key part of the target object performing the action; and a result determination unit, which is used to determine the part detection result according to the key part detection result.

[0012] In an exemplary embodiment, the action recognition module includes: an area determination unit, which is used to determine the area of ​​the part where the target object performs the action in the image to be recognized based on the part detection result; a category prediction unit, which is used to predict the action category of the action performed by the target object in the image to be recognized based on the area of ​​the part where the target object performs the action in the image to be recognized, and obtain the action category of the action performed by the target object; and an action recognition result determination unit, which is used to determine the action recognition result based on the action category of the action performed by the target object.

[0013] In an exemplary embodiment, the action recognition result is obtained by calling an action recognition model; the action recognition model is obtained through a training module; the training module is used to train an initial machine learning model based on at least one sample image; the training module at least includes: a first supervised training unit, used to perform supervised training on key part detection and associated part detection on the machine learning model based on a first label and a second label carried by the sample image during the part detection stage, wherein the first label is used to indicate the position of a key part of the sample object performing an action, and the second label is used to indicate the relative position of the associated part relative to the key part.

[0014] In an exemplary embodiment, the first supervised training unit includes: a feature extraction subunit, used to extract sample image features from the sample image; a first prediction subunit, used to predict the position of the key part of the sample object performing an action in the sample image based on the sample image features to obtain first prediction information, and predict the relative position of the associated part in the sample image relative to the key part to obtain second prediction information; a first loss determination subunit, used to determine a first loss based on the difference between the first label and the first prediction information, and to determine a second loss based on the difference between the second label and the second prediction information; a first parameter adjustment subunit, used to adjust the parameters of the machine learning model corresponding to the part detection stage based on the first loss and the second loss.

[0015] In an exemplary embodiment, the training module also includes: a second supervised training unit, which is used to perform supervised training on the machine learning model for action recognition in the action classification stage based on the third label carried by the sample image and the supervised training results of the part detection stage, wherein the third label is used to indicate the action category of the action performed by the sample object; the second supervised training unit includes: a second prediction subunit, which is used to predict the action category of the action performed by the sample object in the sample image according to the first prediction information and the second prediction information in the supervised training results, and obtain third prediction information; a second loss determination subunit, which is used to determine a third loss based on the difference between the third label and the third prediction information; a second parameter adjustment subunit, which is used to adjust the parameters of the machine learning model corresponding to the action classification stage according to the third loss, and / or the first loss and the second loss in the supervised training results; a model generation subunit, which is used to obtain a machine learning model that has completed training as the action recognition model when the training stop condition is met.

[0016] In an exemplary embodiment, the device also includes: a loss feedback module, used to feed back the third loss to the part detection stage; the first parameter adjustment subunit includes: a joint adjustment subunit, used to adjust the parameters of the machine learning model corresponding to the part detection stage according to the first loss, the second loss and the third loss.

[0017] In an exemplary embodiment, the first prediction information is obtained through the prediction of the key part detection branch in the machine learning model, and the second prediction information is obtained through the prediction of the associated part detection branch in the machine learning model; the joint adjustment subunit includes: a first branch parameter adjustment subunit, used to adjust the parameters of the key part detection branch according to the first loss, the second loss and the third loss; and a second branch parameter adjustment subunit, used to adjust the parameters of the associated part detection branch according to the second loss and the third loss.

[0018] According to one aspect of an embodiment of the present application, an electronic device includes: at least one processor, at least one memory, and at least one communication bus, wherein a computer program is stored in the memory, and the processor reads the computer program in the memory through the communication bus; when the computer program is executed by the processor, the action recognition method as described above is implemented.

[0019] According to one aspect of an embodiment of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor, the action recognition method described above is implemented.

[0020] According to one aspect of an embodiment of the present application, a computer program product includes a computer program, the computer program is stored in a storage medium, a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that when the computer device executes the computer program, the training method of the action recognition model as described above is implemented.

[0021] The beneficial effects of the technical solution provided by this application are:

[0022] Based on the acquired image to be identified, a target image feature is extracted from the image to be identified. Since the target image feature closely associates the key part of the target object performing an action and the associated part that has an associated relationship with the key part, the accuracy of the part detection result obtained by detecting the part of the target object performing an action in the image to be identified based on the target image feature is improved, and the accuracy of the action recognition result obtained by identifying the action performed by the target object in the image to be identified based on the part detection result is also improved. This can not only effectively reduce the false detection rate of non-critical parts, but also avoid the phenomenon of false triggering in interactive scenes, thereby effectively solving the problem of low accuracy of action recognition existing in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.

[0024] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0025] Figure 2 is a flow chart of a method for motion recognition according to an exemplary embodiment;

[0026] Figure 3 is a flow chart of a method for training an action recognition model according to an exemplary embodiment;

[0027] Figure 4 is a schematic diagram of human body joints according to an exemplary embodiment;

[0028] Figure 5 is a schematic diagram of a training process and a prediction process of an action recognition model according to an exemplary embodiment;

[0029] Figure 6 yes Figure 2 A schematic diagram of step 250 in one embodiment of the corresponding embodiment;

[0030] Figure 7 yes Figure 2A schematic diagram of step 270 in one embodiment of the corresponding embodiment;

[0031] Figure 8 is a structural diagram of a motion recognition device according to an exemplary embodiment;

[0032] Fig. 9 is a hardware structure diagram of a server according to an exemplary embodiment;

[0033] Fig.10 The diagram is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0035] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0036] The following is an introduction and explanation of several terms involved in this application:

[0037] Action recognition: usually includes two main tasks: part detection and action classification. Part detection refers to where the key part of the action may appear in the image (or a certain area), and the area of ​​the key part in the image is framed by a bounding box (abbreviated as Bbox). Action classification refers to which category the action performed by the key part framed by the bounding box in the image (or a certain area) belongs to.

[0038] Deep learning model: A machine learning model that uses supervised and unsupervised learning methods to train a deep neural network, which is a neural network structure containing multiple hidden layers. As deep learning has made great progress in image classification tasks, the current mainstream recognition algorithms are mainly based on deep learning models.

[0039] Precision: From the perspective of prediction results, it refers to the number of data predicted as positive samples that are actually positive samples, that is, the proportion of "correct finds". The calculation formula is: predicted to be positive and actually positive / predicted to be positive. The larger the value, the better, and 100% is the ideal state.

[0040] Recall rate: It is a statistical method based on the real sample image set. It refers to the number of actual positive samples found among all positive samples, that is, the proportion of "full find". The calculation formula is: predicted positive and actually positive / total number of positive samples. The larger the value, the better. 100% is the ideal state.

[0041] As mentioned above, motion recognition is mainly used in public places, hospitals, security, control and other interactive / monitoring scenarios, in order to provide a more intuitive, natural and friendly interactive / monitoring method that is more familiar to people. For example, in the monitoring scenarios formed in public places such as museums and exhibition halls, it monitors whether visitors have actions related to uncivilized behaviors such as smoking and taking photos, or in interactive scenarios such as smart home scenarios, it identifies whether home users have actions associated with smart home devices such as looking up, raising hands, squatting, etc.

[0042] In order to ensure that the actions in these interactive / monitoring scenarios can be robustly recognized and with as few false alarms as possible, existing action recognition usually adopts a detection + classification method. For example, a part detection model and an action classification model are used to detect the key parts that perform actions in the interactive / monitoring scenarios and identify the action categories of the actions. Specifically, in the training process, the part detection model and the action classification model are trained using images that are labeled with the areas where the key parts are located and the action categories. In the prediction process, the part detection model first detects the key parts that perform the action, and then enters the action classification stage based on the detected key parts, thereby outputting the action category of the action performed by the key parts, thereby completing the entire action recognition process.

[0043] The above-mentioned action recognition process, whether for more complex actions or simple and intuitive actions, can realize the action recognition function, but in low light or in the presence of some objects similar to key parts (such as hands) (such as hand toys), it is easy to misdetect key parts, which makes it easy to misrecognize the action based on the above-mentioned misdetection results when classifying the action, and finally cause false triggering in the interactive scene.

[0044] From the above, we can see that the existing action recognition still has the limitation of low accuracy.

[0045] In order to overcome the above problems, the present application provides a motion recognition method, which can effectively reduce the false detection rate of non-critical parts, so as to avoid false triggering in interactive scenarios. Accordingly, this motion recognition method is suitable for a motion recognition device, which can be deployed in an electronic device equipped with a von Neumann architecture, for example, the electronic device can be a personal computer (PC), a laptop computer, a server, etc.

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0047] Figure 1 A schematic diagram of an implementation environment involved in a target recognition method. In one embodiment, the implementation environment is applicable to a smart home scenario, as shown in FIG. (a), and the implementation environment includes a gateway 110, an image acquisition device 130 deployed in the gateway 110, a server 150, and a plurality of control devices 170.

[0048] Specifically, the image acquisition device 130 can be a video camera, a still camera, or other electronic devices equipped with a camera, such as a smart phone or a tablet computer, which is not limited here.

[0049] The server 150 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. For example, in this implementation environment, the server 150 provides motion recognition services.

[0050] The image acquisition device 130 is deployed in the gateway 110, and communicates with the gateway 110 through its own configured communication module, thereby realizing interaction with the gateway 110. In one embodiment, the image acquisition device 130 accesses the gateway 110 through a local area network, and is thus deployed in the gateway 110. The process of the image acquisition device 130 accessing the gateway 110 through the local area network includes that the gateway 110 first establishes a local area network, and the image acquisition device 130 accesses the local area network established by the gateway 110 by connecting to the gateway 110. The local area network includes: Bluetooth, WIFI, ZIGBEE or LORA.

[0051] The server 150 establishes a communication connection with the gateway 110 in advance. In one embodiment, the server 150 establishes a communication connection with the gateway 110 via 2G / 3G / 4G / 5G, WIFI, etc. The communication connection is used to realize data transmission with the gateway 110. For example, the transmitted data at least includes an image to be recognized.

[0052] Similar to the image acquisition device 130, the control device 170 is also deployed in the gateway 110, and communicates with the gateway 110 through its own configured communication module, and is then controlled by the gateway 110. In one embodiment, the control device 170 accesses the gateway 110 through a local area network, and is thus deployed in the gateway 110. The process of the control device 170 accessing the gateway 110 through the local area network includes that the gateway 110 first establishes a local area network, and the control device 170 accesses the local area network established by the gateway 110 by connecting to the gateway 110. The local area network includes: Bluetooth, ZIGBEE or LORA.

[0053] Among them, the control device 170 includes but is not limited to: smart TVs, smart printers, smart fax machines, smart speakers, smart cameras, smart air conditioners, smart desk lamps, smart door locks or human body sensors, door and window sensors, temperature and humidity sensors, water immersion sensors, visual sensors, natural gas alarms, smoke alarms, buzzer alarms, wall switches, wall sockets, wireless switches, wireless wall sticker switches, Rubik's Cube controllers, curtain motors, etc., which are not specifically limited here.

[0054] In this implementation environment, if a home user wants to control the device 170 to perform related operations, he can perform the actions corresponding to the related operations in the smart home scene. At this time, the image acquisition device 130 can capture and collect the actions performed by the home user to generate an image to be identified, and send the image to be identified to the server 150 through the gateway 110, so that the server 150 provides action recognition services.

[0055] As for the server 150 , after receiving the image to be recognized, it can detect and recognize the image to be recognized, thereby obtaining a motion recognition result, and send the motion recognition result to the gateway 110 .

[0056] After the server 150 sends the motion recognition result, the gateway 110 forwards the motion recognition result to the corresponding control device 170, thereby triggering the control device 170 to perform corresponding operations according to the instructions of the motion recognition result, thereby realizing simple and efficient device control.

[0057] Of course, in other implementations, the motion recognition result may also be sent directly from the server 150 to the control device 170 , which does not constitute a specific limitation here.

[0058] Different from Figure (a), in Figure (b), the implementation environment also includes a user terminal 190. The user terminal 190 can be operated by a client with a device control function, which can be a desktop computer, a laptop computer, a tablet computer, a smart phone and other electronic devices, which are not limited here. Among them, the client, which has a device control function, can be in the form of an application or a web page. Correspondingly, the user interface for the client to control the device can be in the form of a program window or a web page, which is also not limited here.

[0059] A communication connection is pre-established between the user terminal 190 and the gateway 110, and data transmission between the user terminal 190 and the gateway 110 is achieved through the communication connection. For example, the transmitted data may be the execution result of the control device 170 and the like.

[0060] In this implementation environment, after the control device 170 completes the relevant operation, the execution result can be fed back to the gateway 110. At this time, the gateway 110 can send the execution result to the user terminal 190 for the user terminal 190 to display the execution result, thereby providing a natural, friendly and more familiar interaction method for home users.

[0061] It is worth mentioning that, in one embodiment, the user terminal 190 can also serve as the image acquisition device 130. At this time, the home user can use the camera module configured in the user terminal 190 to shoot and acquire the image to be identified, and use this to initiate a motion recognition request to the server 150, so that the server 150 provides a motion recognition service, thereby realizing the trigger control of the control device 170.

[0062] In another embodiment, motion recognition can be accomplished through a motion recognition model, which can be obtained by offline training of the server 150 and sent to the user terminal 190. At this time, the user terminal 190, as an image acquisition device, also has motion recognition capabilities, and can provide motion recognition services for the captured and collected images to be identified, thereby realizing trigger control of the control device 170.

[0063] See also Figure 2 In an exemplary embodiment, a motion recognition method is provided, and the method is applied to an electronic device as an example for explanation. The electronic device may be Figure 1 The server in Figure 1 The user terminal in.

[0064] The action recognition method may include the following steps:

[0065] Step 210: Obtain an image to be recognized.

[0066] The images to be recognized include dynamic images and static images. Dynamic images refer to videos containing multiple frames of images. Compared with dynamic images, static images refer to photos and pictures containing one frame of images. Based on this, the action recognition in this embodiment can be performed based on videos containing multiple frames of images, or based on photos and pictures containing one frame of images.

[0067] The image to be identified is generated by photographing the environment where the target object is located. As the image acquisition device is arranged around the environment where the target object is located, it is possible to photograph the environment around the target object and acquire images related to the target object, for example, images of the target object performing actions. The environment where the target object is located can be a relatively private scene such as a bedroom, a living room, a hotel room, or a relatively open scene such as a cinema, a library, a conference room, etc., and this embodiment does not limit this.

[0068] In addition, the image acquisition device can be a video camera, a camera, or even other electronic devices with image shooting and acquisition functions, such as smart phones and tablet computers, which are arranged around the environment where the target object is located. For example, the environment around the target object can refer to a corner of the ceiling inside a building, a lamp post outside the building, or a control device such as an intelligent robot. Accordingly, the image to be identified can be any image inside the building or any image outside the building, which is not limited here.

[0069] The target object refers to various objects to be action-recognized in the image to be recognized, for example, at least one of a person, an animal, an intelligent robot, etc., performing gestures, squatting, jumping, etc. in the image to be recognized.

[0070] Step 230: extracting target image features from the image to be identified.

[0071] The target image features are associated with the key parts of the target object for performing actions, and associated parts that have an associated relationship with the key parts.

[0072] In other words, the target image features can not only reflect the key parts of the target object performing the action, but also reflect the associated parts that are associated with the key parts, which is helpful for distinguishing objects that do not have associated parts but are similar to the key parts in the subsequent part detection stage, and thus helps to improve the accuracy of part detection.

[0073] Step 250 , based on the target image features, detect the part where the target object in the image to be identified performs an action, and obtain a part detection result.

[0074] In one embodiment, the part detection result is obtained by detecting the key parts of the target object performing the action in the image to be identified based on the target image features.

[0075] In one embodiment, the part detection result is obtained by detecting the key part of the target object in the image to be identified and the associated part associated with the key part based on the target image features. Specifically, it includes: detecting the key part of the target object in the image to be identified according to the target image features to obtain the key part detection result; detecting the associated part in the image to be identified that has an associated relationship with the key part according to the target image features to obtain the associated part detection result; determining the part detection result according to the key part detection result and the associated part detection result.

[0076] Accordingly, the part detection result is used to indicate whether the key part of the target object performing the action is detected in the image to be identified. In one embodiment, the part detection result is represented by the position of the key part of the target object performing the action.

[0077] Step 270: According to the part detection result, the action performed by the target object in the image to be identified is identified to obtain an action recognition result.

[0078] Among them, recognition is to classify the actions performed by the target object in the image to be recognized, and obtain the action category of the action performed by the target object.

[0079] In one embodiment, the action recognition result is the action category of the action performed by the target object.

[0080] Of course, in other embodiments, a credibility for indicating the credibility of the action category and its corresponding set value are also provided. If the credibility is greater than or equal to the set value, the action recognition model uses the action category of the target object performing the action as the action recognition result; conversely, if the credibility is less than the set value, the action recognition model cannot output the action recognition result. Among them, the set value can be set according to the actual needs of the application scenario, so that the accuracy and recall rate of the action recognition model are balanced. For example, for application scenarios with high accuracy requirements, a relatively high set value is set; for application scenarios with high recall rate requirements, a relatively low set value is set.

[0081] Through the above process, part detection and action recognition are achieved by utilizing the target image features that closely associate the key parts and associated parts of the target object performing the action. This is not only beneficial to improving the detection accuracy of part detection and reducing the false detection of objects that do not have associated parts but are similar to the key parts, but also beneficial to improving the accuracy of action recognition and reducing the false triggering of action recognition on the control device.

[0082] The above-mentioned action recognition process can be implemented by an action recognition algorithm or by calling an action recognition model. The action recognition model is obtained by training an initial machine learning model based on at least one sample image.

[0083] The training method of the action recognition model is now described in detail as follows:

[0084] See also Figure 3 In an exemplary embodiment, a method for training an action recognition model is provided, and the method is applied to an electronic device as an example for explanation. The electronic device may be Figure 1 Servers in .

[0085] The training method of the action recognition model may include the following steps:

[0086] Step 310, obtaining a training set.

[0087] The training set includes at least one sample image, and the sample image refers to a marked and recognized image.

[0088] First of all, it should be noted that the sample images include dynamic images and static images. Dynamic images refer to videos containing multiple frames of images. Compared with dynamic images, static images refer to photos and pictures containing one frame of images. Based on this, the action recognition in this embodiment can be performed based on videos containing multiple frames of images, or based on photos and pictures containing one frame of images.

[0089] The acquisition of sample images may be derived from images captured in real time by an image acquisition device, or may be images captured by an image acquisition device in a historical period of time and pre-stored on a server, such as videos and photos of users performing actions captured and collected by a camera in a smart home scene. Sample images may also be derived from images pre-stored in a third-party image database, such as desktop wallpapers containing animals performing actions, etc., which are not limited here.

[0090] Secondly, the sample image contains at least the key parts and related parts of the sample object performing the action, so as to facilitate the training of the subsequent action recognition model. The sample object refers to various objects performing actions in the sample image, for example, at least one of a person, an animal, an intelligent robot, etc. performing gestures, squatting, jumping, etc. in the sample image.

[0091] Correspondingly, the key part is used to represent the key joint point in the corresponding sample image when the sample object performs an action. The associated part has an associated relationship with the key part and is used to represent at least one other joint point in the corresponding sample image that has an associated relationship with the key joint point when the sample object performs an action. Figure 4is a schematic diagram of the joints of the human body. Now we take a human as an example. Figure 4 As shown, when a person performs a gesture, hand joint 4 or 7 is used as the key joint point. Then, other joint points associated with the hand joint 4 or 7 can be wrist joint 3 or 6, shoulder joint 2 or 5, elbow joint 3 or 6, or head joint 1. It can be seen that the association relationship can specifically refer to the connection relationship between the joints of the human body.

[0092] Again, labeling refers to adding marks to sample images so that the sample images carry labels.

[0093] In one embodiment, the sample image carries at least a first label, and the first label is used to indicate the position of the key part of the sample object performing the action. Specifically, in the sample image, the area where the key part of the sample object performing the action is located is located, and the position of the key part of the sample object performing the action is obtained as the first label.

[0094] As can be seen from the above, the marking of the first label essentially refers to using a positioning frame to frame the area where the key part is located in the sample image, and this positioning frame is regarded as the first label. In one embodiment, the position information / positioning frame can be represented by coordinates. Among them, the shape of the positioning frame can be rectangular, circular, triangular, etc., which is not limited here.

[0095] In one embodiment, the sample image carries at least a second label, and the second label is used to indicate the relative position of the associated part associated with the key part. Specifically, the second label is generated according to the relative position of the associated part relative to the key part in the sample image.

[0096] Here, the relative position refers to the distance between the center position of the key part in the sample image and the center position of the key part. Then, the marking of the second label may refer to framing the area where the associated part is located in the sample image with a positioning frame according to the relative position, and using the positioning frame as the second label. Of course, the relative position may also be directly used as the second label, which is not limited here.

[0097] In one embodiment, the sample image carries at least a third label, and the third label is used to indicate the action category of the action performed by the sample object.

[0098] For example, a photo of a cat performing a jumping action, the sample object is a cat, and the action performed is jumping, and the action category is jumping. Then, by labeling, for example, adding a text mark "jump" to the photo of the cat performing a jumping action, this "jump" mark is regarded as the third label, thereby forming a sample image carrying the third label. Of course, in other embodiments, the labeling of the third label can also be adding a mark in the form of numbers, letters, characters, graphics, colors, etc. to the sample image, which does not constitute a specific limitation here.

[0099] It can be seen that based on the acquired training set, not only the first label but also the second label is introduced, so that the key parts and related parts of the sample object performing the action are closely associated together through the combination of the first label and the second label, so as to avoid detecting objects that do not have related parts but are similar to the key parts as key parts, thereby effectively reducing the false detection rate of non-key parts.

[0100] Step 330, in the part detection stage, based on the first label and the second label carried by the sample image, supervised training is performed on the machine learning model regarding key part detection and associated part detection.

[0101] As mentioned above, action recognition includes two main tasks: part detection and action classification. Based on this, both the training process and the prediction process have two corresponding stages: part detection stage and action classification stage. The part detection stage is used to perform part detection in the action recognition task, and the action classification stage is used to perform action classification in the action recognition task, so as to fully ensure the accuracy of action recognition.

[0102] The inventors realized that, in the process of motion recognition, in low light or in the presence of some objects (such as hand toys) similar to key parts (such as hands), it is easy to misdetect key parts. For this reason, during the training process of the part detection stage, on the premise of supervised training of key part detection based on the first label, supervised training of associated part detection based on the second label is introduced to improve the accuracy of the part detection stage in the motion recognition process by assisting the key parts with the associated parts, which is beneficial to subsequently improve the accuracy of the motion classification stage in the motion recognition process.

[0103] Based on this, the training process of the part detection stage is described as follows:

[0104] Specifically, extracting sample image features from the sample image;

[0105] Predicting the position of a key part of the sample object performing an action in the sample image according to the sample image characteristics to obtain first prediction information, and predicting the relative position of the associated part relative to the key part in the sample image to obtain second prediction information;

[0106] Determining a first loss based on a difference between the first label and the first prediction information, and determining a second loss based on a difference between the second label and the second prediction information;

[0107] According to the first loss and the second loss, the parameters of the machine learning model corresponding to the part detection stage are adjusted.

[0108] In this way, the supervised training results of the part detection stage can be obtained, which is beneficial to the training of the subsequent action classification stage.

[0109] From the above, it can be seen that for the part detection stage, during the training process, on the premise that supervised training on key part detection has been carried out, by adding supervised training on associated part detection, the key parts and associated parts of the sample object performing the action are closely associated together, avoiding detecting objects that do not have associated parts but are similar to key parts as key parts, thereby effectively reducing the false detection rate of non-key parts, thereby fully ensuring the accuracy of part detection.

[0110] Step 350, in the action classification stage, based on the third label carried by the sample image and the supervised training results of the part detection stage, supervised training on action recognition is performed on the machine learning model.

[0111] Specifically, according to the first prediction information and the second prediction information in the supervised training result, predict the action category of the action performed by the sample object in the sample image to obtain third prediction information;

[0112] determining a third loss based on a difference between the third label and the third prediction information;

[0113] According to the third loss, and the first loss and the second loss in the supervised training result, the parameters of the machine learning model corresponding to the action classification stage are adjusted. Of course, in other embodiments, the parameters of the machine learning model corresponding to the action classification stage can also be adjusted based on the third loss alone.

[0114] Therefore, by introducing the supervised training results of this part detection stage to participate in the supervised training of action recognition, the action recognition process can also obtain more part features that are closer to the sample object, which is beneficial to improve the recognition accuracy of actions performed by key parts.

[0115] Step 370, when the training stop condition is met, a machine learning model that has completed training is obtained as an action recognition model.

[0116] That is to say, when the part detection stage completes training and the action classification stage completes training, which can also be understood as the part detection stage converges and the action classification stage converges, then the training stop condition is considered to be met. At this time, the initial machine learning model converges, that is, a machine learning model that has completed training is obtained.

[0117] Furthermore, in one embodiment, a model performance evaluation is performed on the machine learning model that has completed training, and a motion recognition model is obtained based on the model performance evaluation results. Specifically, the accuracy and recall rate corresponding to the machine learning model that has completed training are calculated. If both the accuracy and the recall rate meet the set conditions, the machine learning model that has completed the model performance evaluation converges to obtain the motion recognition model.

[0118] Based on this, the action recognition model has the ability to recognize the actions performed by the target object.

[0119] Through the above process, by fully considering the correlation between the key parts and the associated parts of the sample object performing the action, the false detection rate of non-key parts can be reduced, thereby avoiding the phenomenon of false triggering in the interactive scene, thereby effectively solving the problem of low accuracy of action recognition in the prior art.

[0120] like Figure 5 As shown, in one embodiment, during the part detection stage of the training process, the machine learning model can be divided into a key part detection branch and an associated part detection branch. That is, during the training process, the first prediction information is obtained by prediction through the key part detection branch in the machine learning model, and the second prediction information is obtained by prediction through the associated part detection branch in the machine learning model.

[0121] Now combined Figure 5 , the training process of the key part detection branch and the associated part detection branch is described in detail as follows:

[0122] The first step is to construct an initial machine learning model and determine the first parameter of the key part detection branch and the second parameter of the associated part detection branch in the machine learning model. The machine learning model includes but is not limited to a deep learning model.

[0123] The second step is to update the first parameter and the second parameter according to the first tag and the second tag. Specifically, the following steps are included:

[0124] First, a sample image feature is extracted from one of the sample images carrying at least the first label and the second label, and based on the sample image feature, the position of a key part of the sample object performing an action in one of the sample images is predicted to obtain first prediction information, and the relative position of the associated part with respect to the key part in one of the sample images is predicted to obtain second prediction information.

[0125] The first parameter is randomly initialized, and a first loss function corresponding to the machine learning model is determined based on the randomly initialized first parameter and the first prediction information. Similarly, the second parameter is randomly initialized, and a second loss function corresponding to the machine learning model is determined based on the randomly initialized second parameter and the second prediction information. The first loss function / second loss function includes but is not limited to: cosine loss function, cross entropy function, intra-class distribution function, inter-class distribution function, activation classification function, etc.

[0126] Secondly, the first loss of the first loss function is calculated, that is, the first loss is determined based on the difference between the first label and the first prediction information. Similarly, the second loss of the second loss function is calculated, that is, the second loss is determined based on the difference between the second label and the second prediction information.

[0127] Again, the first parameter and the second parameter are adjusted according to the first loss and the second loss.

[0128] In one embodiment, the first parameter is adjusted according to the first loss and the second loss, and the second parameter is adjusted according to the second loss until the training stop condition is met.

[0129] In one embodiment, a first joint loss is calculated based on the first loss and the second loss, for example, the first joint loss=first loss+second loss, and then the first parameter and the second parameter are adjusted based on the first joint loss until the training stop condition is met.

[0130] The following takes the first joint loss as an example to illustrate the training stop conditions.

[0131] If the first joint loss does not reach the minimum value, it is deemed that the part detection stage has not completed training, and the first parameter and the second parameter are adjusted according to the first joint loss, and the first loss function and the second loss function corresponding to the machine learning model are reconstructed according to the adjusted first parameter, the second parameter, and the next sample image carrying the first label and the second label to continue to determine whether the first joint loss obtained thereby reaches the minimum value.

[0132] On the contrary, if the first joint loss reaches the minimum value, the training stop condition is considered to be met.

[0133] Of course, the iteration threshold can also be set according to the actual needs of the application scenario in order to speed up the training efficiency of the part detection stage. At the same time, setting different iteration thresholds can also meet the different requirements of part detection for accuracy. For example, a larger iteration threshold is conducive to improving the accuracy of part detection.

[0134] Based on this, when the first joint loss reaches the minimum value or the number of iterations reaches the iteration threshold, the training stop condition is considered to be met, and the training of the part detection stage is stopped.

[0135] Under the effect of the above-mentioned embodiments, as far as the part detection stage is concerned, by adding associated part detection branches during the training process, the key parts and associated parts of the sample object performing the action are closely associated together, avoiding detecting objects that do not have associated parts but are similar to key parts as key parts, thereby effectively reducing the false detection rate of non-key parts, thereby fully ensuring the accuracy of the part detection stage.

[0136] In an exemplary embodiment, the training process of the action classification stage may include the following steps:

[0137] The first step is to determine the third parameter of the action classification stage in the machine learning model based on the initial machine learning model, wherein the machine learning model includes but is not limited to a deep learning model.

[0138] The second step is to update the third parameter according to the supervised training results of the part detection stage and the third label. Specifically, the following steps are included:

[0139] First, for one of the sample images carrying the third label, the supervised training results of the part detection stage are obtained, so as to predict the action category of the action performed by the sample object in one of the sample images based on the first prediction information and the second prediction information in the supervised training results, and obtain the third prediction information.

[0140] The third parameter is randomly initialized, and a third loss function corresponding to the machine learning model is determined based on the randomly initialized third parameter and the third prediction information. The third loss function includes but is not limited to: a cosine loss function, a cross entropy function, an intra-class distribution function, an inter-class distribution function, an activation classification function, and the like.

[0141] Secondly, the third loss of the third loss function is calculated, that is, the third loss is determined based on the difference between the third label and the third prediction information.

[0142] Again, according to the third loss, adjust the third parameter.

[0143] In one embodiment, the third parameter is adjusted according to the third loss until the training stop condition is met.

[0144] In one embodiment, based on the aforementioned supervised training results, a first loss and a second loss are obtained to calculate a second joint loss based on the first loss, the second loss and the third loss, for example, the second joint loss = first loss + second loss + third loss, and then the third parameter is adjusted according to this second joint loss until the training stop condition is met.

[0145] The following takes the second joint loss as an example to illustrate the training stop condition.

[0146] If the second joint loss does not reach the minimum value, it is deemed that the action classification stage has not completed training. The third parameter is adjusted according to the second joint loss, and the third loss function corresponding to the machine learning model is reconstructed according to the adjusted third parameter and the next sample image carrying the third label to continue to determine whether the second joint loss obtained thereby reaches the minimum value.

[0147] On the contrary, if the second joint loss reaches the minimum value, the training stop condition is considered to be met.

[0148] Of course, the iteration threshold can also be set according to the actual needs of the application scenario in order to speed up the training efficiency of the action classification stage. At the same time, setting different iteration thresholds can also meet the different requirements of action recognition for accuracy. For example, a larger iteration threshold is conducive to improving the accuracy of action recognition.

[0149] Based on this, when the second joint loss reaches the minimum value or the number of iterations reaches the iteration threshold, the training stop condition is considered to be met, and the training of the action classification stage is stopped.

[0150] It is worth mentioning that the action classification stage can set the same iteration threshold as the part detection stage, or can set a different iteration threshold, which is not limited here.

[0151] Please refer to Figure 5 In one embodiment, during the training process, the third loss is fed back from the action classification stage to the part detection stage. That is, during the training process, the parameters of the part detection stage can be adjusted according to the first loss, the second loss and the third loss. Specifically, the following steps may be included: according to the first loss, the second loss and the third loss, the first parameter of the key part detection branch is adjusted, and according to the second loss and the third loss, the second parameter of the associated part detection branch is adjusted. The update process of the first parameter and the second parameter is as described above and will not be repeated here.

[0152] In the above process, supervised training of the first parameter of the key part detection branch based on the first label, supervised training of the second parameter of the associated part detection branch based on the second label, and supervised training results of the part detection stage and the action classification stage based on the third label on the third parameter are realized, so that the action recognition process can not only obtain the key part features of the sample object performing the action, but also obtain the associated part features of the sample object performing the action, which is beneficial to improve the detection accuracy of the key parts and the recognition accuracy of the actions performed by them.

[0153] In an exemplary embodiment, the prediction process of the part detection stage, such as Figure 6 As shown, step 250 may include the following steps:

[0154] Step 251, based on the target image features, locate the key part of the target object performing the action in the area of ​​the image to be identified, and obtain the position of the key part of the target object performing the action.

[0155] In one embodiment, the position of the key part is related to a positioning frame, which indicates the position and / or size of the key part in the image to be identified. The shape of the positioning frame can be rectangular, circular, triangular, etc., which is not limited here, and the position of the positioning frame in the image to be identified can be represented by coordinates.

[0156] For example, in the image to be identified, the positioning box is the maximum circumscribed rectangle of the key part in the image to be identified, that is, the positioning box can be expressed as (xmin1, ymin1, xmax1, ymax1), thereby indicating that the position of the key part in the image to be identified is between xmin1 and xmax1 on the x-axis and between ymin1 and ymax1 on the y-axis, wherein (xmin1, ymax1) represents the upper left corner vertex of the positioning box, and (xmax1, ymin1) represents the lower right corner vertex of the positioning box.

[0157] Step 253, determining the key part detection result according to the position of the key part of the target object performing the action.

[0158] The key part detection result is used to indicate whether the key part for the target object to perform the action is detected in the image to be identified.

[0159] For example, assuming that the target object is a person and the key part is a hand, if the image to be identified contains a person's hand performing an action, the key part detection result is used to indicate that the person's hand performing an action is detected in the image to be identified. Alternatively, if the image to be identified contains a "hand toy" similar to a person's hand performing an action, the key part detection result is used to indicate that the person's hand performing an action is not detected in the image to be identified.

[0160] Step 255, determining the part detection result according to the key part detection result.

[0161] It can be seen that in this embodiment, the part detection result actually indicates whether the key part of the target object performing the action is detected in the image to be identified, that is, the part detection result = {key part detection result}.

[0162] Of course, in other embodiments, the part detection result can also be obtained by assisting the key part detection result with the associated part detection result, that is, the part detection result = {key part detection result & associated part detection result}, and this embodiment does not constitute a specific limitation on this. For example, for the image to be identified that contains a "hand toy" similar to a hand of a person performing an action, since there is no associated part, at this time, even if the key part detection result indicates that a hand of a person performing an action is detected in the image to be identified, and the associated part detection result indicates that there is no associated part in the image to be identified, ultimately, the part detection result indicates that a hand of a person performing an action is not detected in the image to be identified.

[0163] In one embodiment, the acquisition of the associated part detection result may include the following steps: determining the relative position of the associated part relative to the key part in the image to be identified according to the target image features; and determining the associated part detection result based on the relative position of the associated part relative to the key part. The specific process is similar to the acquisition of the second label described above, and will not be repeated here.

[0164] In an exemplary embodiment, the prediction process of the action classification stage, such as Figure 7 As shown, step 270 may include the following steps:

[0165] Step 271: Determine the area of ​​the target object's action-performing area in the image to be identified based on the part detection result.

[0166] Step 273 , based on the area of ​​the target object performing the action in the image to be identified, predict the action category of the action performed by the target object in the image to be identified, and obtain the action category of the action performed by the target object.

[0167] Among them, action category prediction can be achieved through a classifier (such as a softmax function) set in the action recognition model, that is, the probability that the action performed by the target object in the image to be recognized belongs to different action categories is calculated based on the classifier.

[0168] For example, assuming that the target object is a person, the action categories at least include raising the head, lowering the head, making a "V" gesture, jumping, and other actions.

[0169] Then, calculate the probabilities of the actions performed by the person in the image to be identified, such as looking up, looking down, making a "V" gesture, jumping, etc., which are P1, P2, P3, and P4 respectively. If P1 is the largest, it means that the action performed by the person is looking up; similarly, if P2 is the largest, it means that the action performed by the person is looking down; if P3 is the largest, it means that the action performed by the person is a "V" gesture; if P4 is the largest, it means that the action performed by the person is jumping.

[0170] Of course, in other embodiments, a special action category, namely, a non-critical part, can be set. Then, when the probability that the action performed by the person belongs to this action category is the largest, it actually means that the critical part of the person performing the action has not been detected in the image to be identified, thereby further reducing the phenomenon of false triggering caused by misidentification of actions in interactive scenes.

[0171] Step 275, determining the action recognition result according to the action category of the action performed by the target object.

[0172] Specifically, action recognition result = {action category}.

[0173] Of course, in other embodiments, a credibility for indicating the credibility of the action category and its corresponding set value are also provided. If the credibility is less than the set value, the action recognition model cannot output the action recognition result. Among them, the set value can be set according to the actual needs of the application scenario, so that the accuracy and recall rate of the action recognition model are balanced. For example, for application scenarios with high accuracy requirements, a relatively high set value is set; for application scenarios with high recall rate requirements, a relatively low set value is set. Through the cooperation of the above embodiments, the prediction process for the image to be recognized, that is, the key part detection branch of the part detection stage + the action classification stage is realized.

[0174] From the above, we can see that, unlike the introduction of the associated part detection branch in the training process, in the prediction process, the two main tasks of action recognition are achieved only through the key part detection branch. That is to say, the associated part detection branch is deleted in the prediction process. On the one hand, although the associated part detection branch is deleted in the prediction process, since the associated part detection branch has participated in the training process, the prediction results in the part detection stage already have more part features that are closer to the target object, which is conducive to more accurate key part detection. On the other hand, by deleting the associated part detection branch in the prediction process, it can not only save system memory overhead, but also effectively reduce the computational complexity and complexity of action recognition, which is conducive to improving action recognition efficiency.

[0175] In addition, in an exemplary embodiment, refer back to Figure 5 ,The training set includes not only positive samples, but also negative samples fed back during the prediction process.

[0176] Among them, the negative sample at least includes non-critical parts that are not related to the key parts of the target object performing the action, for example, the "hand toy" contained in the aforementioned image to be identified. It can also be understood that, compared with the key parts and related parts indicating the sample object performing the action, the negative sample is used to indicate the non-critical parts that are not related to the key parts of the sample object performing the action.

[0177] Combination Figure 2 If the part detection result indicates that a key part of the target object performing an action is detected in the image to be identified, step 270 is executed to enter the action classification stage from the part detection stage.

[0178] On the contrary, if the part detection result indicates that the key part of the target object performing the action is not detected in the image to be identified, then the image to be identified in which the key part of the target object performing the action is not detected is used as a negative sample and fed back to the training process of the action recognition model, that is, added to the training set, such as Figure 5 shown.

[0179] Then, in the training process of the action recognition model, the negative samples can be input into the initial machine learning model to participate in the training. In one embodiment, the non-critical parts contained in the negative samples are detected as non-critical parts in the part detection stage, that is, the negative samples participate in the training as "negative samples" in the part detection stage, and the training in the action classification stage may not be involved at this time. In one embodiment, the non-critical parts contained in the negative samples are detected as critical parts in the part detection stage, and are corrected as misidentification in the action classification stage, that is, the negative samples are treated as "positive samples" to participate in the training in the part detection stage, and are only used as "negative samples" to participate in the training in the action classification stage.

[0180] It is assumed here that negative samples are treated as "positive samples" in the part detection stage and participate in the training (the training process is as described above and will not be repeated here), and are only used as "negative samples" in the action classification stage. Therefore, the training process for action recognition can include the following steps:

[0181] The first step is to determine the third parameter of the action classification stage in the machine learning model based on the initial machine learning model, wherein the machine learning model includes but is not limited to a deep learning model.

[0182] The second step is to update the third parameter according to the fourth label. Here, the fourth label is a label carried by the negative sample, which is used to indicate the non-critical parts that are not related to the key parts of the sample object performing the action. This negative sample is a sample image that at least contains the non-critical parts that are not related to the key parts of the sample object performing the action. Specifically, the following steps are included:

[0183] First, based on at least one of the negative samples carrying the fourth label, an action category of the action performed by the sample object in the one of the negative samples is predicted to obtain fourth prediction information.

[0184] The third parameter is randomly initialized, and a fourth loss function corresponding to the machine learning model is determined based on the randomly initialized third parameter and the fourth prediction information. The fourth loss function includes but is not limited to: a cosine loss function, a cross entropy function, an intra-class distribution function, an inter-class distribution function, an activation classification function, and the like.

[0185] Secondly, the fourth loss of the fourth loss function is calculated, that is, the fourth loss is determined based on the difference between the fourth label and the fourth prediction information.

[0186] Again, according to the fourth loss, the third parameter is adjusted until the training stop condition is met. The specific process of meeting the training stop condition is the same as the updating process of the third parameter mentioned above, which will not be repeated here.

[0187] Through the above process, compared with the positive sample, each feature in the prediction process is made closer and more similar to the feature of the positive sample. The negative sample is introduced into the training process of the action recognition model, especially the training of the action classification stage, so that each feature in the prediction process is made farther and less similar to the feature of the negative sample, which is beneficial for the subsequent action recognition model to quickly exclude the images to be recognized that do not contain key parts, avoid the false triggering phenomenon caused by misidentification of actions in interactive scenes, and further help to improve the accuracy of action recognition.

[0188] The following is an embodiment of the device of the present application, which can be used to execute the training device of the action recognition model involved in the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the training method of the action recognition model involved in the present application.

[0189] See also Figure 8 In an exemplary embodiment, a motion recognition device 700 includes but is not limited to: an image acquisition module 710 , a feature extraction module 730 , a part detection module 750 and a motion recognition module 770 .

[0190] The image acquisition module 710 is used to acquire an image to be identified.

[0191] The feature extraction module 730 is used to extract target image features from the image to be identified. The target image features are associated with key parts of the target object performing actions and associated parts that are associated with the key parts.

[0192] The part detection module 750 is used to detect the part where the target object in the image to be identified performs an action according to the target image features, and obtain a part detection result.

[0193] The action recognition module 770 is used to recognize the action performed by the target object in the image to be recognized according to the part detection result to obtain the action recognition result.

[0194] It should be noted that the action recognition device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing action recognition. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the action recognition device will be divided into different functional modules to complete all or part of the functions described above.

[0195] In addition, the embodiments of the motion recognition device and method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0196] Fig. 9 A schematic diagram of the structure of a server is shown according to an exemplary embodiment. The server is suitable for Figure 1 A server 150 implementing the environment is shown.

[0197] It should be noted that the server is only an example adapted to the present application and cannot be considered to provide any limitation on the scope of use of the present application. The server cannot be interpreted as requiring dependence on or having Fig. 9 One or more components of exemplary server 2000 are shown.

[0198] The hardware structure of the server 2000 may vary greatly due to different configurations or performances, such as Fig. 9 As shown, the server 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0199] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the server 2000 .

[0200] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Figure 1 The interaction between the gateway 110 and the server 150 in an implementation environment is shown.

[0201] Of course, in other examples adapted by the present application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Fig. 9 As shown, this is not intended to be a specific limitation.

[0202] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0203] Among them, the operating system 251 is used to manage and control the hardware devices and application programs 253 on the server 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0204] The application program 253 is a computer program that performs at least one specific task based on the operating system 251, and may include at least one module ( Fig. 9 (not shown), each module may include a computer program for the server 2000. For example, the motion recognition device is regarded as an application 253 deployed on the server 2000.

[0205] The data 255 may be photos, pictures, or videos stored on a disk, or may be sample images, positive and negative samples, labels, etc., stored in the memory 250 .

[0206] The central processing unit 270 may include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus to read the computer program stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the training method of the action recognition model is completed by the central processing unit 270 reading a series of computer programs stored in the memory 250.

[0207] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0208] See also Fig.10 In an embodiment of the present application, an electronic device 4000 is provided, and the electronic device 4000 may include:

[0209] exist Fig.10 In the embodiment, the electronic device 4000 includes at least one processor 4001, at least one communication bus 4002 and at least one memory 4003.

[0210] The processor 4001 and the memory 4003 are connected, such as through a communication bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0211] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0212] The communication bus 4002 may include a path to transmit information between the above components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0213] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0214] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002 .

[0215] When the computer program is executed by the processor 4001, the action recognition method in the above-mentioned embodiments is implemented.

[0216] In addition, a storage medium is provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the action recognition method in the above-mentioned embodiments is implemented.

[0217] A computer program product is provided in an embodiment of the present application, the computer program product includes a computer program, the computer program is stored in a storage medium. A processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that the computer device executes the action recognition method in each of the above embodiments.

[0218] Compared with the prior art, the technical solution provided by this application has the following beneficial effects:

[0219] 1. Introduce associated parts that are associated with key parts to participate in the training of the action recognition model, which closely associates the key parts and associated parts, thereby reducing the false detection rate of non-key parts, improving the accuracy of the action recognition model, and thus reducing the false trigger rate caused by false action recognition in the interactive scene;

[0220] 2. Only the associated part detection branch is added in the training process, and the associated part detection branch is deleted in the prediction process. As far as the prediction process is concerned, the system resources consumed by the action recognition model remain unchanged, and no additional calculation amount is added. This is not only beneficial to saving system resources, but also beneficial to improving the efficiency of action recognition, fully ensuring the high-efficiency operation of the entire system;

[0221] 3. By using larger and easier-to-distinguish related parts to assist in the detection of key parts, it is helpful to accurately detect key parts in situations where key parts are difficult to detect, such as in low light or when the key parts are small at a distance, thereby improving the recall rate of the action recognition model and thus helping to enhance the user experience in interactive scenarios.

[0222] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0223] The above is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for motion recognition, characterized in that: The method comprises: Obtain an image to be recognized; Utilizing the part detection stage of the action recognition model, target image features are extracted from the image to be recognized, wherein the target image features are associated with key parts of the target object performing the action and associated parts associated with the key parts; the part detection stage is used to perform part detection in the action recognition task, and during the training process of the part detection stage, an associated part detection branch is introduced on the basis of the key part detection branch, and during the prediction process, the associated part detection branch is deleted while the key part detection branch is retained; According to the target image features, detecting the part where the target object in the image to be identified performs an action, and obtaining a part detection result; The action classification stage of the action recognition model is used to classify the action performed by the target object in the image to be recognized according to the part detection result, to obtain the action category of the action performed by the target object, and to use the action category of the action performed by the target object as the action recognition result; the action classification stage is used to perform action classification in the action recognition task.

2. The method according to claim 1, characterized in that The detecting, based on the target image features, the part where the target object performs an action in the image to be identified to obtain a part detection result includes: Based on the target image features, the key part of the target object performing the action is located in the area of ​​the image to be identified to obtain the position of the key part of the target object performing the action; Determining a key part detection result according to the position of the key part of the target object performing the action; The part detection result is determined according to the key part detection result.

3. The method according to claim 1, characterized in that The detecting, based on the target image features, the part where the target object performs an action in the image to be identified to obtain a part detection result includes: According to the target image features, a key part of the target object in the image to be identified is detected to obtain a key part detection result; According to the target image features, detecting associated parts in the image to be identified that have an associated relationship with the key parts to obtain associated part detection results; The part detection result is determined according to the key part detection result and the associated part detection result.

4. The method according to claim 3, characterized in that The detecting, based on the target image features, the associated parts in the image to be identified that are associated with the key parts, to obtain the associated parts detection results includes: Determining the relative position of the associated part with respect to the key part in the image to be identified according to the target image feature; The associated part detection result is determined based on the relative position of the associated part with respect to the key part.

5. The method according to claim 1, characterized in that The step of classifying the action performed by the target object in the image to be identified according to the part detection result to obtain the action category of the action performed by the target object, and taking the action category of the action performed by the target object as the action recognition result includes: Determine, according to the part detection result, the area of ​​the part where the target object performs the action in the image to be identified; Based on the area of ​​the part where the target object performs the action in the image to be identified, predicting the action category of the action performed by the target object in the image to be identified, and obtaining the action category of the action performed by the target object; The action recognition result is determined according to the action category of the action performed by the target object.

6. The method according to any one of claims 1 to 5, characterized in that: The action recognition model is obtained by training an initial machine learning model based on at least one sample image; the training at least includes: In the part detection stage, the machine learning model is supervisedly trained on key part detection and associated part detection based on the first label and the second label carried by the sample image, the first label is used to indicate the position of the key part of the sample object performing an action, and the second label is used to indicate the relative position of the associated part with respect to the key part; the sample object refers to various objects performing actions in the sample image.

7. The method according to claim 6, characterized in that In the part detection stage, based on the first label and the second label carried by the sample image, supervised training is performed on the machine learning model regarding key part detection and associated part detection, including: Extracting sample image features from the sample image; Predicting the position of a key part of the sample object performing an action in the sample image according to the sample image features to obtain first prediction information, and predicting the relative position of the associated part in the sample image relative to the key part to obtain second prediction information; determining a first loss based on a difference between the first label and the first prediction information, and determining a second loss based on a difference between the second label and the second prediction information; According to the first loss and the second loss, the parameters of the machine learning model corresponding to the part detection stage are adjusted.

8. The method according to claim 7, characterized in that The training also includes: In the action classification stage, based on the third label carried by the sample image and the supervised training result of the part detection stage, the machine learning model is supervisedly trained on action recognition, wherein the third label is used to indicate the action category of the action performed by the sample object; In the action classification stage, based on the third label carried by the sample image and the supervised training result of the part detection stage, the machine learning model is subjected to supervised training on action recognition, including: Predicting an action category of an action performed by the sample object in the sample image according to the first prediction information and the second prediction information in the supervised training result to obtain third prediction information; determining a third loss based on a difference between the third label and the third prediction information; According to the third loss, and / or the first loss and the second loss in the supervised training result, adjusting the parameters of the machine learning model corresponding to the action classification stage; When the training stop condition is met, a machine learning model that has completed training is obtained as the action recognition model.

9. The method according to claim 8, characterized in that The method further comprises: feeding back the third loss to the part detection stage; The step of adjusting the parameters of the machine learning model corresponding to the part detection stage according to the first loss and the second loss includes: According to the first loss, the second loss and the third loss, the parameters of the machine learning model corresponding to the part detection stage are adjusted.

10. The method according to claim 9, characterized in that The first prediction information is obtained by predicting a key part detection branch in the machine learning model, and the second prediction information is obtained by predicting an associated part detection branch in the machine learning model; The step of adjusting the parameters of the machine learning model corresponding to the part detection stage according to the first loss, the second loss, and the third loss includes: According to the first loss, the second loss and the third loss, adjusting the parameters of the key part detection branch; and According to the second loss and the third loss, the parameters of the associated part detection branch are adjusted.

11. A motion recognition device, characterized in that: include: An image acquisition module, used for acquiring an image to be identified; A feature extraction module is used to extract target image features from the image to be recognized by utilizing the part detection phase of the action recognition model, wherein the target image features are associated with key parts of the target object performing the action and associated parts associated with the key parts; the part detection phase is used to perform part detection in the action recognition task, and during the training process of the part detection phase, an associated part detection branch is introduced on the basis of the key part detection branch, and during the prediction process, the associated part detection branch is deleted while the key part detection branch is retained; A part detection module, used to detect the part where the target object in the image to be identified performs an action according to the target image features, and obtain a part detection result; The action recognition module is used to utilize the action classification stage of the action recognition model to classify the action performed by the target object in the image to be recognized according to the part detection result, obtain the action category of the action performed by the target object, and use the action category of the action performed by the target object as the action recognition result; the action classification stage is used to perform action classification in the action recognition task.

12. An electronic device, characterized in that: include: processor; and A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the action recognition method according to any one of claims 1 to 10.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the action recognition method according to any one of claims 1 to 10 is implemented.

Images