Action recognition method, device control method, apparatus, electronic device, and medium

Abstract

The application provides a motion recognition method, a device control method, a device, an electronic device and a storage medium, and relates to the technical field of Internet of Things. The method comprises the following steps: acquiring a to-be-detected image; inputting the to-be-detected image into a motion recognition model, performing motion recognition on the to-be-detected image through a main classification branch and a secondary classification branch in the motion recognition model respectively, and obtaining a first classification result and a second classification result; the motion recognition model is obtained by adding the secondary classification branch on the basis of the main classification branch and training; and a motion recognition result is obtained according to the first classification result corresponding to the main classification branch and the second classification result corresponding to the secondary classification branch. The application solves the problem of device misoperation caused by motion misrecognition in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things, in particular, the present application relates to a kind of action recognition method, equipment control method, device, electronic device and storage medium. BACKGROUND

[0002] With the continuous progress of science and technology, people's understanding of artificial intelligence is also more and more profound, and computer vision technology, as a new artificial intelligence technology, plays a pivotal role in its development process. Smart home, as a new type of home device, controls the technology products in the home through intelligent way, provides great convenience for people, and greatly improves the quality of people's life. For example, through different actions, such as gesture action, body action, limb action, etc., the smart home can be controlled to play music, turn on and off the lamp, and turn on and off the air conditioner, etc. Automatic operation.

[0003] However, the existing action recognition method is prone to misrecognition of similar actions when recognizing actions, resulting in generation of incorrect device control instructions and causing poor user experience.

[0004] As can be seen from the above, how to reduce the action misrecognition rate and avoid device misoperation remains to be solved. SUMMARY

[0005] The embodiments of the present application provide an action recognition method, a device control method, a device, an electronic device and a storage medium, which can solve the problem of device misoperation caused by action misrecognition in related art. The technical solution is as follows:

[0006] According to one aspect of the present application, an action recognition method comprises: acquiring a to-be-detected image; inputting the to-be-detected image into an action recognition model, and performing action recognition on the to-be-detected image through a primary classification branch and a secondary classification branch in the action recognition model to obtain a first classification result and a second classification result; the action recognition model is obtained by adding the secondary classification branch to the primary classification branch and training; and an action recognition result is obtained according to the first classification result corresponding to the primary classification branch and the second classification result corresponding to the secondary classification branch.

[0007] According to one aspect of the present application, an action recognition device comprises: an image acquisition module configured to acquire a to-be-detected image; an action recognition module configured to input the to-be-detected image into an action recognition model, and perform action recognition on the to-be-detected image through a primary classification branch and a secondary classification branch in the action recognition model to obtain a first classification result and a second classification result; and a result output module configured to obtain an action recognition result according to the first classification result corresponding to the primary classification branch and the second classification result corresponding to the secondary classification branch.

[0008] In an example embodiment, the action recognition model comprises a feature extraction layer, a main classification branch and a secondary classification branch connected with the feature extraction layer respectively; the device further comprises a model training module for training of the action recognition model; the model training module comprises: a data set acquisition unit for acquiring a first data set and a second data set; the first data set comprises sample images labeled with a first class label; the second data set comprises sample images labeled with a second class label; a first training unit for training the feature extraction layer and the main classification branch of the action recognition model based on each sample image labeled with the first class label in the first data set; a second training unit for training the secondary classification branch of the action recognition model based on each sample image labeled with the second class label in the second data set and the trained feature extraction layer; an action recognition unit for obtaining the trained action recognition model when the main classification branch and the secondary classification branch are trained.

[0009] In an example embodiment, the model training module is further configured to: acquire a current sample image from the first data set and input the current sample image into the feature extraction layer for feature extraction to obtain image features of the current sample image; input the image features of the current sample image into the main classification branch of the action recognition model for training to obtain a first training result; the first training result represents a predicted action class; calculate a first loss value based on a difference between the first training result and a first class label of the current sample image; the first class label is used to indicate whether the labeled real action class belongs to a set action class or a negative sample class; update parameters of the feature extraction layer and the main classification branch based on the first loss value, and continue to train the feature extraction layer and the main classification branch using other sample images in the first data set until a training stop condition is met to stop training and obtain the trained main classification branch.

[0010] In an example embodiment, the model training module is further configured to: perform feature extraction on a current sample image input from the second data set by using the feature extraction layer to obtain image features of the current sample image; input the image features of the current sample image into the secondary classification branch of the action recognition model to obtain a second training result, the second training result representing a predicted action category; calculate a second loss value based on a difference between the second training result and a second category label of the current sample image, the second category label indicating whether a labeled real action category belongs to a positive sample category or a negative sample category; update parameters of the secondary classification branch based on the second loss value, and continue to train the secondary classification branch by using other sample images in the second data set until a training stop condition is met, and then stop the training to obtain the trained secondary classification branch.

[0011] In an example embodiment, the action recognition module is further configured to: perform feature extraction on the to-be-detected image by using the feature extraction layer to obtain image features of the to-be-detected image; perform action classification prediction on the image features by using the primary classification branch of the action recognition model to obtain the first classification result; and perform action classification prediction on the image features by using the secondary classification branch of the action recognition model to obtain the second classification result.

[0012] In an example embodiment, the result output module is further configured to: if the action category indicated by the first classification result is a set action category, determine whether the action category indicated by the second classification result is a positive sample category; the first classification result is used to indicate whether the action category is a set action category or a negative sample category; the second classification result is used to indicate whether the action category is a positive sample category or a negative sample category; if yes, the first classification result is taken as the action recognition result; if no, the second classification result is taken as the action recognition result; and if the action category indicated by the first classification result is a negative sample category, the first classification result is taken as the action recognition result.

[0013] In an example embodiment, the apparatus further includes a device control module configured to control an intelligent device to perform an automatic operation corresponding to an action category to which an action in the to-be-detected image belongs according to the action recognition result.

[0014] In an example embodiment, the device control module comprises: an instruction generation unit configured to generate a device control instruction corresponding to the action category if the action recognition result indicates that the action category to which the action in the image to be detected belongs is a set action category; and an instruction sending unit configured to send the device control instruction to the smart device, so that the smart device performs a corresponding automatic operation in response to the device control instruction.

[0015] In an example embodiment, the apparatus further comprises: a target detection module configured to perform target detection on the image to be detected to obtain a position of a target object in the image to be detected; extract an image region in which the target object is located from the image to be detected according to the position of the target object to obtain a target image, so that the action recognition is performed based on the target image.

[0016] According to an aspect of the present application, a device control method comprises: determining a trigger condition and associated controlled data in response to an operation triggered in an automatic configuration page; the trigger condition is related to an action category, and the controlled data is used to instruct a smart device to perform an automatic operation corresponding to the action category; generating an automatic scheme corresponding to the action category according to the trigger condition and the associated controlled data; the automatic scheme is used to realize automatic control of the device by recognizing an action category in an image to be detected; the recognition of the action category is realized through a primary classification branch and a secondary classification branch in an action recognition model; the action recognition model is obtained by adding the secondary classification branch on the basis of the primary classification branch and training.

[0017] According to an aspect of the present application, a device control apparatus comprises: a parameter determination module configured to determine a trigger condition and associated controlled data in response to an operation triggered in an automatic configuration page; the trigger condition is related to an action category, and the controlled data is used to instruct a smart device to perform an automatic operation corresponding to the action category; an automatic scheme generation module configured to generate an automatic scheme corresponding to the action category according to the trigger condition and the associated controlled data; the automatic scheme is used to realize automatic control of the device by recognizing an action category in an image to be detected; the recognition of the action category is realized through a primary classification branch and a secondary classification branch in an action recognition model; the action recognition model is obtained by adding the secondary classification branch on the basis of the primary classification branch and training.

[0018] According to an aspect of the present application, an electronic device comprises at least one processor and at least one memory, wherein the memory has computer readable instructions stored thereon; the computer readable instructions are loaded and executed by the processor, so that the electronic device implements the action recognition method as described above.

[0019] According to one aspect of this application, a storage medium stores computer-readable instructions thereon, which are loaded and executed by a processor to implement the action recognition method as described above.

[0020] According to one aspect of this application, an application product includes computer-readable instructions stored in a storage medium, wherein a processor of an electronic device reads the computer-readable instructions from the storage medium, loads and executes the computer-readable instructions, thereby enabling the electronic device to implement the action recognition method as described above.

[0021] The beneficial effects of the technical solution provided in this application are:

[0022] In the above technical solution, an image to be detected is acquired; the image is input into an action recognition model, and action recognition is performed on the image using a main classification branch and a sub-classification branch, respectively, to obtain a first classification result and a second classification result; based on the first classification result corresponding to the main classification branch and the second classification result corresponding to the sub-classification branch, an action recognition result is obtained; based on the action category indicated by the action recognition result, the device is controlled to perform an automated operation corresponding to the action category. Compared with the prior art, the action recognition model of this application adds a sub-classification branch to the main classification branch to distinguish similar actions. This allows the second classification result to assist the first classification result in more accurate action category differentiation, avoiding misidentification of similar actions and effectively solving the problem of device malfunction caused by action misidentification in related technologies. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0024] Figure 1 This is a schematic diagram based on the implementation environment involved in this application;

[0025] Figure 2 This is a flowchart illustrating an action recognition method according to an exemplary embodiment;

[0026] Figure 3 This is a flowchart illustrating a preprocessing method according to an exemplary embodiment;

[0027] Figure 4 This is illustrated according to an exemplary embodiment. Figure 2 Flowchart for step 330;

[0028] Figure 5 This is illustrated according to an exemplary embodiment. Figure 2 Flowchart for step 350;

[0029] Figure 6 is a flowchart of another action recognition method according to an example embodiment;

[0030] Figure 7 is a flowchart of a training process of an action recognition model according to an example embodiment; Figure 6 is a flowchart of step 370;

[0031] Figure 8 is a flowchart of a training process of an action recognition model according to an example embodiment;

[0032] Figure 9 is a flowchart of a training process of a main classification branch according to an example embodiment;

[0033] Figure 10 is a flowchart of a training process of a sub-classification branch according to an example embodiment;

[0034] Figure 11 is a flowchart of an action recognition method according to an example embodiment;

[0035] Figures 12a to 12c is a specific implementation schematic diagram of an action recognition method in an application scenario;

[0036] Figure 13 is a structural block diagram of an action recognition apparatus according to an example embodiment;

[0037] Figure 14 is a hardware structural diagram of an electronic device according to an example embodiment;

[0038] Figure 15 is a hardware structural diagram of a terminal according to an example embodiment;

[0039] Figure 16 is a structural block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0040] Embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, in which the same or similar notations are used to denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are examples only, and are used to explain the present application, and cannot be interpreted as limiting the present application.

[0041] Those skilled in the art can understand that the singular forms "a," "an," and "the" used herein include plural references unless expressly stated to the contrary. It should be further understood that the use of the term "include" in the specification of the application means that the features, integers, steps, operations, elements, and / or components listed exist, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any one of the associated listed items and all combinations thereof.

[0042] As described above, the existing action recognition method has low recognition accuracy for similar actions.

[0043] Taking a gesture action as an example, in the prior art, when performing gesture recognition, a recognized gesture is directly divided as a gesture recognition result, which is easy to cause misrecognition of similar gestures. For example, the gesture "OK" is easy to be misrecognized as the gesture "three".

[0044] In order to improve the quality of life, smart devices can be controlled by different gesture actions, for example, music is played by the gesture "OK", and the air conditioner is turned on by the gesture "three". If the gesture "OK" is misrecognized as the similar gesture "three", an incorrect device control instruction for controlling the air conditioner to be turned on is generated, which causes the air conditioner to be misturned on, not only affecting the user experience, but also causing unnecessary waste of energy consumption for the user.

[0045] As can be seen from the above, the related art still has the problem of action misrecognition leading to device misoperation.

[0046] Therefore, the device control method provided by the present application can effectively reduce the misrecognition rate of action recognition, thereby improving the accuracy and success rate of device control and improving the user experience. Correspondingly, the device control method is suitable for a device control apparatus. The device control apparatus can be deployed in an electronic device, which can be a smart device, for example, the smart device includes a gateway type camera; the electronic device can also be a user terminal, for example, the user terminal includes a smart phone; the electronic device can also be a computer device configured with a von Neumann architecture, for example, the computer device can be a desktop computer, a notebook computer, a server, etc.

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0048] Figure 1 A schematic diagram of an implementation environment involved in a device control method. The implementation environment at least includes a user terminal 110, a smart device 130, a server end 170, and a network device, wherein Figure 1 The network device includes a gateway 150 and a router 190, which are not specifically limited herein.

[0049] The user terminal 110, which can also be referred to as a user end or a terminal, can be an electronic device such as a smartphone, a tablet computer, a smart control panel, a device with display and control functions, etc., which can be deployed (or installed) as a client associated with the smart device 130. The user terminal 110 is not specifically limited herein.

[0050] The client is associated with the smart device 130. In other words, the user registers an account in the client and configures the smart device 130 in the client. For example, the configuration includes adding an automatic control instruction to the smart device 130, so that the smart device performs a configured automatic operation when detecting a corresponding gesture. The client can be in the form of an application program or a web page. Correspondingly, the interface displayed by the client can be in the form of a program window or a web page. The client is not specifically limited herein.

[0051] The smart device 130 is deployed in the gateway 150 and communicates with the gateway 150 through a communication module configured by itself, and is controlled by the gateway 150. It should be understood that the smart device 130 generally refers to one of a plurality of smart devices, and the embodiments of the present application are only exemplified by the smart device 130, that is, the number and type of smart devices deployed in the gateway 150 are not limited by the embodiments of the present application. In one application scenario, the smart device 130 accesses the gateway 150 through a local area network, and is thus deployed in the gateway 150. The process of the smart device 130 accessing the gateway 150 through a local area network includes: first establishing a local area network by the gateway 150, and the smart device 130 connects the gateway 150 to join the local area network established by the gateway 150. The local area network includes but is not limited to ZIGBEE or Bluetooth. The smart device 130 can be a smart printer, a smart fax machine, a smart camera, a smart air conditioner, a smart door lock, a smart lamp, or a human body sensor, a door and window sensor, a temperature and humidity sensor, a water immersion sensor, a natural gas alarm, a smoke alarm, a wall switch, a wall socket, a wireless switch, a wireless wall switch, a magic cube controller, a curtain motor, a millimeter wave radar, etc. configured with a communication module. It is explained here that the millimeter wave radar can emit an FMCW signal to other smart devices 130, and then receive a reflected echo signal reflected by other smart devices 130. Since the echo signal contains all distance segments, moving state objects within the detection range of the millimeter wave radar can be detected. Based on the characteristics of the millimeter wave radar, other smart devices can be effectively monitored. The micro-motion signal refers to the vibration wave generated by other smart devices 130 due to work, such as a smart air conditioner, a fan, a sweeper, an air purifier, and a computer case with a fan rotating, which can generate a micro-motion signal. It is worth mentioning that the smart camera itself can also act as a gateway, that is, the smart camera is a gateway type, which not only has a gateway function, but also has an image acquisition function. At this time, other smart devices 130 can be considered to be deployed in the gateway-type camera, and communicate with the gateway-type camera through a communication module configured by other smart devices 130, and thus realize interaction with the user terminal 110 and the server 170 through the gateway-type camera.

[0052] The interaction between the user terminal 110 and the intelligent device 130 can be implemented through a local area network, and can also be implemented through a wide area network. In one application scenario, the user terminal 110 establishes a communication connection in a wired or wireless manner between the router 190 and the gateway 150, for example, the wired or wireless manner includes but is not limited to WIFI and the like, so that the user terminal 110 and the gateway 150 are deployed in the same local area network, and then the user terminal 110 can implement the interaction with the intelligent device 130 through a local area network path. In another application scenario, the user terminal 110 establishes a communication connection in a wired or wireless manner between the server end 170 and the gateway 150, for example, the wired or wireless manner includes but is not limited to 2G, 3G, 4G, 5G, WIFI and the like, so that the user terminal 110 and the gateway 150 are deployed in the same wide area network, and then the user terminal 110 can implement the interaction with the intelligent device 130 through a wide area network path.

[0053] The server end 170 can also be referred to as a cloud, a cloud platform, a platform end, a service end, and the like. The service end 170 can be a server, a server cluster composed of multiple servers, or a cloud computing center composed of multiple servers, so as to better provide background services to a large number of user terminals 110. For example, the background services include device control services.

[0054] Based on the pre-configured automatic scheme corresponding to the action category, if the user expects to control the intelligent device 130 to perform the automatic operation corresponding to the action category, the action can be made.

[0055] For the gateway type camera deployed with the action recognition model, the process of making the action for the user is photographed, so as to collect the to-be-detected image, and the action recognition model with the secondary classification branch added on the basis of the primary classification branch is used to recognize the action of the to-be-detected image, to obtain the first classification result corresponding to the primary classification branch and the second classification result corresponding to the secondary classification branch. After obtaining the action recognition result according to the first classification result and the second classification result, the intelligent device 130 can be controlled to perform the automatic operation corresponding to the action category of the action in the to-be-detected image indicated by the action recognition result.

[0056] Referring to Figure 2 The embodiment of the present application provides a device control method, which is suitable for an electronic device, which can be Figure 1 The intelligent device in the implementation environment shown in the figure, for example, the intelligent device includes a gateway type camera.

[0057] In the following method embodiment, in order to facilitate description, the execution subject of each step of the method is taken as an example for description, but this does not constitute a specific limitation.

[0058] As shown in Figure 2 , the method can include the following steps:

[0059] Step 310, obtaining a to-be-detected image.

[0060] The to-be-detected image is obtained by photographing and collecting the action execution process. Therefore, the to-be-detected image is an image for describing the object performing the action. The object refers to an object that can perform an action, for example, the object can be a person, a robot, etc. The action includes but is not limited to a gesture action, a body action, a limb action, etc.

[0061] It can be understood that the photographing can be single photographing, and can also be continuous photographing. For continuous photographing, a video can be obtained, and the to-be-detected image can be any frame of the video. For multiple photographing, multiple photos can be obtained, and the to-be-detected image can be any one of the multiple photos. In other words, the to-be-detected image in the embodiment can come from a dynamic image, for example, any frame of a video, and can also come from a static image, for example, any one of multiple photos. It can also be understood that the action recognition in the embodiment is performed in units of frames.

[0062] On the one hand, the to-be-detected image can not contain the object performing the action, and on the other hand, the to-be-detected image usually also contains a background image region irrelevant to the image region where the action is located, resulting in a large amount of calculation in action recognition, and the background image region often affects the accuracy of action recognition. Based on this, in a possible implementation, the electronic device obtains the to-be-detected image, and pre-processes the to-be-detected image to obtain a pre-processed to-be-detected image for action recognition.

[0063] Specifically, as shown in Figure 3 , the pre-processing of the to-be-detected image can include the following steps:

[0064] Step 311, performing target detection on the to-be-detected image to obtain the position of the target object in the to-be-detected image.

[0065] In the embodiment, the target object refers to the object performing the action contained in the to-be-detected image, and specifically refers to the object to be detected in the to-be-detected image. For example, the object can be a person, a robot, etc.

[0066] In a possible implementation, the target detection on the to-be-detected image is implemented through a target detection model. The target detection model is a machine learning model that is trained and has the capability of target detection on target objects in the to-be-detected image. The machine learning model includes but is not limited to an SSD network (Single Shot MultiBox Detector), an R-CNN network (Region-Convolutional Neural Networks), a Fast R-CNN network (Fast Region-Convolutional Network), a Faster R-CNN (Faster Region-Convolutional Network), and the like, which are not limited in the present application.

[0067] In step 313, an image region where the target object is located is extracted from the to-be-detected image according to the position of the target object, to obtain a target image, so that the action recognition is performed based on the target image.

[0068] In order to sufficiently guarantee the calculation amount and accuracy in the action recognition process, in a possible implementation, the target image is preprocessed, and the preprocessing includes but is not limited to size adjustment, normalization processing, and the like.

[0069] Then, according to the position of the target object, the target image is segmented from the to-be-detected image according to the position of the target object, the target image is adjusted in size to a set size and normalized, to obtain a preprocessed target image, so that the electronic device performs action recognition on the preprocessed target image, to obtain an action recognition result.

[0070] In this way, the to-be-detected image is preprocessed to obtain a target image containing only the target object, so that the electronic device performs action recognition on the target image, reduces the calculation amount in the subsequent action recognition process, is more friendly to low-power devices, and is conducive to improving the accuracy of the subsequent action recognition.

[0071] In step 330, the to-be-detected image is input to the action recognition model, and the to-be-detected image is recognized through the main classification branch and the secondary classification branch in the action recognition model, to obtain a first classification result and a second classification result.

[0072] The first classification result is used to indicate whether the action category belongs to a set action category or a negative sample category. That is, the action category indicated by the first classification result includes the set action category and the negative sample category. For example, taking a hand action as an example, the set action category includes “one”, “two”, “three”, and the like, and the negative sample category includes “other” and the like.

[0073] The second classification result is different from the first classification result, and is used to indicate whether the action category belongs to a positive sample category or a negative sample category. That is, the action category indicated by the second classification result includes the positive sample category and the negative sample category. For example, “one”, “two”, “three”, and the like belong to the positive sample category, and “other” and the like belong to the negative sample category. That is, the difference between the first classification result and the second classification result is that the set action category in the first classification result is considered as the positive sample category in the second classification result. For example, the action categories “one” and “two” are considered as different set action categories in the first classification result, and are considered as the same positive sample category in the second classification result.

[0074] In a possible implementation, the above-mentioned action recognition model is obtained by adding a secondary classification branch to the main classification branch and training. Specifically, the action recognition model includes a feature extraction layer for feature extraction, and a main classification branch and a secondary classification branch connected with the feature extraction layer respectively. The main classification branch is used to identify whether a specific action category in the to-be-detected image belongs to a specific action category in the set action category or a negative sample category, and the secondary classification branch is used to identify whether the action category in the to-be-detected image belongs to a positive sample category or a negative sample category.

[0075] In a possible implementation, as shown in Figure 4 The step 330 can include the following steps:

[0076] In step 331, the feature extraction layer is used to perform feature extraction on the to-be-detected image, to obtain image features of the to-be-detected image.

[0077] The image features are used to uniquely identify the to-be-detected image, and can also be considered as accurately describing the to-be-detected image in a digital manner. It should be understood that the to-be-detected image is different, and the corresponding image features will also be different.

[0078] In step 333, the main classification branch of the action recognition model is used to perform action classification prediction on the image features, to obtain a first classification result.

[0079] The action classification prediction refers to calculating probabilities of the action in the to-be-detected image belonging to different action categories based on the image features. The action categories include the set action categories and the negative sample category.

[0080] For example, the set action categories can include 11 action categories of “one”, “two”, “three”, “four”, “five”, “eight”, “ten”, “love”, “great”, “ok”, and “not great”, and the negative sample category can include one action category of “other”.

[0081] At step 335, the action classification prediction is performed on the image features by the sub-classification branch of the action recognition model, to obtain a second classification result.

[0082] Similarly to the main classification branch, the action classification prediction refers to calculating probabilities of the action in the to-be-detected image belonging to different action categories based on the image features. The action categories include the positive sample category and the negative sample category.

[0083] Still taking the foregoing example as an illustration, the 11 action categories of “one”, “two”, “three”, “four”, “five”, “eight”, “ten”, “love”, “great”, “ok”, and “not great” are regarded as the positive sample category, and the one action category of “other” is regarded as the negative sample category.

[0084] At step 350, the action recognition result is obtained according to the first classification result corresponding to the main classification branch and the second classification result corresponding to the sub-classification branch.

[0085] The action recognition result is used to indicate the action category to which the action in the to-be-detected image belongs.

[0086] In a possible implementation, as shown in Figure 5 The step 350 can include the following steps.

[0087] At step 351, if the action category indicated by the first classification result is the set action category, it is determined whether the action category indicated by the second classification result is the positive sample category.

[0088] The first classification result is used to indicate whether the action category is the set action category or the negative sample category, and the second classification result is used to indicate whether the action category is the positive sample category or the negative sample category.

[0089] If the action category indicated by the second classification result is the positive sample category, step 353 is performed, otherwise, if the action category indicated by the second classification result is the negative sample category, step 355 is performed.

[0090] Step 353, taking the first classification result as the action recognition result.

[0091] That is, taking the set action category indicated by the first classification result as the action recognition result.

[0092] Step 355, taking the second classification result as the action recognition result.

[0093] That is, taking the negative sample category indicated by the second classification result as the action recognition result.

[0094] Step 357, if the action category indicated by the first classification result is the negative sample category, taking the first classification result as the action recognition result.

[0095] That is, taking the negative sample category indicated by the first classification result as the action recognition result.

[0096] For example, if the action category indicated by the first classification result is the set action category “one”, and the action category indicated by the second classification result is that the set action category “one” belongs to the positive sample category “1”, taking the first classification result “one” as the action recognition result. Here, “1” means that the action in the image to be detected belongs to the positive sample category.

[0097] If the action category indicated by the first classification result is the set action category “one”, and the action category indicated by the second classification result is that the set action category “one” belongs to the negative sample category “0”, taking the second classification result “0” as the action recognition result. Here, “0” means that the action in the image to be detected belongs to the negative sample category.

[0098] If the action category indicated by the first classification result is the negative sample category “other”, taking the first classification result “other” as the action recognition result, which means that the action in the image to be detected belongs to the negative sample category.

[0099] In an exemplary embodiment, as shown in FIG. 3B, after step 350, the method can further include the following steps: Figure 6

[0100] Step 370, according to the action category to which the action in the image to be detected belongs as indicated by the action recognition result, controlling the intelligent device to perform an automatic operation corresponding to the action category.

[0101] Specifically, as shown in FIG. 3B, step 370 can include the following steps: Figure 7

[0102] Step 371, if the action category to which the action in the image to be detected belongs as indicated by the action recognition result is a set action category, generating a device control instruction corresponding to the action category. ​​

[0103] Step 373, the sending device controls the instruction to the smart device, so that the smart device executes the corresponding automation operation in response to the device control instruction.

[0104] That is, if the action recognition result indicates that the action category to which the action in the to-be-detected image belongs is the negative sample category, the corresponding device control instruction will not be generated, and only in the case where the action recognition result indicates that the action category to which the action in the to-be-detected image belongs is the set action category, the corresponding device control instruction is generated, thereby reducing the probability of device false triggering.

[0105] For example, when the action recognition result indicates that the action category to which the action in the to-be-detected image belongs is the set action category "two", the smart sound box is controlled to play music; when the action recognition result indicates that the action category to which the action in the to-be-detected image belongs is the set action category "four", the smart lamp is controlled to turn on.

[0106] Through the above process, the secondary classification branch is added on the basis of the primary classification branch of the action recognition model, which is used to distinguish similar actions. Therefore, the first classification result is assisted by the second classification result to make more accurate action category distinction, so as to avoid similar actions from being misrecognized, thereby effectively solving the problem of high action recognition misrecognition rate in related technologies, and further improving the accuracy of device control and the user experience.

[0107] Please refer to Figure 8 In an exemplary embodiment, the training process of the above-mentioned action recognition model can include the following steps:

[0108] Step 410, obtaining a first data set and a second data set.

[0109] The first data set includes sample images labeled with a first class label; the second data set includes sample images labeled with a second class label. The first class label is used to indicate whether the labeled real action category belongs to the set action category or the negative sample category, and the second class label is used to indicate whether the labeled real action category belongs to the positive sample category or the negative sample category.

[0110] Specifically, the first data set is used to train the main classification branch of the action recognition model, and the first class label of the sample image in the first data set is used to represent the real specific action category corresponding to the sample image, for example, whether it belongs to a specific action category in the set action category or belongs to the negative sample category. For example, the first class label includes labels such as "one", "two", "three", and "other". Among them, "one", "two", "three" and the like represent that the real action category corresponding to the sample image is a specific action category in the set, and the "other" label represents that the real action category corresponding to the sample image is a negative sample category.

[0111] The second data set is used to train the secondary classification branch of the action recognition model, and the second class label of the sample image in the second data set is used to represent whether the real action category corresponding to the sample image belongs to the positive sample category or the negative sample category. For example, the second class label includes labels such as 1 and 0. Among them, the label 1 represents that the real action category corresponding to the sample image belongs to the positive sample category, and the label 0 represents that the real action category corresponding to the sample image belongs to the negative sample category.

[0112] It is explained here that both the first data set and the second data set can be obtained by manual collection and annotation, for example, in the process of people making various actions, the actions are photographed, and then the action location and action category in the photographed image are annotated; or the original image can be obtained from the network by crawling, downloading and the like, and the action image area is segmented and annotated on the basis of the original image.

[0113] Step 430, based on each sample image in the first data set annotated with the first class label, the feature extraction layer and the main classification branch of the action recognition model are trained.

[0114] As shown in Figure 9 The training process of the above-mentioned main classification branch can include the following steps:

[0115] Step 431, a current sample image is obtained from the first data set and input into the feature extraction layer for feature extraction to obtain the image feature of the current sample image.

[0116] Step 433, input the image feature of the current sample image into the main classification branch of the action recognition model for training to obtain a first training result.

[0117] Among them, the first training result represents the action category predicted by the main classification branch.

[0118] Step 435, according to the difference between the first training result and the first class label of the current sample image, a first loss value is calculated.

[0119] The first type of label is used to indicate whether the labeled real action category belongs to a set action category or a negative sample category.

[0120] If the first loss value meets the training stop condition, step 437 is performed, otherwise, if the first loss value does not meet the training stop condition, step 439 is performed.

[0121] The training stop condition can be flexibly set according to the actual needs of the application scenario, for example, the training stop condition includes that the number of training times reaches a preset condition, the amount of training data reaches a preset condition, and the loss value reaches a preset condition, which is not limited here.

[0122] Step 437, update the parameters of the feature extraction layer and the main classification branch based on the first loss value, and continue to train the feature extraction layer and the main classification branch using other sample images in the first data set until the training stop condition is met, and then stop training to obtain the trained main classification branch.

[0123] Specifically, the current one sample image obtained from the first data set is resized and normalized to obtain an input image, and the obtained input image is input into the action recognition model to obtain a first training result through the action recognition of the main classification branch of the action recognition model.

[0124] Based on the first type of label indicating the labeled real action category of the current one sample image and the first training result indicating the predicted action category, a cross-entropy loss function is used to calculate the difference between the two, and a first loss value loss1 is obtained. Determine whether the first loss value loss1 meets the training stop condition. If yes, stop training to obtain the trained feature extraction layer and main classification branch; otherwise, use the first loss value loss1 for backpropagation training, update the parameters of the feature extraction layer and the main classification branch of the action recognition model, and input the next sample image in the first data set into the action recognition model for continuous training until the training of the feature extraction layer and the main classification branch is completed.

[0125] Step 450, based on the sample images labeled with the second type of label in the second data set and the trained feature extraction layer, train the secondary classification branch of the action recognition model.

[0126] After the training of the feature extraction layer and the main classification branch is completed using the first data set, the parameters of the feature extraction layer and the main classification branch are kept unchanged, and the secondary classification branch is trained using the second data set. That is, only the parameters of the secondary classification branch are updated during the training of the secondary classification branch.

[0127] In this way, the main classification branch and the secondary classification branch share the feature extraction layer, which not only effectively reduces the parameter quantity in the model training process and improves the model training rate, but also helps to reduce the size of the action recognition model, so that the action recognition model is more suitable for deployment in electronic devices with limited hardware resource configuration, for example, the electronic device can be a gateway type camera.

[0128] As shown in Figure 10 the training process of the secondary classification branch can include the following steps:

[0129] Step 451, a current sample image is obtained from the second data set and input into the feature extraction layer for feature extraction, to obtain image features of the current sample image.

[0130] Step 453, the image features of the current sample image are input into the secondary classification branch of the action recognition model for training, to obtain a second training result.

[0131] The second training result represents the action category predicted by the secondary classification branch.

[0132] Step 455, a second loss value is calculated according to the difference between the second training result and the second class label of the current sample image.

[0133] The second class label is used to indicate whether the labeled real action category belongs to the positive sample category or the negative sample category.

[0134] If the second loss value meets the training stop condition, step 457 is performed, otherwise, if the second loss value does not meet the training stop condition, step 459 is performed.

[0135] The training stop condition can be flexibly set according to the actual needs of the scene, for example, the training stop condition includes that the training times reach a preset condition, the training data amount reaches a preset condition, and the loss value reaches a preset condition, which is not limited here.

[0136] Step 457, the parameters of the secondary classification branch are updated based on the second loss value, and the secondary classification branch is further trained using other sample images in the second data set until the training stop condition is met, and the training is stopped to obtain the trained secondary classification branch.

[0137] Specifically, the current sample image obtained from the second data set is size-adjusted and normalized to obtain an input image, the obtained input image is input into the action recognition model, and the secondary classification branch of the action recognition model is used for action recognition to obtain the second training result.

[0138] Based on the second type of label indicated by the current sample image and the predicted action category indicated by the second training result, a difference between the two is calculated using a cross-entropy loss function to obtain a second loss value loss2. It is determined whether the second loss value loss2 meets a training stop condition. If yes, the training is stopped to obtain a completed secondary classification branch; otherwise, the second loss value loss2 is used for back propagation training to update the parameters of the secondary classification branch of the action recognition model, and the next sample image in the second data set is input into the action recognition model for continuous training until the secondary classification branch training is completed.

[0139] Step 470, in the case where the primary classification branch and the secondary classification branch are trained, an action recognition model is obtained.

[0140] After training, an action recognition model is obtained, which is trained and has the ability to recognize the action category in the to-be-detected image. Based on the action recognition model, the action in the to-be-detected image can be recognized as the basis for realizing automatic control of the equipment.

[0141] Please refer to Figure 11 The embodiment of the present application provides a device control method, which is suitable for an electronic device, which can be Figure 1 The user terminal 110 in the implementation environment shown.

[0142] In the following method embodiment, in order to facilitate description, the execution subject of each step of the method is taken as an example of an electronic device, but this does not constitute a specific limitation.

[0143] As Figure 11 shown, the method can include the following steps:

[0144] Step 510, in response to an operation triggered in the automation configuration page, determining the trigger condition and the associated controlled data.

[0145] Among them, the automation configuration page is an interactive page for indicating that the user configures the automation scheme, and the user can configure the trigger condition and the controlled data associated with the automation scheme in the page. The trigger condition is related to the action category, and the controlled data is used to indicate that the device performs an automatic operation corresponding to the action category.

[0146] Step 530, generating an automation scheme corresponding to the action category according to the trigger condition and the associated controlled data.

[0147] The automation scheme is used to achieve automated control of the device by recognizing the action categories in the image to be detected. The recognition of action categories is achieved through the main classification branch and the sub-classification branch in the action recognition model. The action recognition model is obtained by adding a sub-classification branch on the basis of the main classification branch and training it.

[0148] After configuring the automation scheme on the user terminal through the above process, the automated control of the equipment can be realized based on the automation scheme, thereby improving the user experience.

[0149] Figures 12a to 12c The diagram illustrates the specific implementation of a device control method in an application scenario. In this scenario, the action is a gesture; correspondingly, the target detection model is a hand detection model, and the action recognition model is a gesture recognition model.

[0150] like Figure 12a As shown, through Figure 1 The diagram illustrates how a gateway-type camera in the implementation environment captures and collects images of hand gestures, obtaining an image to be detected containing the gesture. After obtaining the image, a hand detection algorithm provided by a hand detection model is used to detect the hand in the image. If no hand is detected, the process continues. If a hand is detected, a gesture classification algorithm provided by an action recognition model is used to classify and predict the gesture. If the gesture belongs to a negative sample category, the process continues. If the gesture belongs to a positive sample category, the device can be controlled to perform corresponding automated operations based on the specific gesture category.

[0151] The following is combined Figure 12b and Figure 12c The following explains the process of automating equipment control using gestures:

[0152] like Figure 12b As shown, after the user clicks "When", they can enter page 401, which is used to configure trigger conditions, from the automation configuration page 301. They can then view the trigger conditions that can generate automation schemes. For example, on page 401, the user can select one of the trigger conditions by clicking "Trigger Condition 1-4", or they can enter the trigger conditions they expect in the "Enter Trigger Condition" dialog box to complete the configuration of the trigger conditions in the automation scheme.

[0153] Continue reading Figure 12bAfter the user clicks on "OK", the user can enter the page 501 for configuring the controlled data from the automation configuration page 301, and view the controlled data that can generate the automation scheme, for example, in the page 501, the user can select one of the controlled data by clicking on "Controlled data 1-4", or input the controlled data expected by the user in the "Input controlled data" dialog box, so as to complete the configuration of the controlled data in the automation scheme.

[0154] In Figure 12c As shown in the table, the gesture actions can include "two", "four", "five", and "eight", and the corresponding automation operations can include, but are not limited to, controlling the smart sound to play, turn off the music; controlling the smart lamp to turn on and off the light; controlling the smart air conditioner to turn on and off the air conditioner, and the like.

[0155] After the configuration of the trigger condition and the controlled data is completed, the corresponding automation scheme can be generated, please refer to Figure 12b In the automation scheme 601, the trigger condition is to recognize the gesture action "two", and the controlled data is to control the smart sound to play the music.

[0156] Then, based on Figure 12a As shown in the flow, the gesture action is recognized as "two", which is regarded as that the trigger condition in the automation scheme 601 is met, at this time, the corresponding device control instruction is generated based on the controlled data in the automation scheme 601 and sent to the smart sound, and finally the smart sound is controlled to play the music, so as to complete the automation control of the smart sound based on the gesture action.

[0157] In this application scenario, the automation control of the smart device can be accurately realized by a simple gesture, which greatly improves the user experience.

[0158] The following is an apparatus embodiment of the present application, which can be used to execute the action recognition method or the device control method involved in the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the action recognition method or the device control method involved in the present application.

[0159] Please refer to Figure 13 In the embodiment of the present application, an action recognition apparatus 600 is provided, which includes but is not limited to an image acquisition module 610, an action recognition module 630, and a result output module 650.

[0160] The image acquisition module 610 is configured to acquire a to-be-detected image.

[0161] The action recognition module 630 is configured to input the to-be-detected image into an action recognition model, perform action recognition on the to-be-detected image through a main classification branch and a secondary classification branch of the action recognition model respectively, and obtain a first classification result and a second classification result. The action recognition model is obtained by adding the secondary classification branch to the main classification branch and training.

[0162] The result output module 650 is configured to obtain an action recognition result according to the first classification result corresponding to the main classification branch and the second classification result corresponding to the secondary classification branch.

[0163] In an example embodiment, the action recognition model includes a feature extraction layer, a main classification branch and a secondary classification branch connected to the feature extraction layer respectively; the device further includes a model training module configured to train the action recognition model; the model training module includes a data set acquisition unit configured to acquire a first data set and a second data set; the first data set includes sample images labeled with a first type of label; the second data set includes sample images labeled with a second type of label; a first training unit configured to train the feature extraction layer and the main classification branch of the action recognition model based on each sample image labeled with the first type of label in the first data set; a second training unit configured to train the secondary classification branch of the action recognition model based on each sample image labeled with the second type of label in the second data set and the trained feature extraction layer; and an action recognition unit configured to obtain the trained action recognition model in the case that the main classification branch and the secondary classification branch are trained.

[0164] In an example embodiment, the model training module is further configured to acquire a current sample image from the first data set, input the current sample image into the feature extraction layer for feature extraction, and obtain image features of the current sample image; input the image features of the current sample image into the main classification branch of the action recognition model for training, and obtain a first training result; the first training result represents a predicted action category; calculate a first loss value according to a difference between the first training result and a first type of label of the current sample image; the first type of label is used to indicate whether the labeled real action category belongs to a set action category or a negative sample category; update parameters of the feature extraction layer and the main classification branch based on the first loss value, and continue to train the feature extraction layer and the main classification branch using other sample images in the first data set until a training stop condition is met, and then stop training to obtain the trained main classification branch.

[0165] In an example embodiment, the model training module is further configured to: perform feature extraction on a current sample image input from the second data set by using the feature extraction layer to obtain image features of the current sample image; input the image features of the current sample image into the secondary classification branch of the action recognition model to obtain a second training result, the second training result representing a predicted action category; calculate a second loss value based on a difference between the second training result and a second category label of the current sample image, the second category label indicating whether a labeled real action category belongs to a positive sample category or a negative sample category; update parameters of the secondary classification branch based on the second loss value, and continue to train the secondary classification branch using other sample images in the second data set until a training stop condition is met, and then stop training to obtain the trained secondary classification branch.

[0166] In an example embodiment, the action recognition module is further configured to: perform feature extraction on the to-be-detected image by using the feature extraction layer to obtain image features of the to-be-detected image; perform action classification prediction on the image features by using the primary classification branch of the action recognition model to obtain the first classification result; and perform action classification prediction on the image features by using the secondary classification branch of the action recognition model to obtain the second classification result.

[0167] In an example embodiment, the result output module is further configured to: if the action category indicated by the first classification result is a set action category, determine whether the action category indicated by the second classification result is a positive sample category; the first classification result is used to indicate whether the action category is a set action category or a negative sample category; the second classification result is used to indicate whether the action category is a positive sample category or a negative sample category; if yes, the first classification result is taken as the action recognition result; if no, the second classification result is taken as the action recognition result; and if the action category indicated by the first classification result is a negative sample category, the first classification result is taken as the action recognition result.

[0168] In an example embodiment, the apparatus further includes a device control module configured to control an intelligent device to perform an automatic operation corresponding to an action category to which an action in the to-be-detected image belongs according to the action recognition result.

[0169] In an example embodiment, the device control module comprises: an instruction generation unit configured to generate a device control instruction corresponding to the action category if the action recognition result indicates that the action category to which the action in the image to be detected belongs is a set action category; and an instruction sending unit configured to send the device control instruction to the smart device, so that the smart device performs a corresponding automatic operation in response to the device control instruction.

[0170] In an example embodiment, the apparatus further comprises: a target detection module configured to perform target detection on the image to be detected to obtain a position of a target object in the image to be detected; extract an image region in which the target object is located from the image to be detected according to the position of the target object to obtain a target image, so that the action recognition is performed based on the target image.

[0171] In an example embodiment, the apparatus further comprises: a target detection module configured to perform target detection on the image to be detected to obtain a position of a target object in the image to be detected; extract an image region in which the target object is located from the image to be detected according to the position of the target object to obtain a target image, so that the action recognition is performed based on the target image.

[0172] In an example embodiment, the apparatus further comprises: a target detection module configured to perform target detection on the image to be detected to obtain a position of a target object in the image to be detected; extract an image region in which the target object is located from the image to be detected according to the position of the target object to obtain a target image, so that the action recognition is performed based on the target image.

[0173] The automation scheme generation module is configured to generate an automation scheme corresponding to the action category according to the trigger condition and the associated controlled data; the automation scheme is used to realize the automation control of the device by identifying the action category in the image to be detected; the identification of the action category is realized by a main classification branch and a secondary classification branch in an action recognition model; the action recognition model is obtained by adding the secondary classification branch on the basis of the main classification branch and training.

[0174] It should be noted that the apparatus provided in the above embodiments is only described above in terms of the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, i.e., the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above.

[0175] In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments, which will not be described here again.

[0176] Figure 14 is a structural schematic diagram of an electronic device according to an example embodiment. The electronic device can comprise: a gateway type camera.

[0177] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or depending on any specific feature. Figure 14 One or more components of the exemplary electronic device 2000 shown.

[0178] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 14 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0179] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0180] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, to perform... Figure 1 The diagram illustrates the interaction between user terminal 110 and gateway 150 in the implementation environment.

[0181] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 14 As shown, this does not constitute a specific limitation.

[0182] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0183] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0184] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 14 (Not shown), each module may contain computer-readable instructions for electronic device 2000. For example, the motion recognition device may be considered as application program 253 deployed on electronic device 2000.

[0185] The data 255 can be photos, pictures, etc. stored in a disk, and can also be images to be detected stored in the memory 250.

[0186] The central processor 270 can include one or more than one processor, and is configured to communicate with the memory 250 through at least one communication bus, to read the computer readable instructions stored in the memory 250, and to implement the operation and processing of the mass data 255 in the memory 250. For example, the action recognition method is completed by the central processor 270 reading a series of computer readable instructions stored in the memory 250.

[0187] In addition, the present application can also be implemented by hardware circuit or hardware circuit combined with software, and therefore, the implementation of the present application is not limited to any specific hardware circuit, software and combination of the two.

[0188] Please refer to Figure 15 , Figure 15 is a structural schematic diagram of a terminal according to an exemplary embodiment. The terminal is suitable for Figure 1 a user terminal 110 in the implementation environment shown.

[0189] It should be noted that the terminal is only an example suitable for the present application, and should not be considered as providing any limitation on the use range of the present application. The terminal should also not be interpreted as requiring to depend on or must have Figure 15 one or more components in the exemplary terminal 1100 shown in the

[0190] As shown in Figure 15 , the terminal 110 includes a memory 101, a storage controller 103, one or more (only one is shown in the Figure 15 ) processors 105, a peripheral interface 107, a radio frequency module 109, a positioning module 111, a camera module 113, an audio module 115, a touch screen 117, and a key module 119. These components communicate with each other through one or more communication buses / signal lines 121.

[0191] The memory 101 can be used to store computer programs and modules, such as the computer programs and modules corresponding to the device control method and device of the exemplary embodiments of the present application. The processor 105 performs various functions and data processing by running the computer programs stored in the memory 101, i.e., completes the device control method.

[0192] The memory 101 as a carrier for resource storage can be a random access memory, for example, a high-speed random access memory, a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other solid state memories. The storage mode can be temporary storage or permanent storage.

[0193] The peripheral interface 107 can include at least one wired or wireless network interface, at least one serial or parallel interface, at least one input / output interface, at least one USB interface, and the like, for coupling external input / output devices to the memory 101 and the processor 105 to enable communication with the external input / output devices.

[0194] The radio frequency module 109 is configured to transceive electromagnetic waves, and to convert electromagnetic waves and electrical signals to each other, so as to communicate with other devices through a communication network. The communication network can include a cellular telephone network, a wireless local area network, or a metropolitan area network, which can use various communication standards, protocols, and technologies.

[0195] The positioning module 111 is configured to obtain a current geographical position of the terminal 1100. Examples of the positioning module 111 include, but are not limited to, a global positioning system (GPS), a wireless local area network-based positioning technology, or a mobile communication network-based positioning technology.

[0196] The camera module 113 is affiliated to a camera, and is configured to capture pictures or videos. The captured pictures or videos can be stored in the memory 101, and can also be transmitted to a host computer through the radio frequency module 109.

[0197] The audio module 115 provides an audio interface for a user, and can include one or more microphone interfaces, one or more speaker interfaces, and one or more earphone interfaces. The audio module 115 is configured to interact with other devices through the audio interfaces. The audio data can be stored in the memory 101, and can also be transmitted through the radio frequency module 109.

[0198] The touch screen 117 provides an input / output interface between the terminal 1100 and a user. Specifically, the user can perform input operations, such as click, touch, swipe, and the like, through the touch screen 117, so that the terminal 1100 responds to the input operations. The terminal 1100 displays output content in the form of text, pictures, or videos, or any combination thereof, to the user through the touch screen 117.

[0199] The key module 119 includes at least one key, which provides an interface for the user to input to the terminal 1100. The user can press different keys to make the terminal 1100 perform different functions. For example, a sound adjustment key can be used by the user to adjust the volume of the sound played by the terminal 1100.

[0200] It can be understood that the structure shown is only schematic, and the terminal 1100 can include more or fewer components than those shown in FIG. 1, or have different components from those shown in FIG. 1. Figure 15 It can be understood that the structure shown is only schematic, and the terminal 1100 can include more or fewer components than those shown in FIG. 1, or have different components from those shown in FIG. 1. Figure 15 It can be understood that the structure shown is only schematic, and the terminal 1100 can include more or fewer components than those shown in FIG. 1, or have different components from those shown in FIG. 1. Figure 15 It can be understood that the structure shown is only schematic, and the terminal 1100 can include more or fewer components than those shown in FIG. 1, or have different components from those shown in FIG. 1. Figure 15The components shown in the figures can be implemented in hardware, software, or a combination thereof.

[0201] Please refer to Figure 16 The electronic device 4000 provided by the embodiment of the present application can include a gateway type camera.

[0202] In Figure 16 The electronic device 4000 includes at least one processor 4001, at least one communication bus 4002, and at least one memory 4003.

[0203] The processor 4001 and the memory 4003 are connected, for example, through the communication bus 4002. Optionally, the electronic device 4000 can further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0204] The processor 4001 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure content of the present application. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0205] The communication bus 4002 can include a channel for transmitting information between the above components. The communication bus 4002 can be a PCI (Peripheral Component Interconnect, peripheral component interconnect) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience, Figure 16 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or only one type of bus.

[0206] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program instructions in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0207] The memory 4003 stores computer readable instructions, and the processor 4001 reads the computer readable instructions stored in the memory 4003 through the communication bus 4002.

[0208] The computer readable instructions are executed by the processor 4001 to implement the method in each of the above embodiments.

[0209] In addition, the embodiments of the present application provide a storage medium, and the storage medium stores computer readable instructions, and the computer readable instructions are executed by one or more processors to implement the method in each of the above embodiments.

[0210] The embodiments of the present application provide a computer program product, and the computer program product includes computer readable instructions stored in a storage medium. One or more processors of an electronic device read the computer readable instructions from the storage medium, load and execute the computer readable instructions, so that the electronic device executes the method in each of the above embodiments.

[0211] Compared with the related art, the action recognition model in the present application adds a secondary classification branch on the basis of the primary classification branch, which is used to distinguish similar actions, so as to assist the first classification result by the second classification result to distinguish the action category more accurately, avoid the similar actions from being misrecognized, and thus effectively solve the problem of high action misrecognition rate in the related art leading to device misoperation.

[0212] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order following the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not strictly limited to the order indicated by the arrows, and can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of which is not necessarily sequential, but can be round-robin or alternating with at least some of the other steps or sub-steps or stages of other steps.

[0213] The above only describes some embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. An action recognition method, characterized in that, The method includes: The image to be detected is acquired, and target detection is performed on the image to be detected to obtain the target image; the target image refers to an image containing the target object performing the action. The target image is input into an action recognition model. The main classification branch and secondary classification branch of the action recognition model are used to perform action recognition on the target image, respectively, to obtain a first classification result and a second classification result. The action recognition model is obtained by adding a secondary classification branch to the main classification branch and then training it. The main classification branch is used to identify the action category of the target object in the target image; the secondary classification branch is used to identify whether the action category of the target object in the target image is a positive or negative sample category. If the action category indicated by the first classification result is a set action category, then determine whether the action category indicated by the second classification result is a positive sample category; the first classification result is used to indicate whether the action category is a set action category or a negative sample category; the second classification result is used to indicate whether the action category is a positive sample category or a negative sample category. If so, the first classification result is taken as the action recognition result; If not, the second classification result will be used as the action recognition result; If the action category indicated by the first classification result is a negative sample category, then the first classification result is taken as the action recognition result.

2. The method as described in claim 1, characterized in that, The action recognition model includes a feature extraction layer, a main classification branch and a secondary classification branch respectively connected to the feature extraction layer; The training process of the action recognition model includes: Obtain a first dataset and a second dataset; the first dataset includes sample images labeled with a first class of labels; the second dataset includes sample images labeled with a second class of labels; Based on the sample images labeled with the first class of labels in the first dataset, the feature extraction layer and the main classification branch of the action recognition model are trained; Based on the sample images labeled with the second class of labels in the second dataset and the completed feature extraction layer, the sub-classification branch of the action recognition model is trained. Once the main classification branch and the sub-classification branch have completed training, the trained action recognition model is obtained.

3. The method as described in claim 2, characterized in that, The training process of the main classification branch includes: The current sample image is obtained from the first dataset and input into the feature extraction layer for feature extraction to obtain the image features of the current sample image; The image features of the current sample image are input into the main classification branch of the action recognition model for training, and a first training result is obtained; the first training result represents the predicted action category. A first loss value is calculated based on the difference between the first training result and the first class label of the current sample image; the first class label is used to indicate whether the labeled true action category belongs to the set action category or the negative sample category; The parameters of the feature extraction layer and the main classification branch are updated based on the first loss value, and the feature extraction layer and the main classification branch are trained using other sample images in the first dataset until the training stopping condition is met, at which point training stops and the trained main classification branch is obtained.

4. The method as described in claim 2, characterized in that, The training process for the sub-classification branch includes: The current sample image is input from the second dataset to complete the training of the feature extraction layer, which then extracts features to obtain the image features of the current sample image. The image features of the current sample image are input into the sub-classification branch of the action recognition model for training, resulting in a second training result; the second training result represents the predicted action category. The second loss value is calculated based on the difference between the second training result and the second class label of the current sample image; the second class label is used to indicate whether the labeled true action category belongs to the positive sample category or the negative sample category. The parameters of the subclassification branch are updated based on the second loss value, and the subclassification branch is trained using other sample images in the second dataset until the training stopping condition is met, at which point training is stopped and the trained subclassification branch is obtained.

5. The method as described in claim 2, characterized in that, The step of performing action recognition on the target image through the main classification branch and the secondary classification branch in the action recognition model to obtain a first classification result and a second classification result includes: The feature extraction layer is used to extract features from the target image to obtain the image features of the target image; The image features are predicted to be classified into actions using the main classification branch of the action recognition model, thus obtaining the first classification result. By using the sub-classification branch of the action recognition model, the image features are predicted to be classified into actions, thus obtaining the second classification result.

6. The method as described in claim 1, characterized in that, The method further includes: Based on the action category of the action in the target image indicated by the action recognition result, the intelligent device is controlled to perform automated operations corresponding to the action category.

7. The method as described in claim 6, characterized in that, The step of controlling the smart device to perform automated operations corresponding to the action category indicated by the action recognition result in the target image includes: If the action recognition result indicates that the action in the target image belongs to a set action category, then a device control command corresponding to the action category is generated; The device control command is sent to the smart device, causing the smart device to perform corresponding automated operations in response to the device control command.

8. The method according to any one of claims 1 to 7, characterized in that, The step of performing target detection on the image to be detected to obtain the target image includes: Target detection is performed on the image to be detected to obtain the position of the target object in the image to be detected; Based on the location of the target object, the image region where the target object is located is extracted from the image to be detected to obtain the target image, so that action recognition is based on the target image.

9. A device control method, characterized in that, The method includes: In response to an operation triggered in the automation configuration page, a trigger condition and associated controlled data are determined; the trigger condition is related to the action category, and the controlled data is used to instruct the smart device to perform an automated operation corresponding to the action category. Based on the triggering conditions and the associated controlled data, an automation scheme corresponding to the action category is generated. The automation scheme is used to achieve automated control of the smart device by identifying the action category of a target object in a target image. The target image is obtained by performing target detection on an image to be detected. The target image refers to an image containing a target object performing an action. The identification of the action category of the target object in the target image is achieved through the first classification result corresponding to the main classification branch and the second classification result corresponding to the secondary classification branch in the action recognition model. The first classification result indicates whether the action category is a positive action category or a negative sample category. The second classification result indicates whether the action category is a positive sample category or a negative sample category. The action recognition model is obtained by adding a secondary classification branch to the main classification branch and training it. The main classification branch is used to identify the action category of the target object in the target image. The secondary classification branch is used to identify whether the action category of the target object in the target image is a positive sample category or a negative sample category. The identification of the action category of the target object in the target image includes: if the action category indicated by the first classification result is a set action category, then determining whether the action category indicated by the second classification result is a positive sample category; if yes, then the first classification result is used as the action recognition result; if no, then the second classification result is used as the action recognition result; if the action category indicated by the first classification result is a negative sample category, then the first classification result is used as the action recognition result; the action recognition result is used to indicate the action category of the target object in the target image.

10. A motion recognition device, characterized in that, The device includes: The image acquisition module is used to acquire the image to be detected and perform target detection on the image to be detected to obtain the target image; the target image refers to an image containing the target object performing the action; An action recognition module is used to input the target image into an action recognition model, and to perform action recognition on the target image through a main classification branch and a secondary classification branch in the action recognition model, respectively, to obtain a first classification result and a second classification result; the action recognition model is obtained by adding a secondary classification branch to the main classification branch and then training it; the main classification branch is used to identify the action category of the target object in the target image; the secondary classification branch is used to identify whether the action category of the target object in the target image is a positive sample category or a negative sample category; The result output module is used to determine whether the action category indicated by the second classification result is a positive sample category if the action category indicated by the first classification result is a set action category; the first classification result is used to indicate whether the action category is a set action category or a negative sample category; the second classification result is used to indicate whether the action category is a positive sample category or a negative sample category; if yes, the first classification result is used as the action recognition result; if no, the second classification result is used as the action recognition result; if the action category indicated by the first classification result is a negative sample category, the first classification result is used as the action recognition result.

11. A device control apparatus, characterized in that, The device includes: The parameter determination module is used to determine the triggering conditions and associated controlled data in response to an operation triggered in the automation configuration page; the triggering conditions are related to the action category, and the controlled data is used to instruct the smart device to perform an automated operation corresponding to the action category; An automated scheme generation module is used to generate automated schemes corresponding to action categories based on triggering conditions and associated controlled data. The automated schemes are used to achieve automated control of the intelligent device by identifying the action category of a target object in a target image. The target image is obtained by performing target detection on an image to be detected. The target image refers to an image containing a target object performing an action. The identification of the action category of the target object in the target image is achieved through the first classification result corresponding to the main classification branch and the second classification result corresponding to the secondary classification branch in the action recognition model. The first classification result indicates whether the action category is a positive action category or a negative sample category; the second classification result indicates whether the action category is a positive sample category or a negative sample category. The action recognition model is obtained by adding a secondary classification branch to the main classification branch and training it. The main classification branch is used to identify the action category of the target object in the target image; the secondary classification branch is used to identify whether the action category of the target object in the target image is a positive sample category or a negative sample category. The identification of the action category of the target object in the target image includes: if the action category indicated by the first classification result is a set action category, then determining whether the action category indicated by the second classification result is a positive sample category; if yes, then the first classification result is used as the action recognition result; if no, then the second classification result is used as the action recognition result; if the action category indicated by the first classification result is a negative sample category, then the first classification result is used as the action recognition result; the action recognition result is used to indicate the action category of the target object in the target image.

12. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by the processor, causing the electronic device to implement the action recognition method or device control method as described in any one of claims 1 to 9.

13. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by a processor to implement the action recognition method or device control method as described in any one of claims 1 to 9.

Images