A method, device, storage medium and electronic device for constructing an action recognition model

Abstract

The present invention discloses a method, device, storage medium and electronic device for constructing an action recognition model, the method comprising: obtaining an image training sample and inputting the image training sample into a target neural network; using a detection branch network in the target neural network to process each image in the image training sample to obtain a corresponding first direction vector, a second direction vector and a third direction vector; training a preset neural network model according to the action state of a target person in each image and the first direction vector, the second direction vector and the third direction vector to obtain an action recognition model. The method can achieve real-time performance through a target neural network; the action recognition model obtained by training the action state of each image and the corresponding direction vector does not require complex post-processing and can directly recognize fine-grained actions.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile driving assistance terminals, and in particular to a method, device, storage medium and electronic equipment for constructing an action recognition model. Background Art

[0002] Hands-off detection and safe belt fastness detection are two important aspects of cockpit driver status detection. Judging whether the driver is holding the steering wheel or wearing a seat belt based on traditional sensors is easily subject to human interference. For example, a pressure-sensitive steering wheel can use a clip with appropriate force or interference from outsiders. Similarly, wearing a seat belt is also susceptible to human interference. Currently, vision-based recognition solutions have become mainstream. Existing vision-based recognition solutions all identify objects (hands, seat belts, steering wheels), and then use complex post-processing logic to determine whether the driver is holding the steering wheel or wearing a seat belt. However, the post-processing logic is complex and it is difficult to distinguish fine-grained action states. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method, device, storage medium and electronic device for constructing an action recognition model to solve the technical problems in the prior art that complex post-processing logic is required when outputting recognition results through an action recognition model and it is difficult to distinguish fine-grained action states.

[0004] The technical solution proposed by the present invention is as follows:

[0005] A first aspect of an embodiment of the present invention provides a method for constructing an action recognition model, which includes: obtaining image training samples and inputting the image training samples into a target neural network; using a detection branch network in the target neural network to detect the entire body of a target person, the key parts of the target person corresponding to the action to be identified, and the object to be identified contained in each image in the image training samples, to obtain multiple body center points of the target person, multiple key part center points of the key parts of the target person corresponding to the action to be identified, and multiple object center points of the object to be identified, each body center point, key part center point, and object center point corresponds to a direction vector; according to the multiple body center points and the direction vectors corresponding to each body center point, the multiple key part center points and the direction vectors corresponding to each key part center point, and the multiple object center points and the direction vectors corresponding to each object center point, a matching network in the target neural network is used to perform regression matching processing to obtain a unique body center point and its corresponding direction vector, a unique key part center point, and a direction vector corresponding to each body center point. The invention relates to a method for identifying a target person's body action point and its corresponding direction vector and a unique object center point and its corresponding direction vector; determining the distance center point and its corresponding direction vector of the target person's body action point corresponding to the action to be identified from the action object according to the unique key part center point and its corresponding direction vector and the unique object center point and its corresponding direction vector; determining the first direction vector between the distance center point and the unique key part center point according to the distance center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector, determining the second direction vector between the distance center point and the unique object center point according to the distance center point and its corresponding direction vector and the unique object center point and its corresponding direction vector, determining the third direction vector between the unique body center point and the unique key part center point according to the unique body center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector; training a preset neural network model according to the action state of the target person in each image and the first direction vector, the second direction vector and the third direction vector to obtain an action recognition model.

[0006] Optionally, the method also includes: taking the multiple body center points, the multiple key part center points and the multiple object center points as multiple target center points to be identified; for any type of multiple target center points to be identified, grouping the multiple target center points to be identified; sorting the multiple target center points to be identified in each group after grouping according to detection confidence; calculating the distance between the direction vector corresponding to the target center point to be identified with the highest detection confidence and the direction vectors corresponding to other target center points to be identified in each group; comparing the distance with a preset threshold; suppressing other center points whose distances are less than the preset threshold and their corresponding direction vectors to obtain the target center point to be identified with the highest detection confidence and its corresponding direction vector, and taking the target center point to be identified with the highest detection confidence and its corresponding direction vector as the only center point of the type and its corresponding direction vector.

[0007] Optionally, the target neural network includes any one of VoVNet, ResNet, DenseNet, EfficientNet, ShuffleNet, and MobileNet.

[0008] A second aspect of an embodiment of the present invention provides an action recognition method, which includes: obtaining an image to be recognized; inputting the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; inputting the image to be recognized and the first direction vector, the second direction vector, and the third direction vector into the action recognition model obtained by the action recognition model construction method described in the first aspect of the embodiment of the present invention and any one of the first aspect to process and obtain an action recognition result.

[0009] The third aspect of an embodiment of the present invention provides an action recognition model construction device, which includes: an acquisition module, which is used to acquire image training samples and input the image training samples into a target neural network; a detection module, which is used to use the detection branch network in the target neural network to detect the entire body of the target person contained in each image in the image training samples, the key parts of the target person corresponding to the action to be identified, and the object to be identified, respectively, to obtain multiple body center points of the target person's entire body, multiple key part center points of the key parts of the target person corresponding to the action to be identified, and multiple object center points of the object to be identified, each body center point, key part center point, and object center point corresponds to a direction vector; a matching module, which is used to perform regression matching processing using the matching network in the target neural network according to the multiple body center points and the direction vectors corresponding to each body center point, the multiple key part center points and the direction vectors corresponding to each key part center point, and the multiple object center points and the direction vectors corresponding to each object center point, to obtain a unique body center point and its corresponding direction vector, a unique key part center point, and a direction vector corresponding to each body center point. a first determining module, used to determine the distance center point and the corresponding direction vector of the target person's body action point from the action object corresponding to the action to be identified according to the unique key part center point and the corresponding direction vector and the unique object center point and the corresponding direction vector; a second determining module, used to determine the first direction vector of the distance center point and the unique key part center point according to the distance center point and the corresponding direction vector and the unique key part center point and the corresponding direction vector, determine the second direction vector of the distance center point and the unique object center point according to the distance center point and the corresponding direction vector and the unique object center point and the corresponding direction vector, and determine the third direction vector between the unique body center point and the unique key part center point according to the unique body center point and the corresponding direction vector and the unique key part center point and the corresponding direction vector; a training module, used to train a preset neural network model according to the action state of the target person in each image and the first direction vector, the second direction vector and the third direction vector to obtain an action recognition model.

[0010] Optionally, the device also includes: a third determination module, which is used to take the multiple body center points, the multiple key part center points and the multiple object center points as multiple target center points to be identified; a grouping module, which is used to group the multiple target center points to be identified for any type of multiple target center points to be identified; a sorting module, which is used to sort the multiple target center points to be identified corresponding to each group after grouping according to detection confidence; a calculation module, which is used to calculate the distance between the direction vector corresponding to the target center point to be identified with the highest detection confidence and the direction vectors corresponding to other target center points to be identified in each group; a comparison module, which is used to compare the distance with a preset threshold; a processing module, which is used to suppress other center points whose distances are less than the preset threshold and their corresponding direction vectors to obtain the target center point to be identified with the highest detection confidence and its corresponding direction vector, and use the target center point to be identified with the highest detection confidence and its corresponding direction vector as the only center point of the type and its corresponding direction vector.

[0011] Optionally, the target neural network includes any one of VoVNet, ResNet, DenseNet, EfficientNet, ShuffleNet, and MobileNet.

[0012] A fourth aspect of an embodiment of the present invention provides a motion recognition device, which includes: a second acquisition module, used to acquire an image to be recognized; an input module, used to input the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; a recognition module, used to input the image to be recognized and the first direction vector, the second direction vector, and the third direction vector into the motion recognition model obtained by the motion recognition model construction method described in the first aspect of the embodiment of the present invention and any one of the first aspects for processing to obtain a motion recognition result.

[0013] A fifth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the action recognition model construction method as described in the first aspect of the embodiment of the present invention and any one of the first aspects, or to execute the action recognition method as described in the second aspect of the embodiment of the present invention.

[0014] A sixth aspect of an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the action recognition model construction method as described in the first aspect of the embodiment of the present invention and any one of the first aspects, or executing the action recognition method as described in the second aspect of the embodiment of the present invention.

[0015] The technical solution provided by the present invention has the following effects:

[0016] The method for constructing an action recognition model provided by an embodiment of the present invention obtains image training samples and inputs the image training samples into a target neural network; uses a detection branch network in the target neural network to process each image in the image training samples to obtain a corresponding first direction vector, a second direction vector, and a third direction vector; trains a preset neural network model according to the action state of the target person in each image and the first direction vector, the second direction vector, and the third direction vector to obtain an action recognition model. The method can achieve real-time performance through the target neural network; the action recognition model obtained by training the action state of each image and the corresponding direction vector does not require complex post-processing and can directly recognize fine-grained actions.

[0017] The action recognition method provided by the embodiment of the present invention obtains an image to be recognized; inputs the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; inputs the image to be recognized and the first direction vector, the second direction vector, and the third direction vector into the action recognition model obtained by the action recognition model construction method to process and obtain an action recognition result. This method performs action recognition based on a single frame image and is faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a flowchart of a method for constructing an action recognition model according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a motion recognition method according to an embodiment of the present invention;

[0021] Figure 3 is a flow chart of a motion recognition method according to an embodiment of the present invention;

[0022] Figure 4 is a structural block diagram of a device for constructing a motion recognition model according to an embodiment of the present invention;

[0023] Figure 5 is a structural block diagram of a motion recognition device according to an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] The embodiment of the present invention provides a method for constructing an action recognition model. Figure 1 As shown, the method comprises the following steps:

[0028] Step S101: Obtain image training samples and input the image training samples into a target neural network. Specifically, the image training samples include a plurality of acquired images to be processed.

[0029] Step S102: Utilize the detection branch network in the target neural network to respectively detect the entire body of the target person, the key parts of the target person corresponding to the action to be identified, and the object of the action to be identified contained in each image in the image training sample to obtain multiple body center points of the target person's entire body, multiple key part center points of the key parts of the target person corresponding to the action to be identified, and multiple object center points of the object of the action to be identified, each body center point, key part center point, and object center point corresponds to a direction vector.

[0030] Specifically, the target neural network contains two branches: a detection branch and a matching branch. Before identifying the action state of each image, it is necessary to determine the entire body of the target person contained in each image, the key parts of the target person corresponding to the action to be identified, the center point of the object to be identified, and the direction vector corresponding to each center point. The detection branch network can be used to obtain the entire body of the target person contained in each image, the key parts of the target person corresponding to the action to be identified, and multiple center points of the object to be identified, as well as the direction vector corresponding to each center point. Among them, the key parts of the target person corresponding to the action to be identified are determined according to the action to be identified of the target person contained in the image.

[0031] Each center point corresponds to a direction vector, and different points are matched by the direction vector. First, the detection branch outputs the center point position, and then the matching branch will have a corresponding direction vector at the corresponding position. Specifically, the detection branch detects the corresponding center point on the image and displays the confidence of each center point at the position of each center point. The coordinates (u, v) of the vector at the position of each center point in the image and the corresponding confidence are obtained by the matching branch. There can be multiple direction vectors at each position, and the direction vector with the highest confidence can be used as the direction vector of the corresponding center point. The method of determining the corresponding confidence from the multiple direction vectors at each position can be determined according to the relative orientation relationship between the detected body center point, the key part center point and the object center point. The direction vector that best reflects the relative orientation relationship between the three is set to have the highest confidence. Since each center point position on the image detected by the detection branch corresponds to each center point position matched by the matching branch, if the detection branch detects multiple center points, there will be as many direction vectors at the corresponding positions on the matching branch.

[0032] In one embodiment, hands on the steering wheel and wearing a seat belt are identified. Image samples obtained in the vehicle are input into the target neural network, wherein the key part of the target person corresponding to the hands on the steering wheel identification task is the hands, and the key part of the target person corresponding to the wearing seat belt identification task is the upper torso of the target person. Using the detection branch network, multiple center points of the target person's entire body, multiple center points corresponding to the hands, and multiple center points corresponding to the upper torso of the target person, multiple center points corresponding to the steering wheel, and multiple center points corresponding to the seat belt, and the direction vector corresponding to each center point of the body, the direction vector corresponding to each center point of the hand, and the direction vector corresponding to each center point of the upper torso, the direction vector corresponding to each center point of the steering wheel, and the direction vector corresponding to each center point of the seat belt can be obtained.

[0033] Step S103: Based on the multiple body center points and the direction vectors corresponding to each body center point, the multiple key part center points and the direction vectors corresponding to each key part center point, and the multiple object center points and the direction vectors corresponding to each object center point, a matching network in the target neural network is used to perform regression matching processing to obtain a unique body center point and its corresponding direction vector, a unique key part center point and its corresponding direction vector, and a unique object center point and its corresponding direction vector.

[0034] For example, the detection branch initially outputs points at many different locations, such as 10 for the hand, 5 for the steering wheel, and 3 for the driver. At the same time, the matching branch will also have 10, 5, and 3 direction vectors at the corresponding locations. Then, matching is performed based on the direction vectors to find the optimal matching relationship between the hand and the steering wheel, and between the hand and the driver.

[0035] Specifically, for 10 hands and 5 steering wheels, there are a total of 10*5 deviation values ​​d. For each hand, there are 5 deviation values ​​d. The average value of the 5 deviation values ​​d is used to get d'. After processing the 10 hands, 10 d' can be obtained. Then these 10 d' are sorted from large to small according to the confidence of the 10 hands. When the adjacent d' values ​​are close (such as the deviation is less than 30%), the hand with lower confidence is removed. Similarly, the same processing operation is performed on the 5 steering wheels. After the processing is completed, the hand and steering wheel with the highest confidence can be obtained, that is, the optimal hand and steering wheel. At the same time, the matching relationship between the optimal hand and steering wheel can be determined according to the direction vector between the optimal hand and steering wheel.

[0036] Among them, d=d12+d21. Specifically, the direction vector of the center point of the hand is added to the direction vector of the center point of the steering wheel to determine the position of the steering wheel, the deviation between the obtained position of the steering wheel and the actual position of the steering wheel is calculated, and the absolute value of the deviation is obtained to obtain d12; the direction vector of the center point of the steering wheel is added to the direction vector of the center point of the hand to determine the position of the hand, the deviation between the obtained position of the hand and the actual position of the hand is calculated, and the absolute value of the deviation is obtained to obtain d21. Among them, the position of the steering wheel corresponding to the center point corresponding to the direction vector with the highest confidence is the actual position of the steering wheel, and the actual position of the hand is the same, which will not be repeated.

[0037] In one embodiment, a matching network in a target neural network is used to perform regression analysis on the obtained multiple body center points and the direction vector corresponding to each body center point, the multiple center points of each hand and the direction vector corresponding to each center point, the multiple center points of the upper torso and the direction vector corresponding to each center point, the multiple center points of the steering wheel and the direction vector corresponding to each center point, and the multiple center points of each seat belt and the direction vector corresponding to each center point to obtain a unique body center point and its corresponding direction vector, a unique center point of each hand and its corresponding direction vector, a unique center point of each upper torso and its corresponding direction vector, a unique center point of each steering wheel and its corresponding direction vector, and a unique center point of each seat belt and its corresponding direction vector.

[0038] Step S104: Determine the distance center point of the target person's body action point corresponding to the action to be identified and the action object and its corresponding direction vector based on the center point of the unique key part and its corresponding direction vector and the center point of the unique object and its corresponding direction vector. When identifying the action state of each image, it is also necessary to determine the distance center point of the target person's body action point corresponding to the action to be identified and the action object and its corresponding direction vector. Specifically, it is determined based on the center point of the unique key part and its corresponding direction vector and the center point of the unique object and its corresponding direction vector.

[0039] In one embodiment, after determining the unique center point of the hand and its corresponding direction vector and the unique center point of the steering wheel and its corresponding direction vector, the midpoint between the unique center point of the hand and the unique center point of the steering wheel is the center point of the distance between the target person's body point of action and the object of action corresponding to the action to be identified, and the midpoint between the unique center point of the hand and its corresponding direction vector and the unique center point of the steering wheel and its corresponding direction vector is the direction vector corresponding to the center point of the distance between the target person's body point of action and the object of action corresponding to the action to be identified. When holding the steering wheel, the actions to be identified include: holding, placing above or below the steering wheel, moving away from or approaching the steering wheel; when wearing a seat belt, the actions to be identified include: wearing normally, not wearing, or wearing.

[0040] Step S105: Determine the first direction vector between the distance center point and the unique key part center point according to the distance center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector; determine the second direction vector between the distance center point and the unique object center point according to the distance center point and its corresponding direction vector and the unique object center point and its corresponding direction vector; determine the third direction vector between the unique body center point and the unique key part center point according to the unique body center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector.

[0041] Specifically, after obtaining the distance center point and its corresponding direction vector, the unique key part center point and its corresponding direction vector, the unique object center point and its corresponding direction vector, and the unique body center point and its corresponding direction vector in each image in the image training sample, it is necessary to determine the relationship between each unique center point and its corresponding direction vector and other unique center points and their corresponding direction vectors.

[0042] In one embodiment, when the hand is holding the steering wheel, the first direction vector between the distance center point and the unique center point of the hand can be determined according to the distance center point and its corresponding direction vector and the unique center point of the hand and its corresponding direction vector. Specifically, when the hand is holding the steering wheel, according to the action point of the holding action and the direction of the action and the center point of the hand and its corresponding direction vector, it can be determined whether the left hand or the right hand is currently holding the steering wheel, that is, the direction vector between the holding action and the hand is determined.

[0043] Step S106: According to the action state of the target person in each image and the first direction vector, the second direction vector, and the third direction vector, the preset neural network model is trained to obtain an action recognition model. Specifically, after determining the relationship between each unique center point and its corresponding direction vector and other unique center points and their corresponding direction vectors, that is, after obtaining the first direction vector, the second direction vector, and the third direction vector, the three direction vectors and the action state of the target person in each image are input into the preset neural network model for training.

[0044] Specifically, training is performed using the Adam optimizer, including: setting the initial learning rate to 5e-4, the single-card batch size to 64, and training for 200 epochs.

[0045] The method for constructing an action recognition model provided by an embodiment of the present invention obtains image training samples and inputs the image training samples into a target neural network; uses a detection branch network in the target neural network to process each image in the image training samples to obtain a corresponding first direction vector, a second direction vector, and a third direction vector; trains a preset neural network model according to the action state of the target person in each image and the first direction vector, the second direction vector, and the third direction vector to obtain an action recognition model. The method can achieve real-time performance through the target neural network; the action recognition model obtained by training the action state of each image and the corresponding direction vector does not require complex post-processing and can directly recognize fine-grained actions.

[0046] As an optional implementation of an embodiment of the present invention, when performing detection according to the detection branch network in the target neural network, multiple body center points, multiple key part center points, and multiple object center points will be obtained, and each center point corresponds to a direction vector. Therefore, it is necessary to perform maximum value suppression processing on these center points and direction vectors to obtain the most correct center point and direction vector.

[0047] Specifically, the detected multiple body center points, multiple key part center points and multiple object center points are used as multiple target center points to be identified, and the multiple target center points to be identified of any type are grouped, and the multiple target center points to be identified in each group after grouping are sorted from large to small according to the detection confidence, and then the distance between the direction vector corresponding to the target center point to be identified with the highest detection confidence and the direction vector corresponding to other target center points to be identified is calculated in each group, and the distance value is compared with a preset threshold value. When the distance value is less than the preset threshold value, the corresponding other center points and their corresponding direction vectors are suppressed to obtain the target center point to be identified with the highest detection confidence and its corresponding direction vector, and the target center point to be identified with the highest detection confidence and its corresponding direction vector are used as the only center point of this type and its corresponding direction vector. The suppression processing method includes: discarding the target center point to be identified and its corresponding direction vector that is less than the preset threshold value, and retaining the target center point to be identified and its corresponding direction vector that is greater than the preset threshold value.

[0048] Among them, the confidence is obtained when the detection branch outputs the key point position, and the value range is 0-1;

[0049] In one embodiment, on a two-dimensional graph, each position represents the coordinates of a key point, and the value at each position is the confidence level.

[0050] The preset threshold is generally 0.5, which can be changed according to the actual situation. For example, if the scene is dark and unclear, you can try to lower it. The design of the network structure will also have an impact. The FCOS confidence is generally low and can be set to 0.3. The present invention has a low dependence on the preset threshold. Finally, the maximum value of the direction vector is suppressed to obtain the only credible key point. Therefore, when setting the preset threshold, the present invention only needs to consider the balance between the recall rate and efficiency of the model, and does not further limit the specific value.

[0051] In one embodiment, assume that there are 10 hand points and corresponding 10 direction vectors, and 4 [holding] action points and corresponding 4 direction vectors. The hand has ten vectors pointing to the action, and the action has 4 vectors pointing to the hand. There are 10*4 possible permutations and combinations here. The hand points are divided into 10 groups. At this time, an action needs to be assigned to each hand point. However, there are not enough action points to assign, so suppression processing is performed: the vector distance in each group is calculated. When the two distances are close and less than a certain threshold, the point with higher confidence is selected, that is, the close combination is considered to be the same combination, and the group with higher confidence is selected as the final matching result. The final retained combination is less than or equal to 4.

[0052] As an optional implementation of an embodiment of the present invention, the target neural network includes: any one of VoVNet, ResNet, DenseNet, EfficientNet, ShuffleNet, and MobileNet.

[0053] In one embodiment, VoVNet is used as the backbone network, and the output of the network P3 stage is upsampled to obtain a feature map of 1 / 4 the size of the original image. The center point detection branch uses convolution with a 3x3 convolution kernel, followed by ReLU, and then convolution with a 1x1 convolution kernel. The stride of this process is 1, and finally FocalLoss is used to learn the Heatmap of the center point. The matching branch uses convolution with a 3x3 convolution kernel, followed by ReLU, and then convolution with a 1x1 convolution kernel, and finally smooth_l1_loss to learn the direction vector of each center point.

[0054] The embodiment of the present invention also provides a method for motion recognition, such as Figure 2 As shown, the method comprises the following steps:

[0055] Step S201: Acquire an image to be recognized.

[0056] Step S202: Input the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector. Specifically, after obtaining the image to be recognized, the image to be recognized is input into the target neural network for processing to obtain the first direction vector, the second direction vector, and the third direction vector. The specific implementation process refers to the processing process of the image to be recognized in the action recognition model construction method, which will not be repeated here.

[0057] Step S203: Input the image to be recognized and the first direction vector, the second direction vector and the third direction vector into the action recognition model obtained by the action recognition model construction method to obtain an action recognition result. Specifically, the recognition process of the image to be recognized can be completed using the trained action recognition model.

[0058] In one embodiment, if Figure 3 As shown, point d is the only body center point, point c is the only key part center point, point b is the only object center point, and point a is the distance center point. The first direction vector is The second direction vector is The third direction vector is After inputting the action recognition model for processing, the action recognition result of the target person can be obtained as [driver]'s [left hand] [holding] [steering wheel].

[0059] The action recognition method provided by the embodiment of the present invention obtains an image to be recognized; inputs the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; inputs the image to be recognized and the first direction vector, the second direction vector, and the third direction vector into the action recognition model obtained by the action recognition model construction method to process and obtain an action recognition result. This method performs action recognition based on a single frame image and is faster.

[0060] The embodiment of the present invention also provides a device for constructing an action recognition model. Figure 4 As shown, the device comprises:

[0061] The acquisition module 301 is used to acquire image training samples and input the image training samples into the target neural network; for details, please refer to the relevant description of step S101 in the above method embodiment.

[0062] The detection module 302 is used to use the detection branch network in the target neural network to respectively detect the entire body of the target person contained in each image in the image training sample, the key parts of the target person corresponding to the action to be identified, and the object to be identified, to obtain multiple body center points of the target person's entire body, multiple key part center points of the key parts of the target person corresponding to the action to be identified, and multiple object center points of the object to be identified, each body center point, key part center point and object center point corresponds to a direction vector; for details, see the relevant description of step S102 in the above method embodiment.

[0063] The matching module 303 is used to perform regression matching processing using the matching network in the target neural network based on the multiple body center points and the direction vectors corresponding to each body center point, the multiple key part center points and the direction vectors corresponding to each key part center point, and the multiple object center points and the direction vectors corresponding to each object center point, so as to obtain a unique body center point and its corresponding direction vector, a unique key part center point and its corresponding direction vector, and a unique object center point and its corresponding direction vector; for details, see the relevant description of step S103 in the above method embodiment.

[0064] The first determination module 304 is used to determine the distance between the target person's body action point corresponding to the action to be identified and the action object's action center point and its corresponding direction vector based on the unique key part center point and its corresponding direction vector and the unique object center point and its corresponding direction vector; for details, please refer to the relevant description of step S104 in the above method embodiment.

[0065] The second determination module 305 is used to determine the first direction vector between the distance center point and the unique key part center point according to the distance center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector, determine the second direction vector between the distance center point and the unique object center point according to the distance center point and its corresponding direction vector and the unique object center point and its corresponding direction vector, and determine the third direction vector between the unique body center point and the unique key part center point according to the unique body center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector; for details, please refer to the relevant description of step S105 in the above method embodiment.

[0066] The training module 306 is used to train the preset neural network model according to the action state of the target person in each image and the first direction vector, the second direction vector, and the third direction vector to obtain an action recognition model; for details, please refer to the relevant description of step S106 in the above method embodiment.

[0067] The action recognition model construction device provided by the embodiment of the present invention obtains image training samples and inputs the image training samples into the target neural network; uses the detection branch network in the target neural network to process each image in the image training samples to obtain the corresponding first direction vector, second direction vector, and third direction vector; trains the preset neural network model according to the action state of the target person in each image and the first direction vector, second direction vector, and third direction vector to obtain the action recognition model. Real-time performance can be achieved through the target neural network; the action recognition model obtained by training the action state of each image and the corresponding direction vector does not require complex post-processing and can directly recognize fine-grained actions.

[0068] As an optional implementation of the embodiment of the present invention, the device also includes: a third determination module, which is used to take the multiple body center points, the multiple key part center points and the multiple object center points as multiple target center points to be identified; a grouping module, which is used to group the multiple target center points to be identified for any type of multiple target center points to be identified; a sorting module, which is used to sort the multiple target center points to be identified corresponding to each group after grouping according to detection confidence; a calculation module, which is used to calculate the distance between the direction vector corresponding to the target center point to be identified with the highest detection confidence and the direction vectors corresponding to other target center points to be identified in each group; a comparison module, which is used to compare the distance with a preset threshold; a processing module, which is used to suppress other center points whose distances are less than the preset threshold and their corresponding direction vectors to obtain the target center point to be identified with the highest detection confidence and its corresponding direction vector, and use the target center point to be identified with the highest detection confidence and its corresponding direction vector as the unique center point of the type and its corresponding direction vector.

[0069] As an optional implementation of an embodiment of the present invention, the target neural network includes: any one of VoVNet, ResNet, DenseNet, EfficientNet, ShuffleNet, and MobileNet.

[0070] For a detailed description of the functions of the motion recognition model building device provided in the embodiment of the present invention, please refer to the description of the motion recognition model building method in the above embodiment.

[0071] The embodiment of the present invention also provides a motion recognition device, such as Figure 5 As shown, the device comprises:

[0072] The second acquisition module 401 is used to acquire the image to be recognized; for details, please refer to the relevant description of step S201 in the above method embodiment.

[0073] The input module 402 is used to input the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; for details, please refer to the relevant description of step S202 in the above method embodiment.

[0074] The recognition module 403 is used to input the image to be recognized and the first direction vector, the second direction vector, and the third direction vector into the action recognition model obtained by the action recognition model construction method for processing to obtain the action recognition result; for details, please refer to the relevant description of step S203 in the above method embodiment.

[0075] The action recognition device provided by the embodiment of the present invention obtains an image to be recognized; inputs the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; inputs the image to be recognized and the first direction vector, the second direction vector, and the third direction vector into the action recognition model obtained by the action recognition model construction method to process and obtain an action recognition result. Action recognition based on a single frame image is faster.

[0076] For a detailed description of the functions of the motion recognition device provided in the embodiment of the present invention, please refer to the description of the motion recognition method in the above embodiment.

[0077] The embodiment of the present invention also provides a storage medium, such as Figure 6 As shown, a computer program 601 is stored thereon, and when the instruction is executed by the processor, the steps of the action recognition model construction method or the action recognition method in the above-mentioned embodiment are implemented. The storage medium also stores audio and video stream data, feature frame data, interaction request signaling, encrypted data, and preset data size, etc. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0078] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0079] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, the electronic device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0080] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0081] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiments of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, the action recognition model construction method or the action recognition method in the above method embodiment is implemented.

[0082] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device and at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0083] The one or more modules are stored in the memory 52 and when executed by the processor 51, perform the following steps: Figure 1 The action recognition model construction method in the embodiment shown in the figure or Figure 2-3 The action recognition method in the illustrated embodiment.

[0084] For details of the above electronic equipment, please refer to Figures 1 to 3 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0085] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing an action recognition model, characterized in that: The steps include: Acquire image training samples and input the image training samples into a target neural network; Utilizing the detection branch network in the target neural network, the entire body of the target person, the key parts of the target person corresponding to the action to be identified, and the object to be identified contained in each image of the image training sample are detected to obtain multiple body center points of the entire body of the target person, multiple key part center points of the key parts of the target person corresponding to the action to be identified, and multiple object center points of the object to be identified, each body center point, key part center point, and object center point corresponding to a direction vector; According to the multiple body center points and the direction vectors corresponding to each body center point, the multiple key part center points and the direction vectors corresponding to each key part center point, and the multiple object center points and the direction vectors corresponding to each object center point, a matching network in the target neural network is used to perform regression matching processing to obtain a unique body center point and its corresponding direction vector, a unique key part center point and its corresponding direction vector, and a unique object center point and its corresponding direction vector; Determine the distance between the target person's body action point corresponding to the action to be identified and the action object's action point and its corresponding direction vector according to the unique key part center point and its corresponding direction vector and the unique object center point and its corresponding direction vector; Determine the first direction vector between the distance center point and the unique key part center point according to the distance center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector, determine the second direction vector between the distance center point and the unique object center point according to the distance center point and its corresponding direction vector and the unique object center point and its corresponding direction vector, and determine the third direction vector between the unique body center point and the unique key part center point according to the unique body center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector; The preset neural network model is trained according to the action state of the target person in each image and the first direction vector, the second direction vector, and the third direction vector to obtain an action recognition model.

2. The method according to claim 1, characterized in that The method further comprises: Taking the multiple body center points, the multiple key part center points and the multiple object center points as multiple target center points to be identified; For any type of multiple target center points to be identified, grouping the multiple target center points to be identified; Sorting the plurality of target center points to be identified corresponding to each group after grouping according to the detection confidence; Calculating the distance between the direction vector corresponding to the center point of the target to be identified with the highest detection confidence and the direction vectors corresponding to the center points of other targets to be identified in each group; comparing the distance with a preset threshold; Suppression processing is performed on other center points whose distances are less than the preset threshold and their corresponding direction vectors to obtain the center point of the target to be identified with the highest detection confidence and its corresponding direction vector, and the center point of the target to be identified with the highest detection confidence and its corresponding direction vector are used as the only center point of the type and its corresponding direction vector.

3. The method according to claim 1, characterized in that The target neural network includes: any one of VoVNet, ResNet, DenseNet, EfficientNet, ShuffleNet, and MobileNet.

4. A method for motion recognition, characterized in that: The steps include: Obtain an image to be recognized; Inputting the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; The image to be recognized and the first direction vector, the second direction vector and the third direction vector are input into the action recognition model obtained by the action recognition model construction method according to any one of claims 1 to 3 for processing to obtain an action recognition result.

5. A motion recognition model construction device, characterized in that: include: An acquisition module, used for acquiring image training samples and inputting the image training samples into a target neural network; A detection module is used to use the detection branch network in the target neural network to detect the entire body of the target person, the key parts of the target person corresponding to the action to be identified, and the object to be identified contained in each image of the image training sample, to obtain multiple body center points of the entire body of the target person, multiple key part center points of the key parts of the target person corresponding to the action to be identified, and multiple object center points of the object to be identified, each body center point, key part center point, and object center point corresponding to a direction vector; A matching module, for performing regression matching processing using a matching network in the target neural network based on the multiple body center points and the direction vectors corresponding to each body center point, the multiple key part center points and the direction vectors corresponding to each key part center point, and the multiple object center points and the direction vectors corresponding to each object center point, to obtain a unique body center point and its corresponding direction vector, a unique key part center point and its corresponding direction vector, and a unique object center point and its corresponding direction vector; The first determination module is used to determine the distance center point and the corresponding direction vector of the target person's body action point corresponding to the action to be identified from the action object based on the center point of the unique key part and the corresponding direction vector and the center point of the unique object and the corresponding direction vector; A second determination module is used to determine a first direction vector between the distance center point and the unique key part center point according to the distance center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector, determine a second direction vector between the distance center point and the unique object center point according to the distance center point and its corresponding direction vector and the unique object center point and its corresponding direction vector, and determine a third direction vector between the unique body center point and the unique key part center point according to the unique body center point and its corresponding direction vector and the unique key part center point and its corresponding direction vector; The training module is used to train the preset neural network model according to the action state of the target person in each image and the first direction vector, the second direction vector, and the third direction vector to obtain an action recognition model.

6. The device according to claim 5, characterized in that The device also includes: A third determination module is used to use the multiple body center points, the multiple key part center points and the multiple object center points as multiple target center points to be identified; A grouping module, used for grouping the multiple target center points to be identified of any type; A sorting module, used for sorting the plurality of target center points to be identified corresponding to each group after grouping according to the detection confidence; A calculation module, used for calculating the distance between the direction vector corresponding to the center point of the target to be identified with the highest detection confidence and the direction vectors corresponding to the center points of other targets to be identified in each group; A comparison module, used for comparing the distance with a preset threshold; A processing module is used to suppress other center points whose distances are less than the preset threshold and their corresponding direction vectors to obtain the center point of the target to be identified with the highest detection confidence and its corresponding direction vector, and use the center point of the target to be identified with the highest detection confidence and its corresponding direction vector as the only center point of the type and its corresponding direction vector.

7. The device according to claim 5, characterized in that The target neural network includes: any one of VoVNet, ResNet, DenseNet, EfficientNet, ShuffleNet, and MobileNet.

8. A motion recognition device, characterized in that: include: A second acquisition module is used to acquire an image to be identified; An input module, used for inputting the image to be recognized into the target neural network to obtain the first direction vector, the second direction vector, and the third direction vector; The recognition module is used to input the image to be recognized and the first direction vector, the second direction vector and the third direction vector into the action recognition model obtained by the action recognition model construction method according to any one of claims 1 to 3 for processing to obtain an action recognition result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the action recognition model construction method according to any one of claims 1 to 3, or the action recognition method according to claim 4.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the action recognition model construction method as described in any one of claims 1 to 3, or the action recognition method as described in claim 4, by executing the computer instructions.

Images