Gesture recognition method, device, processor and electronic device

Abstract

The present application discloses a posture recognition method, device, processor and electronic device. It relates to the field of artificial intelligence, and the method includes: collecting a target image; detecting the posture parameters of a target object included in the target image; and determining the posture of the target object in the target image according to the posture parameters. Through the present application, the problem that the abnormal behavior monitoring of tellers at closed counters in the related art is mainly to collect data through cameras, and then randomly select and check. When the tellers are abnormal, it is difficult to detect them in a timely and effective manner, resulting in poor timeliness of monitoring.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a gesture recognition method, device, processor, and electronic device. Background Art

[0002] With the rapid development of science and technology, online transactions have become the best choice for many customers. However, there are still many customers who conduct financial transactions in banks, especially the elderly customers who need the assistance of tellers. If there are abnormal characteristics in special groups, including physical discomfort, heart disease, fainting and falling, or customers or tellers who suddenly have abnormal physical conditions, they may not be able to get timely help, resulting in worsening of their condition or missing the best time for rescue. At the same time, the current monitoring of abnormal behavior of tellers at closed counters is mainly through data collection by cameras, and random inspections are carried out later.

[0003] To prevent abnormal situations, most outlets adopt teller training and EAD and other hardware equipment, but they still cannot predict the occurrence of abnormal situations in advance, react slowly, and fail to grasp the best rescue opportunity. At the same time, monitoring the abnormal behavior of tellers also has the disadvantages of being time-consuming, labor-intensive, costly, inefficient, and prone to omissions.

[0004] In the related technology, the abnormal behavior monitoring of tellers at closed counters is mainly carried out by collecting data through cameras and then randomly checking. When the tellers behave abnormally, it is difficult to detect them in a timely and effective manner, resulting in poor timeliness of monitoring. No effective solution has been proposed so far. Summary of the invention

[0005] The main purpose of this application is to provide a posture recognition method, device, processor and electronic device to solve the problem that in the related art, the abnormal behavior monitoring of tellers at closed counters is mainly carried out by collecting data through cameras and then randomly checking. When the tellers show abnormalities, it is difficult to detect them in a timely and effective manner, resulting in poor timeliness of monitoring.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a posture recognition method is provided, comprising: acquiring a target image; detecting posture parameters of a target object included in the target image; and determining the posture of the target object in the target image according to the posture parameters.

[0007] Optionally, before detecting the posture parameters of the target object included in the target image, the method also includes: obtaining a first training sample, wherein the first training sample includes a first image of the sample object and standard posture parameters of the sample object; using the first training sample to perform machine training to obtain a detection model, wherein the detection model is a mask region convolutional neural network model Mask RCNN.

[0008] Optionally, obtaining the first training sample includes: collecting multiple images of the sample object in the target environment; amplifying the collected images to obtain multiple test images; and labeling the sample objects in the multiple test images according to the standard posture parameters to obtain the first training sample.

[0009] Optionally, detecting the posture parameters of the target object included in the target image includes: inputting the target image into the detection model; determining, through the detection model, the region of interest of the target image, the category of the region of interest, the mask of the target object region in the target image, and the coordinates of the key points of the target object.

[0010] Optionally, determining the region of interest of the target image through the feature extraction network and region proposal network of the detection model includes: extracting features from the target image to obtain a corresponding feature map; processing the feature map to obtain multiple candidate regions; and selecting a candidate region with the highest score from the multiple candidate regions as the region of interest.

[0011] Optionally, using the first training sample to perform machine training to obtain a detection model includes: inputting the first training sample into the detection model to obtain the region of interest of the first training sample, the category of the region of interest, the mask and key point coordinates of the sample object region; determining the loss value of the detection model based on the region of interest, the category, the mask and the key point coordinates and the standard posture parameters; and determining that the detection model training is successful when the loss value does not exceed a preset value.

[0012] Optionally, the standard posture parameters include: the category, center coordinates and size of the region of interest in the first training sample, the mask of the sample object in the first training sample and the coordinates of the key points. Determining the loss value of the detection model based on the region of interest, the category, the mask and the key point coordinates and the standard posture parameters includes: determining the classification error based on the category of the region of interest and the category corresponding to the region of interest in the first training sample; determining the coordinate error based on the center coordinates and size of the region of interest and the center coordinates and size of the region of interest in the first training sample; determining the segmentation error based on the intersection-and-union ratio of the mask and the classification error, wherein the intersection-and-union ratio is the intersection-and-union ratio of the mask and the mask of the sample object in the first training sample; determining the key point error based on the Euclidean distance between the predicted coordinates of multiple key points and the coordinates of the key points of the sample object in the first training sample; determining the loss value based on the classification error, the coordinate error, the segmentation error and the key point error, and their respective corresponding preset weights.

[0013] Optionally, before determining the posture of the target object in the target image according to the posture parameters, the method also includes: obtaining a second training sample, wherein the second training sample includes a second image marked with standard posture parameters of the sample object, and the posture of the sample object; performing machine training through the second training sample to obtain a classification model, wherein the classification model includes but is not limited to: a decision tree classification model, a binary classification model.

[0014] Optionally, determining the posture of the target object in the target image according to the posture parameters includes: annotating the target image according to the posture parameters; and inputting the annotated target image into the classification model to obtain the posture corresponding to the target object in the target image.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a posture recognition device is provided, including: an acquisition module for acquiring a target image; a detection module for detecting posture parameters of a target object included in the target image; and a recognition module for determining the posture of the target object in the target image according to the posture parameters.

[0016] In order to achieve the above objective, according to another aspect of the present application, a processor is provided, wherein the processor is used to run a program, wherein the program executes any one of the gesture recognition methods described above when running.

[0017] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the gesture recognition methods described above.

[0018] Through this application, the following steps are adopted: collecting a target image; detecting the posture parameters of the target object included in the target image; and determining the posture of the target object in the target image according to the posture parameters. By detecting the posture parameters of the target object in the target image, it is determined whether the target image has a target object. If there is a target object, the posture parameters of the target object are detected, and the posture of the target object is identified according to the posture parameters. This solves the problem in the related art that the abnormal behavior monitoring of tellers at closed counters is mainly through collecting data through cameras, and then randomly sampling and checking. When the teller is abnormal, it is difficult to detect it in a timely and effective manner, resulting in poor timeliness of monitoring. Thus, the effect of timely and accurate identification of the teller's posture is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a flowchart of a gesture recognition method provided according to an embodiment of the present application;

[0021] Figure 2 is a flowchart of a method for identifying abnormal posture provided in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a human body detection model structure provided according to an embodiment of the present application;

[0023] Figure 4 is a flow chart of a method for identifying abnormalities of network points provided according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a gesture recognition device provided according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0030] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flowchart of a gesture recognition method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0031] Step S101, collecting a target image;

[0032] Step S102, detecting the posture parameters of the target object included in the target image;

[0033] Step S103: determining the posture of the target object in the target image according to the posture parameters.

[0034] The execution subject of the above steps can be a processor or a controller. Through the above steps, the target image is collected; the posture parameters of the target object included in the target image are detected; and the posture of the target object in the target image is determined according to the posture parameters. By detecting the posture parameters of the target object in the target image, it is determined whether the target image has the target object. In the case of the target object, the posture parameters of the target object are detected, and the posture of the target object is identified according to the posture parameters. This solves the problem that the abnormal behavior monitoring of tellers at closed counters in the related technology is mainly through the collection of data by cameras, and random sampling and inspection are carried out later. When the teller is abnormal, it is difficult to detect it in time and effectively, resulting in poor timeliness of monitoring. Thus, the effect of timely and accurate identification of the teller's posture is achieved.

[0035] The target object may be a specific monitored operator or worker, and the target image may be acquired through an image acquisition device, such as a camera. The image acquisition device may be set at a fixed position to acquire an image of the target object in the environment. Therefore, the target image may be an image of the environment, which may be a specific physical space range, or a working range of the target object, such as a teller's work station.

[0036] The posture parameters of the target object included in the above-mentioned detection target image can be detected by a detection model, and the above-mentioned detection model can be a mask region convolutional neural network model Mask RCNN. The detection model of this embodiment can adopt an optimized Mask RCNN, and the specific optimization method includes training optimization. The optimized MaskRCNN has a higher target object detection accuracy and more stable detection performance.

[0037] The above detection model detects the posture parameters of the target object in the target image. When the posture parameters of the target object in the target image are detected, it can be determined that the target image includes the target object. If the posture parameters of the target object in the target image cannot be detected, it can be considered that the target image does not include the target object, and the subsequent process will not be triggered. The target image can be re-captured to continue detection until the target image includes the target object, and then the subsequent process is executed. The above re-capture of the target image can be re-captured at a preset frequency or a preset time point, rather than re-captured immediately, which can avoid the problem of a low probability of identifying the target object and an increased processing burden due to too fast a capture frequency.

[0038] It should be noted that the above detection model determines whether there is a target object in the target image based on whether the posture parameters of the target object in the target image are detected. The above detection model can also be a machine learning model including a convolutional neural network, which is trained by multiple groups of first training samples, each group of first training samples, including a first image of a sample object, and standard posture parameters of the sample object. The above sample object is also the object in the sample, corresponding to the above target object. For example, if the target object is a person, the sample object in the sample should also be a person. Specifically, the detection model outputs a detection result, that is, the posture parameters of the target object in the target image are detected, indicating that the target image includes the target object. The detection model does not output a result, that is, the posture parameters of the target object in the target image cannot be detected, indicating that the target image does not include the target object. The above posture parameters include the area of ​​interest where the target object exists, the classification of the area of ​​interest, the mask of the target object, the coordinates of key points, etc.

[0039] Through the above-mentioned posture parameters, the classification model can be used to identify and classify the posture of the target object. It should be noted that the above-mentioned classification model can also be a machine learning model including a convolutional neural network, for example, a decision tree classification model, which is trained by multiple groups of second training samples, each group of second training samples including a second image annotated with standard posture parameters of the sample object, and the posture of the sample object. On the other hand, considering that the posture recognition result can be normal or abnormal, and can also be a limited category such as a certain posture or not a certain posture, the above-mentioned prediction model can also be a binary classification model, etc.

[0040] The target object detection model is used to detect whether the target image has a target object, and if the target object exists, the target object's posture parameters are detected. The posture parameters are used to identify the target object's posture, or whether the posture is abnormal. This solves the problem that the abnormal behavior monitoring of tellers at closed counters in related technologies is mainly based on collecting data through cameras, and then randomly sampling and checking. When the tellers are abnormal, it is difficult to find them in a timely and effective manner, resulting in poor timeliness of monitoring. This achieves the effect of timely and accurate identification of the teller's posture.

[0041] Optionally, before detecting the posture parameters of the target object included in the target image, the method also includes: obtaining a first training sample, wherein the first training sample includes a first image of the sample object and standard posture parameters of the sample object; using the first training sample to perform machine training to obtain a detection model, wherein the detection model is a mask region convolutional neural network model Mask RCNN.

[0042] When training the above detection model, the image samples of the target object's environment can be used for training, that is, the above first image can be an image of the sample object in the above environment, which can improve the recognition accuracy of the detection model for the image of the target object's environment. For example, an image of a bank teller at a closed workstation. Through training with the image of the workstation, the detection model has better detection performance for whether there is a teller at the workstation, and can provide more accurate detection results when there is a teller.

[0043] The above-mentioned standard posture parameters may be standard posture parameters corresponding to the posture parameters of the target object. For example, the above-mentioned posture parameters may include at least one of the following: the region of interest, the category of the region of interest, the mask of the target object region, and the coordinates of the key points of the target object. Then the standard posture parameters may correspond to at least one of the following: the real region of interest, the real category of the real region of interest, the real mask of the target object region, and the real coordinates of the key points of the target object, thereby forming a comparison with the detection results during the training of the detection model to correct and train the detection model.

[0044] Optionally, obtaining the first training sample includes: collecting multiple images of the sample object in the target environment; amplifying the collected images to obtain multiple test images; and labeling the sample objects in the multiple test images according to the standard posture parameters to obtain the first training sample.

[0045] The above-mentioned augmentation processing includes at least one of the following: flip processing, noise adding processing, random cropping processing, image scaling processing. The above-mentioned collected images include images of various postures. Considering increasing the diversity of samples, the collected images can be flipped, noise added, randomly cropped, image scaled, etc. to obtain new image samples. In this way, the diversity of samples is improved, and the accuracy and efficiency of detection model training are indirectly improved.

[0046] The target object of the test image is labeled according to the standard posture parameters to obtain the first training sample, which can be labeled according to the specific parameters of the standard posture parameters, including labeling the region of interest, the category of the region of interest, the real mask of the target object region, and the real coordinates of the key points of the target object.

[0047] Optionally, detecting the posture parameters of the target object included in the target image includes: inputting the target image into a detection model; determining the region of interest of the target image, the category of the region of interest, the mask of the target object region in the target image, and the coordinates of the key points of the target object through the detection model.

[0048] Determining the region of interest of a target image, the category of the region of interest, the mask of a target object region in the target image, and the coordinates of key points of the target object through a detection model may include: determining the region of interest of a first training sample through a feature extraction network and a region proposal network of the detection model; determining the category of the region of interest through a fully connected layer of the detection model, wherein the category is a category that includes the target object and a category that does not include the target object; determining the mask of the target object region in the first training sample and the coordinates of key points of the target object through a convolutional neural network of the detection model.

[0049] That is, the above detection model includes a feature extraction network and a region proposal network, and may also include a fully connected layer, and may also include a convolutional neural network. Of course, when the above detection model is MaskRCNN, the above feature extraction network and region proposal network, fully connected layer and convolutional neural network are all available. It can also be determined that the detection result output by the detection model includes the region of interest, the category of the region of interest, the mask of the target object region, and the key point coordinates of the target object. The key point can be multiple, including the top of the head, chin, left and right shoulders, left and right elbows, left and right fingers, left and right hips, left and right knees, left and right feet and other key points of each posture. In this embodiment, the number of key points is selected as 14, that is, the 14 key points of the top of the head, chin, left and right shoulders, left and right elbows, left and right fingers, left and right hips, left and right knees, and left and right feet.

[0050] Optionally, determining the region of interest of the target image through the feature extraction network and region proposal network of the detection model includes: extracting features from the target image to obtain a corresponding feature map; processing the feature map to obtain multiple candidate regions; and selecting a candidate region with the highest score from the multiple candidate regions as the region of interest.

[0051] The feature extraction network is used to extract features from the first training sample to obtain a corresponding feature map. The target image can be scaled to obtain images of multiple sizes, which enter the feature extraction network respectively. In this embodiment, ResNet101 is selected to generate a feature map with higher-order features. The feature maps of multiple sizes are passed through the region proposal network RPN to generate multiple candidate regions.

[0052] The above suppression algorithm may be a non-maximum suppression algorithm, which scores the above multiple candidate regions and selects the candidate region with the highest score as the above region of interest. The principle of the above maximum suppression algorithm is equivalent to screening multiple candidate regions by size and selecting a candidate region with the most appropriate size that is closest to the size of the circumscribed rectangle of a normal target object. It should be noted that the above candidate regions are all rectangles.

[0053] Optionally, the first training sample is used for machine training to obtain a detection model, including: inputting the first training sample into the detection model to obtain the region of interest of the first training sample, the category of the region of interest, the mask and key point coordinates of the sample object area; determining the loss value of the detection model according to the region of interest, category, mask and key point coordinates and standard posture parameters; when the loss value does not exceed a preset value, determining that the detection model training is successful.

[0054] When the detection model is trained, the first image of the first training sample is processed in the same manner as the target image. Specifically, the first training sample is input into the detection model to obtain the region of interest, the category of the region of interest, the mask of the sample object region and the coordinates of the key points of the first training sample, including: determining the region of interest of the first image, the category of the region of interest, the mask of the sample object region in the first image, and the coordinates of the key points of the sample object through the detection model, which may include: determining the region of interest of the first image through the feature extraction network and the region proposal network of the detection model; determining the category of the region of interest through the fully connected layer of the detection model, wherein the category is a category containing the sample object and a category not containing the sample object; determining the mask of the sample object region in the first image and the coordinates of the key points of the sample object through the convolutional neural network of the detection model.

[0055] Then, according to the region of interest, category, mask and key point coordinates of the first image and the standard posture parameters of the first training sample, the loss value of the detection model is determined; when the loss value does not exceed a preset value, it is determined that the detection model training is successful.

[0056] Optionally, the standard posture parameters include: the category, center coordinates and size of the region of interest in the first training sample, the mask of the sample object in the first training sample and the coordinates of the key points. According to the region of interest, category, mask and key point coordinates, and the standard posture parameters, determining the loss value of the detection model includes: determining the classification error according to the category of the region of interest and the category of the corresponding region of interest in the first training sample; determining the coordinate error according to the center coordinates and size of the region of interest and the center coordinates and size of the region of interest in the first training sample; determining the segmentation error according to the intersection-and-union ratio of the mask and the classification error, wherein the intersection-and-union ratio is the intersection-and-union ratio of the mask and the mask of the sample object in the first training sample; determining the key point error according to the Euclidean distance between the predicted coordinates of multiple key points and the coordinates of the key points of the sample object in the first training sample; determining the loss value according to the classification error, coordinate error, segmentation error and key point error, and their respective corresponding preset weights.

[0057] According to the classification error, coordinate error, segmentation error and key point error, and their corresponding preset weights, the loss value is determined, that is, the weighted sum of the classification error, coordinate error, segmentation error and key point error is used as the final loss value, as shown in Formula 1, α, β, γ are the preset weights corresponding to the coordinate error, segmentation error and key point error respectively. Where L class Represents the classification error. The classification error is determined based on the category of the region of interest and the true category of the corresponding region of interest in the first training sample, as shown in Formula 2. i represents the probability that the region of interest is predicted to contain the target object category, Indicates whether it is a real table object; L bbox is the coordinate error, and the coordinate error is determined according to the predicted coordinates of multiple key points and the true coordinates in the first training sample, as shown in Formula 3, where (x i ,y i ),(w i ,h i ) represent the center coordinates and width and height of the i-th region of interest, respectively. are the coordinates and actual width and height of the corresponding real area of ​​interest respectively; L mask is the segmentation error, and the segmentation error is determined according to the IoU of the mask and the classification error, as shown in Formula 4, where the IoU of the Mask is defined as Formula 5, which is the IoU of the mask and the true mask of the target object in the first training sample; L kpsThe key point error is determined according to the Euclidean distance between the predicted coordinates and the true coordinates of multiple key points, as shown in Formula 6, which uses the Euclidean distance between the true key point coordinates and the predicted key point coordinates as the key point error function.

[0058] Loss = L class +αL bbox +βL mask +γL kps Formula 1

[0059]

[0060]

[0061] L mask =L clas * s IoU mas Formula 4

[0062]

[0063]

[0064] Optionally, before determining the posture of the target object in the target image according to the posture parameters, the method also includes: obtaining a second training sample, wherein the second training sample includes a second image marked with standard posture parameters of the sample object, and the posture of the sample object; performing machine training through the second training sample to obtain a classification model, wherein the classification model includes but is not limited to: a decision tree classification model, a binary classification model.

[0065] The above classification model may also be a machine learning model including a convolutional neural network, for example, a decision tree classification model, which is trained by a plurality of groups of second training samples, each group of second training samples including a second image annotated with standard posture parameters of the sample object, and the posture of the sample object. On the other hand, considering that the posture recognition result may be normal or abnormal, or may be a limited category such as a certain posture or not a certain posture, the above prediction model may also be a binary classification model, etc.

[0066] Optionally, determining the posture of the target object in the target image according to the posture parameters includes: annotating the target image according to the posture parameters; and inputting the annotated target image into a classification model to obtain the posture corresponding to the target object in the target image.

[0067] Thus, in the case of a target object, the posture parameters of the target object are determined, the target image is annotated by the posture parameters, and the annotated target image is input into the classification model, so as to identify the posture of the target object according to the posture parameters, and determine whether the posture is abnormal according to the identified posture. For example, if the identified posture is a normal sitting posture, it is a normal posture, and if the identified posture is an abnormal lying on the ground, it is an abnormal posture. This solves the problem in the related art that the abnormal behavior monitoring of tellers at closed counters is mainly through data collection by cameras, and random sampling and inspection are carried out later. When the tellers are abnormal, it is difficult to detect them in a timely and effective manner, resulting in poor timeliness of monitoring. Thus, the effect of timely and accurate identification of the teller's posture is achieved.

[0068] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] This embodiment also provides an optional implementation mode, which is described in detail below.

[0070] This embodiment aims at the defects and deficiencies in the prior art. It provides a device and method for automatically identifying abnormal conditions based on posture estimation. Through the real-time target image of the camera device, each frame of the image is analyzed, the human body is detected, and it can be predicted whether the branch customer or teller has an abnormal condition. At the same time, based on the original algorithm, according to the characteristics of this scenario, the algorithm is improved in a targeted manner to improve the accuracy and robustness of the prediction.

[0071] The present embodiment provides an apparatus and method for automatically identifying abnormal conditions, including a background server, a processor, and a surveillance camera connected to the background server. The surveillance camera is fully installed inside the outlet, and a camera is installed above each position of the closed counter.

[0072] The backend server stores the human body detection model, which is the target object detection model mentioned above, and can detect customers and tellers in the outlets.

[0073] The backend server also stores a posture recognition model, that is, the posture recognition model mentioned above, which can predict the posture state of the detected human body. The human body detection model and the posture recognition model can be prediction models obtained through neural network training.

[0074] Another purpose of this embodiment is achieved through the following technical solutions:

[0075] A method for automatically identifying abnormal conditions, Figure 2is a flowchart of a method for identifying abnormal posture according to an embodiment of the present application. Figure 2 As shown, the following steps are included:

[0076] Step 1: The surveillance cameras installed in the outlets monitor the outlets in real time, and the collected target images are transmitted to the backend server, and the target images are detected by the human body detection model; the above-mentioned human body detection model is also the above-mentioned detection model.

[0077] Step 2: If the presence of a human body is detected in step 1, the human body is the target object, and the posture of the human body is estimated and predicted through the posture recognition model. If the prediction result is abnormal, the branch staff will be informed immediately through voice, information, etc. The posture recognition model is also the classification model.

[0078] The human body detection model stored in the backend server is a model obtained by performing image acquisition and processing in the early stage and training through a convolutional neural network, such as Figure 3 As shown, Figure 3 It is a schematic diagram of the human body detection model structure provided according to the implementation mode of the present application, and the posture recognition model stored in the background server.

[0079] Specifically, the early image data collection and processing and model training process of the human body detection model is as follows, combined with infrared detection to improve the accuracy of detection:

[0080] Step 11: Collect human body images, which are equivalent to the first image of the first training sample mentioned above, including various images with different standing postures, sitting postures, walking postures, curling up, etc. The collected data are amplified, including horizontal flipping, adding noise, random cropping, image scaling, etc., and the test sample library is increased to improve the accuracy of training; key points of the collected human body images are annotated, including 14 key points such as the top of the head, chin, left and right shoulders, left and right elbows, left and right fingers, left and right hips, left and right knees, left and right feet, and the corresponding key point coordinates are stored in order.

[0081] Step 12: Establish a human detection model. Specifically, target detection algorithms such as SSD (Single Shot MultiBox Detector), Faster RCNN (Fast Regional Convolutional Neural Network, RCNN), and Yolo (You Only Look Once) can be used for construction. The above algorithms are well-known technologies that are widely studied and applied, and will not be repeated here. Since the human detection model needs to ensure the recall rate (the ratio of the number of correct predictions to the total number of correct predictions), algorithms with high human detection accuracy such as Faster RCNN can be selected here. In order to improve the accuracy and robustness of human detection, this implementation is optimized based on the original Faster RCNN algorithm. The network structure diagram is shown in the figure. Figure 3 As shown in the figure, the network mainly consists of two stages. The first stage uses the Region Proposal Net (RPN) to generate candidate bounding boxes where objects may exist. The second stage uses the regional feature aggregation method (RoIAlign) to align each candidate box generated in the first stage, and then performs category prediction, mask prediction, bounding box regression, and key point prediction.

[0082] Step A: The input image is scaled and enters the feature extraction network. ResNet101 is selected here to generate feature maps with higher-order features. Feature maps of multiple sizes are passed through the region proposal network RPN to generate multiple candidate regions.

[0083] Step B: For the multiple candidate regions obtained in step A, the non-maximum suppression method is used to eliminate redundant candidate regions, and the candidate regions with the highest scores are obtained as the regions of interest. These candidate regions are aligned by RoIAlign and normalized on each branch to obtain feature maps of the same size. Then, the fully connected layer is used to predict the category of each region of interest RoI and the corresponding candidate region.

[0084] Step C: When predicting the key points of the human body and the mask area of ​​the human body, it is necessary to rely on the bounding box coordinates of the target to eliminate the instance segmentation error caused by coordinate deviation. In order to reduce this dependence, this embodiment independently predicts the human body area Mask and key points. Since the Mask category prediction must be consistent with the prediction result of the category to which the RoI belongs, the category information in step B can be directly referenced here. The candidate area and category obtained in step B are also part of the model regression. For the regression of the Mask prediction, a new evaluation indicator is set here. The mask overlap IoUmask measures the prediction quality of the Mask, which represents the intersection-union ratio of the real mask and the predicted mask.

[0085] Step D: The key point detection is similar to step C. The final predicted key points are obtained through the same convolutional neural network, and the prediction results are used as part of the model regression step.

[0086] Step E: Determination of loss function. For the network structure of this embodiment, an independent loss function is involved. The loss value is determined according to the classification error, coordinate error, segmentation error and key point error, and their corresponding preset weights. That is, the weighted sum of the classification error, coordinate error, segmentation error and key point error is used as the final loss value, as shown in Formula 1, where α, β, and γ are the preset weights corresponding to the coordinate error, segmentation error and key point error, respectively. Where L class Represents the classification error. The classification error is determined based on the category of the region of interest and the true category of the corresponding region of interest in the first training sample, as shown in Formula 2. i represents the probability that the region of interest is predicted to contain the target object category, Indicates whether it is a real table object; L bbox is the coordinate error, and the coordinate error is determined according to the predicted coordinates of multiple key points and the true coordinates in the first training sample, as shown in Formula 3, where (x i ,y i ),(w i ,h i ) represent the center coordinates and width and height of the i-th region of interest, respectively. are the coordinates and actual width and height of the corresponding real area of ​​interest respectively; L mask is the segmentation error, and the segmentation error is determined according to the IoU of the mask and the classification error, as shown in Formula 4, where the IoU of the Mask is defined as Formula 5, which is the IoU of the mask and the true mask of the target object in the first training sample; L kps The key point error is determined according to the Euclidean distance between the predicted coordinates and the true coordinates of multiple key points, as shown in Formula 6, which uses the Euclidean distance between the true key point coordinates and the predicted key point coordinates as the key point error function.

[0087] Loss = L class +αL bbox +βL mask +γL kps Formula 1

[0088]

[0089]

[0090] L mask =L clas * s IoU mas Formula 4

[0091]

[0092]

[0093] Figure 4 is a flow chart of a method for identifying abnormal network points according to an embodiment of the present application, such as Figure 4 Specifically, the steps of image data collection and processing and model training for the posture recognition model are as follows:

[0094] Step 21: Posture recognition model training. The posture recognition model is also the above-mentioned classification model. The collected human body images are classified into normal and abnormal states, and 14 key points are marked on the images, which are stored in order in coordinate format. At this time, a simple decision tree algorithm can be used to train a binary classification model, or other classification algorithms can be used.

[0095] Step 22: Posture classification. The corresponding posture recognition model trained in step 21 extracts and predicts key points of the human body image detected in step 1, and outputs the corresponding posture category.

[0096] This implementation can predict abnormal physical conditions of customers, tellers, etc. in advance, buy more rescue time, and supervise tellers with abnormal behavior to reduce costs and improve efficiency. Targeted improvements are made to existing algorithms to improve the accuracy and robustness of the algorithms.

[0097] The embodiment of the present application also provides a gesture recognition device. It should be noted that the gesture recognition device of the embodiment of the present application can be used to execute the gesture recognition method provided by the embodiment of the present application. The gesture recognition device provided by the embodiment of the present application is introduced below.

[0098] Figure 5 is a schematic diagram of a gesture recognition device according to an embodiment of the present application. Figure 5 As shown, the device includes: a collection module 52, a detection module 54 and a recognition module 56. The device is described in detail below.

[0099] The acquisition module 52 is used to acquire the target image; the detection module 54 is connected to the acquisition module 52 and is used to detect the posture parameters of the target object included in the target image; the recognition module 56 is connected to the detection module 54 and is used to determine the posture of the target object in the target image according to the posture parameters.

[0100] Through the above device, the acquisition module 52 acquires the target image; the detection module 54 detects the posture parameters of the target object included in the target image; the identification module 56 determines the posture of the target object in the target image according to the posture parameters. By detecting the posture parameters of the target object in the target image, it is determined whether the target image has the target object. In the case of the target object, the posture parameters of the target object are detected, and the posture of the target object is identified according to the posture parameters. This solves the problem that the abnormal behavior monitoring of tellers at closed counters in the related art is mainly through the collection of data by cameras, and random sampling and inspection are carried out later. When the teller is abnormal, it is difficult to find it in time and effectively, resulting in poor timeliness of monitoring. Thus, the effect of timely and accurate identification of the teller's posture is achieved.

[0101] The gesture recognition device includes a processor and a memory. The acquisition module 52, the detection module 54 and the recognition module 56 are stored in the memory as program units. The processor executes the program units stored in the memory to realize corresponding functions.

[0102] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to detect whether the target image has a target object through the target object detection model, and if the target object has a target object, identify whether the posture of the target object is abnormal.

[0103] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0104] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the gesture recognition method is implemented.

[0105] An embodiment of the present invention provides a processor, which is used to run a program, wherein the gesture recognition method is executed when the program is running.

[0106] like Figure 6 As shown, an embodiment of the present invention provides an electronic device, the electronic device 60 includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program:

[0107] Acquire a target image; detect posture parameters of a target object included in the target image; and determine the posture of the target object in the target image according to the posture parameters.

[0108] Optionally, before detecting the posture parameters of the target object included in the target image, the method also includes: obtaining a first training sample, wherein the first training sample includes a first image of the sample object and standard posture parameters of the sample object; using the first training sample to perform machine training to obtain a detection model, wherein the detection model is a mask region convolutional neural network model Mask RCNN.

[0109] Optionally, obtaining the first training sample includes: collecting multiple images of the sample object in the target environment; amplifying the collected images to obtain multiple test images; and labeling the sample objects in the multiple test images according to standard posture parameters to obtain the first training sample.

[0110] Optionally, detecting the posture parameters of the target object included in the target image includes: inputting the target image into a detection model; determining the region of interest of the target image, the category of the region of interest, the mask of the target object region in the target image, and the coordinates of the key points of the target object through the detection model.

[0111] Optionally, determining the region of interest of the target image through the feature extraction network and region proposal network of the detection model includes: extracting features from the target image to obtain a corresponding feature map; processing the feature map to obtain multiple candidate regions; and selecting a candidate region with the highest score from the multiple candidate regions as the region of interest.

[0112] Optionally, the first training sample is used for machine training to obtain a detection model, including: inputting the first training sample into the detection model to obtain the region of interest of the first training sample, the category of the region of interest, the mask and key point coordinates of the sample object area; determining the loss value of the detection model according to the region of interest, category, mask and key point coordinates and standard posture parameters; when the loss value does not exceed a preset value, determining that the detection model training is successful.

[0113] Optionally, the standard posture parameters include: the category, center coordinates and size of the region of interest in the first training sample, the mask of the sample object in the first training sample and the coordinates of the key points. According to the region of interest, category, mask and key point coordinates, and the standard posture parameters, determining the loss value of the detection model includes: determining the classification error according to the category of the region of interest and the category of the corresponding region of interest in the first training sample; determining the coordinate error according to the center coordinates and size of the region of interest and the center coordinates and size of the region of interest in the first training sample; determining the segmentation error according to the intersection-and-union ratio of the mask and the classification error, wherein the intersection-and-union ratio is the intersection-and-union ratio of the mask and the mask of the sample object in the first training sample; determining the key point error according to the Euclidean distance between the predicted coordinates of multiple key points and the coordinates of the key points of the sample object in the first training sample; determining the loss value according to the classification error, coordinate error, segmentation error and key point error, and their respective corresponding preset weights.

[0114] Optionally, before determining the posture of the target object in the target image according to the posture parameters, the method also includes: obtaining a second training sample, wherein the second training sample includes a second image marked with standard posture parameters of the sample object, and the posture of the sample object; performing machine training through the second training sample to obtain a classification model, wherein the classification model includes but is not limited to: a decision tree classification model, a binary classification model.

[0115] Optionally, determining the posture of the target object in the target image according to the posture parameters includes: annotating the target image according to the posture parameters; and inputting the annotated target image into a classification model to obtain the posture corresponding to the target object in the target image.

[0116] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0117] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: acquiring a target image; detecting posture parameters of a target object included in the target image; and determining the posture of the target object in the target image based on the posture parameters.

[0118] Optionally, before detecting the posture parameters of the target object included in the target image, the method also includes: obtaining a first training sample, wherein the first training sample includes a first image of the sample object and standard posture parameters of the sample object; using the first training sample to perform machine training to obtain a detection model, wherein the detection model is a mask region convolutional neural network model Mask RCNN.

[0119] Optionally, obtaining the first training sample includes: collecting multiple images of the sample object in the target environment; amplifying the collected images to obtain multiple test images; and labeling the sample objects in the multiple test images according to standard posture parameters to obtain the first training sample.

[0120] Optionally, detecting the posture parameters of the target object included in the target image includes: inputting the target image into a detection model; determining the region of interest of the target image, the category of the region of interest, the mask of the target object region in the target image, and the coordinates of the key points of the target object through the detection model.

[0121] Optionally, determining the region of interest of the target image through the feature extraction network and region proposal network of the detection model includes: extracting features from the target image to obtain a corresponding feature map; processing the feature map to obtain multiple candidate regions; and selecting a candidate region with the highest score from the multiple candidate regions as the region of interest.

[0122] Optionally, the first training sample is used for machine training to obtain a detection model, including: inputting the first training sample into the detection model to obtain the region of interest of the first training sample, the category of the region of interest, the mask and key point coordinates of the sample object area; determining the loss value of the detection model according to the region of interest, category, mask and key point coordinates and standard posture parameters; when the loss value does not exceed a preset value, determining that the detection model training is successful.

[0123] Optionally, the standard posture parameters include: the category, center coordinates and size of the region of interest in the first training sample, the mask of the sample object in the first training sample and the coordinates of the key points. According to the region of interest, category, mask and key point coordinates, and the standard posture parameters, determining the loss value of the detection model includes: determining the classification error according to the category of the region of interest and the category of the corresponding region of interest in the first training sample; determining the coordinate error according to the center coordinates and size of the region of interest and the center coordinates and size of the region of interest in the first training sample; determining the segmentation error according to the intersection-and-union ratio of the mask and the classification error, wherein the intersection-and-union ratio is the intersection-and-union ratio of the mask and the mask of the sample object in the first training sample; determining the key point error according to the Euclidean distance between the predicted coordinates of multiple key points and the coordinates of the key points of the sample object in the first training sample; determining the loss value according to the classification error, coordinate error, segmentation error and key point error, and their respective corresponding preset weights.

[0124] Optionally, before determining the posture of the target object in the target image according to the posture parameters, the method also includes: obtaining a second training sample, wherein the second training sample includes a second image marked with standard posture parameters of the sample object, and the posture of the sample object; performing machine training through the second training sample to obtain a classification model, wherein the classification model includes but is not limited to: a decision tree classification model, a binary classification model.

[0125] Optionally, determining the posture of the target object in the target image according to the posture parameters includes: annotating the target image according to the posture parameters; and inputting the annotated target image into a classification model to obtain the posture corresponding to the target object in the target image.

[0126] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0130] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0131] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0132] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0133] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0135] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A gesture recognition method, It is characterized in that include: Acquire target images; Detecting posture parameters of a target object included in the target image; Determine the posture of the target object in the target image according to the posture parameter; Before detecting the posture parameters of the target object included in the target image, the method further includes: obtaining a first training sample, wherein the first training sample includes a first image of the sample object and standard posture parameters of the sample object; using the first training sample to perform machine training to obtain a detection model, wherein the detection model is a mask region convolutional neural network model Mask RCNN; Using the first training sample to perform machine training to obtain a detection model includes: inputting the first training sample into the detection model to obtain an area of ​​interest, a category of the area of ​​interest, a mask of a sample object area, and key point coordinates of the first training sample; determining a loss value of the detection model according to the area of ​​interest, the category, the mask, the key point coordinates, and the standard posture parameter; and determining that the detection model is successfully trained if the loss value does not exceed a preset value; The standard posture parameters include: the category, center coordinates and size of the region of interest in the first training sample, the mask of the sample object in the first training sample and the coordinates of the key points. According to the region of interest, the category, the mask and the key point coordinates, and the standard posture parameters, determining the loss value of the detection model includes: determining the classification error according to the category of the region of interest and the category of the corresponding region of interest in the first training sample; determining the coordinate error according to the center coordinates and size of the region of interest and the center coordinates and size of the region of interest in the first training sample; determining the segmentation error according to the intersection-and-union ratio of the mask and the classification error, wherein the intersection-and-union ratio is the intersection-and-union ratio of the mask and the mask of the sample object in the first training sample; determining the key point error according to the Euclidean distance between the predicted coordinates of multiple key points and the coordinates of the key points of the sample object in the first training sample; determining the loss value according to the classification error, the coordinate error, the segmentation error and the key point error, and the corresponding preset weights.

2. The method according to claim 1, It is characterized in that Obtaining the first training sample includes: Collect multiple images of the sample object in the target environment; Amplifying the collected images to obtain multiple test images; The sample objects of the multiple test images are labeled according to the standard posture parameters to obtain the first training samples.

3. The method according to claim 1, It is characterized in that Detecting the posture parameters of the target object included in the target image includes: Inputting the target image into the detection model; The detection model is used to determine the region of interest of the target image, the category of the region of interest, the mask of the target object region in the target image, and the coordinates of the key points of the target object.

4. The method according to claim 3, It is characterized in that Determining the region of interest of the target image by using the feature extraction network and the region proposal network of the detection model includes: Performing feature extraction on the target image to obtain a corresponding feature map; Processing the feature map to obtain multiple candidate regions; A candidate region with the highest score is selected from the multiple candidate regions as the region of interest.

5. The method according to claim 3, It is characterized in that Before determining the posture of the target object in the target image according to the posture parameter, the method further includes: Acquire a second training sample, wherein the second training sample includes a second image annotated with standard posture parameters of the sample object, and the posture of the sample object; Machine training is performed through the second training sample to obtain a classification model, wherein the classification model includes but is not limited to: a decision tree classification model and a binary classification model.

6. The method according to claim 5, It is characterized in that Determining the posture of the target object in the target image according to the posture parameter comprises: Annotating the target image by using the posture parameters; The labeled target image is input into the classification model to obtain the posture corresponding to the target object of the target image.

7. A gesture recognition device, It is characterized in that include: An acquisition module, used for acquiring a target image; A detection module, used for detecting posture parameters of a target object included in the target image; a recognition module, configured to determine the posture of the target object in the target image according to the posture parameter; Before detecting the posture parameters of the target object included in the target image, the method further includes: obtaining a first training sample, wherein the first training sample includes a first image of the sample object and standard posture parameters of the sample object; using the first training sample to perform machine training to obtain a detection model, wherein the detection model is a mask region convolutional neural network model Mask RCNN; Using the first training sample to perform machine training to obtain a detection model includes: inputting the first training sample into the detection model to obtain an area of ​​interest, a category of the area of ​​interest, a mask of a sample object area, and key point coordinates of the first training sample; determining a loss value of the detection model according to the area of ​​interest, the category, the mask, the key point coordinates, and the standard posture parameter; and determining that the detection model is successfully trained if the loss value does not exceed a preset value; The standard posture parameters include: the category, center coordinates and size of the region of interest in the first training sample, the mask of the sample object in the first training sample and the coordinates of the key points. According to the region of interest, the category, the mask and the key point coordinates, and the standard posture parameters, determining the loss value of the detection model includes: determining the classification error according to the category of the region of interest and the category of the corresponding region of interest in the first training sample; determining the coordinate error according to the center coordinates and size of the region of interest and the center coordinates and size of the region of interest in the first training sample; determining the segmentation error according to the intersection-and-union ratio of the mask and the classification error, wherein the intersection-and-union ratio is the intersection-and-union ratio of the mask and the mask of the sample object in the first training sample; determining the key point error according to the Euclidean distance between the predicted coordinates of multiple key points and the coordinates of the key points of the sample object in the first training sample; determining the loss value according to the classification error, the coordinate error, the segmentation error and the key point error, and the corresponding preset weights.

8. A processor, It is characterized in that The processor is used to run a program, wherein the program executes the gesture recognition method according to any one of claims 1 to 6 when running.

9. An electronic device, It is characterized in that It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the gesture recognition method described in any one of claims 1 to 6.

Images