Behavior recognition method, electronic device, and computer-readable storage medium

Abstract

The present application discloses a behavior recognition method, an electronic device, and a computer-readable storage medium, which relate to the fields of image processing and computer vision. The behavior recognition method comprises: detecting an image to be processed, determining the head and shoulder area and key points of a human body contained in the image to be processed; and determining whether a human body has a target behavior based on the position distribution characteristics of the head and shoulder area and key points. In the above manner, the behavior recognition method of the present application can improve the accuracy of human behavior recognition by processing the head and shoulder area and key points of the human body in the image to identify whether a human body has a target behavior.

Description

Technical Field

[0001] The present application relates to the fields of image processing and computer vision, and specifically to a behavior recognition method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of big data and deep learning, technologies based on human behavior recognition are playing an increasingly important role in various fields. However, when current human behavior recognition technologies recognize multiple behaviors or multiple human bodies, they have low recognition accuracy due to interference from environmental factors and mutual occlusion between multiple individuals. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a behavior recognition method, an electronic device and a computer-readable storage medium to improve the accuracy of human behavior recognition.

[0004] In order to solve the above technical problems, a technical solution adopted by the present application is to provide a behavior recognition method, which includes:

[0005] The image to be processed is detected to determine the head and shoulder area and key points of the human body contained in the image to be processed; based on the position distribution characteristics of the head and shoulder area and key points, it is determined whether the human body has the target behavior.

[0006] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program data, and the processor executes the program data stored in the memory to achieve: detecting the image to be processed, determining the head and shoulder area and key points of the human body contained in the image to be processed; based on the position distribution characteristics of the head and shoulder area and the key points, determining whether the human body has the target behavior.

[0007] In order to solve the above technical problems, a technical solution adopted in the present application is: providing a computer-readable storage medium, which stores program instructions internally, and the program instructions are executed to achieve: detecting the image to be processed, determining the head and shoulder area and key points of the human body contained in the image to be processed; based on the position distribution characteristics of the head and shoulder area and key points, determining whether the human body has target behavior.

[0008] Different from the existing technology, the present application uses the detection of the image to be processed to identify human behavior, and uses the position distribution characteristics of the head and shoulder area and key points of the human body contained in the detection and processing image to identify the target behavior of the human body. This can avoid the problem of incorrect behavior identification due to insufficient single feature information, and can enhance the accuracy of human behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of the first embodiment of the behavior recognition method of the present application;

[0010] Figure 2 It is a flowchart of the second embodiment of the behavior recognition method of the present application;

[0011] Figure 3 yes Figure 2 A specific flow chart of step S202;

[0012] Figure 4 yes Figure 2 A specific flow chart of step S202;

[0013] Figure 5 yes Figure 4 A specific flow chart of step S404;

[0014] Figure 6 yes Figure 2 A specific flow chart of step S202;

[0015] Figure 7 It is a flowchart of the third embodiment of the behavior recognition method of the present application;

[0016] Figure 8 It is a structural diagram of an embodiment of the joint Trident deep residual model of the present application;

[0017] Fig. 9 It is a structural diagram of another embodiment of the joint Trident deep residual model of the present application;

[0018] Fig.10 It is a structural diagram of an embodiment of the cascade pyramid model of the present application;

[0019] Fig.11 It is a structural diagram of an overall network embodiment of the cascaded pyramid model and the joint trident deep residual model of the present application.

[0020] Fig.12 It is a flowchart of an embodiment of the cascade pyramid model and the joint trident deep residual model training strategy of the present application;

[0021] Fig.13 It is a flowchart of the fourth embodiment of the behavior recognition method of the present application;

[0022] Fig.14 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0023] Fig.15 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] This application first proposes a behavior recognition method, such as Figure 1 As shown, Figure 1 This is a flow chart of the first embodiment of the behavior recognition method of the present application. The behavior recognition method specifically includes steps S101 to S102:

[0026] Step S101: Detect the image to be processed to determine the head and shoulder area and key points of the human body contained in the image to be processed.

[0027] Obtain an image to be processed for behavior recognition, and use a target detection model to detect the image to be processed. The target detection model can output detection frames such as a face frame, a head and shoulder region frame, and key points of the human body of the image to be processed. In this embodiment, the target detection model can be used to determine the head and shoulder region and key points of the human body contained in the image to be processed.

[0028] Step S102: Determine whether the human body has the target behavior based on the position distribution characteristics of the head and shoulder area and key points.

[0029] Based on the position distribution characteristics of the head and shoulder area and key points of the human body, this application pre-selects the target behavior and presets certain rules to filter out other human behaviors and determine whether the human body has the target behavior.

[0030] Different from the existing technology, the behavior recognition method of the present application utilizes the characteristics of key points of the human body and a combination of image features to identify human behavior, which can avoid the problem of incorrect behavior recognition caused by insufficient single feature information and can enhance the accuracy of human behavior recognition.

[0031] This application further proposes a behavior recognition method, such as Figure 2 As shown, Figure 2 This is a flow chart of the second embodiment of the behavior recognition method of the present application. The above behavior recognition method can be applied to an examination room, the target behavior can include cheating behavior in the examination room, and the key points include sub-key points associated with the hands of the human body. The second embodiment of the behavior recognition method specifically includes steps S201 to S202:

[0032] Step S201: Detect the image to be processed to determine the head and shoulder area and key points of the human body contained in the image to be processed.

[0033] Obtain the image to be processed of the examination room, input the image to be processed into the target detection model for detection, and determine the head and shoulder area and key points of the human body contained in the image to be processed.

[0034] Step S202: Determine the reference points of the head and shoulder area, and determine whether the human body has cheated in the examination based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area, or determine whether the human body has cheated in the examination based on the size information of the head and shoulder area.

[0035] The reference points of the head and shoulder area of ​​the human body and the sub-key points associated with the hands of the human body are obtained. Based on the position distribution between the reference points and the sub-key points and the size information of the head and shoulder area, or with reference to the size information of the head and shoulder area, certain rules are set for cheating behavior in the examination room to pre-select, so as to determine whether the human body has cheated in the examination room.

[0036] Optionally, the cheating behavior in the examination room includes lying on the table, the reference point includes the bottom center point of the head and shoulder area, and the sub-key points include at least the finger key points and wrist key points of the same hand of the human body. Figure 3 The method shown implements the step of determining whether a person has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area in step S202. The specific implementation steps include steps S301 to S302:

[0037] Step S301: Determine a first reference straight line passing through finger key points and wrist key points of the same hand.

[0038] Step S302: Based on the distance from the bottom center point to the first reference straight line, determine whether the human body is lying on the table.

[0039] Specifically, in response to the distance from the bottom center point to the first reference straight line being less than a first preset threshold, it is determined that the human body is lying on the table.

[0040] When judging whether a person is lying on the table, obtain the image to be processed in the examination room and perform detection on the image to be processed. In the examination room, due to the camera angle or the presence of a single camera, there is a problem of occlusion between the people in the examination room. It is only necessary to obtain the bottom center point of the head and shoulder area and the finger key points and wrist key points of the same hand of the person, and then determine the first reference straight line passing through the finger key points and wrist key points of the same hand of the person. When the distance from the bottom center point of the head and shoulder area to the first reference straight line is less than the first preset threshold, it is determined that the person is lying on the table.

[0041] Optionally, the cheating behavior in the examination room includes the behavior of passing objects, the reference points include the upper corner point, the upper boundary, the left shoulder key point and the right shoulder key point of the head and shoulder area, and the sub-key points include the wrist key point and the elbow key point of the same hand of the human body. Figure 4 The method shown implements the step of determining whether a person has cheated in the examination based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area in step S202. The specific implementation steps include steps S401 to S404:

[0042] Step S401: Determine a second distance from the upper corner point to the corresponding wrist key point.

[0043] Step S402: Determine whether the second distance is greater than half of the width of the head and shoulder region, and whether the minimum horizontal distance between the wrist key point and the upper boundary is less than a second preset threshold.

[0044] Step S403: If the second distance is greater than half the width of the head and shoulder area, and the minimum horizontal distance between the wrist key point and the upper boundary is less than the second preset threshold, a second reference straight line passing through the left shoulder key point and the right shoulder key point is determined, and a third reference straight line passing through the elbow key point and the left shoulder key point or the right shoulder key point on the same side as the elbow key point is determined.

[0045] Step S404: Based on the second reference straight line, the third reference straight line and the wrist key point, determine whether the human body is transferring an object.

[0046] When judging whether a human body is passing an object, obtain the image to be processed in the examination room and perform detection on the image to be processed. At this time, it is necessary to obtain the upper corner point, upper boundary, left shoulder key point, right shoulder key point and width of the head and shoulder area, and obtain the wrist key point and elbow key point of the same hand of the human body.

[0047] Determine the second distance from the upper corner point to the corresponding wrist key point. When the key point of the left wrist of the human body is obtained, the second distance from the upper left corner point of the head and shoulder area to the left wrist key point is confirmed. When the key point of the right wrist of the human body is obtained, the second distance from the upper right corner point of the head and shoulder area to the right wrist key point is confirmed. Determine whether the second distance is greater than half of the width of the head and shoulder area, and whether the minimum horizontal distance between the wrist key point and the upper boundary of the head and shoulder area is less than a second preset threshold. If the second distance is less than half of the width of the head and shoulder area and / or the minimum horizontal distance between the wrist key point and the upper boundary is greater than the second preset threshold, it is determined that the human body does not pass the item.

[0048] If the second distance is greater than half the width of the head and shoulder area, and the minimum horizontal distance between the wrist key point and the upper boundary is less than the second preset threshold, a second reference straight line passing through the left shoulder key point and the right shoulder key point is determined, and a third reference straight line passing through the elbow key point and the left shoulder key point or the right shoulder key point on the same side as the elbow key point is determined.

[0049] Then, based on the relationship between the second reference straight line, the third reference straight line and the wrist key point, it is confirmed whether the human body is passing the object.

[0050] Optionally, the sub-key points further include the wrist key points of the other hand of the human body. Figure 5 The method steps shown implement step S404, and the specific steps include S501 to S504:

[0051] Step S501: Calculate the cosine distance between the second reference straight line and the third reference straight line.

[0052] Step S502: Calculate the third distance between the wrist key point and the wrist key point of the other hand.

[0053] Step S503: Determine whether the cosine distance is less than a third preset threshold and whether the third distance is greater than a fourth preset threshold.

[0054] Step S504: In response to the cosine distance being less than the third preset threshold and the third distance being greater than the fourth preset threshold, it is determined that the human body is in the process of transferring an object.

[0055] When confirming whether the human body has transferred an object based on the relationship between the second reference straight line, the third reference straight line and the wrist key point, it is necessary to obtain the wrist key point of the other hand of the human body, first calculate the cosine distance between the second reference straight line and the third reference straight line, and then calculate the third distance between the wrist key point and the wrist key point of the other hand, and then determine whether the cosine distance is less than the third preset threshold and whether the third distance is greater than the fourth preset threshold. If the cosine distance is less than the third preset threshold and the third distance is greater than the fourth preset threshold, it is determined that the human body has transferred an object.

[0056] Optionally, the cheating behavior in the examination room includes turning around, and the size information of the head-shoulder region includes the width of the head-shoulder region and the height of the head-shoulder region. Figure 6 The method shown implements the step of determining whether a person has cheated in an examination based on the size information of the head and shoulder area in step S202, and the specific implementation steps include steps S601 to S602:

[0057] Step S601: Calculate the ratio of the width of the head-shoulder region to the height of the head-shoulder region.

[0058] Step S602: Determine whether the human body has turned around based on the ratio.

[0059] Specifically, in response to the ratio being less than a fifth preset threshold, it is determined that the human body has a turning behavior.

[0060] When judging whether the human body has turned around, the image to be processed in the examination room is obtained and detected. At this time, the width and height of the head and shoulder area need to be obtained, so as to calculate the ratio of the width of the head and shoulder area to the height of the head and shoulder area. If the ratio is less than the fifth preset threshold, it is determined that the human body has turned around.

[0061] Compared with the existing technology, the present application, through statistical analysis of a large amount of experimental data, draws a conclusion on the position distribution of the head and shoulder area and key points of the human body in cheating behaviors in general examination scenarios. By setting cheating behaviors and pre-selection rules in the examination room, other normal behaviors of the human body can be filtered out quickly and accurately. The cheating behaviors in the examination room targeted by this application are mainly lying on the table, passing objects and turning around. These three cheating behaviors in the image to be processed are identified, thereby greatly reducing the computational complexity of the subsequent secondary identification of cheating behaviors in the examination room.

[0062] like Figure 7 As shown, Figure 7 This is a flow chart of the third embodiment of the behavior recognition method provided by the present application. Specifically, it includes steps S701 to S706:

[0063] Step S701: Detect the image to be processed to determine the head and shoulder area and key points of the human body contained in the image to be processed.

[0064] Step S701 is the same as step S101 and will not be described again.

[0065] Step S702: Determine whether the human body has the target behavior based on the position distribution characteristics of the head and shoulder area and key points.

[0066] Step S702 is consistent with step S102 and will not be described again.

[0067] Step S703: Use the joint trident deep residual model to perform image recognition on the head and shoulder region to obtain image features of the head and shoulder region.

[0068] Based on the position distribution characteristics of the head and shoulder area and key points, after determining whether the human body has the target behavior, the head and shoulder area of ​​the image to be processed in which the human body has the target behavior is first obtained, and then the head and shoulder area is subjected to image recognition through the above-mentioned joint trident deep residual model to obtain the image features of the head and shoulder area.

[0069] When constructing the joint Trident deep residual model, the present application introduces the Trident network into the deep residual network and makes improvements. The present application utilizes a multi-branch Trident network based on spatial convolution, and improves the Trident network to make it more adaptable to the structure of the classification network, thereby better integrating the deep residual network. The backbone network used for behavior recognition in the present application can be a deep residual network resnet50. In other implementations, it can be other neural networks used for behavior recognition, which are not limited here.

[0070] The Trident network is a scale-sensitive detection framework, and the training process also needs to be scale-sensitively trained. This application introduces the Trident network into the deep residual network and improves it to increase the detection accuracy of behavior recognition, and there will be no additional parameters and additional calculations, which reduces the amount of calculation in the behavior recognition method.

[0071] Furthermore, the present application considers using dilated convolution to construct a multi-branch Trident network. Dilated convolution fills ordinary convolution with 0s to expand the receptive field of the network layer. The parameter of dilated convolution is the dilation rate. When the dilation rate is d, (d-1) spaces will be added to the ordinary convolution image. Assuming that the size of the original convolution kernel is k, and the size of the convolution kernel after adding (d-1) spaces is n, then n=k+(k-1)*(d-1). Therefore, if Figure 8 and Fig. 9 As shown, Figure 8 is a schematic diagram of the structure of an embodiment of the joint Trident deep residual model of the present application, Fig. 9 It is a structural diagram of another embodiment of the joint trident deep residual model of the present application. For the dilation rates of the joint trident deep residual model, the dilated convolutions with dilation rates of d=1, d=2, and d=3 respectively, the receptive fields become 3*3, 5*5, and 7*7 respectively.

[0072] When identifying cheating behaviors in the examination room, in order to solve the problem of multiple occlusions of candidates in the examination room environment, this application adopts the Trident network based on void convolution to optimize the deep residual network for the identification and classification of multiple types of cheating behaviors. It can analyze the distribution of the head and shoulder area and key points of the human body under normal behavior patterns, and then identify the target behavior, which can effectively improve the accuracy of the behavior recognition method.

[0073] Step S704: using the cascade pyramid model to identify key points in the head and shoulder region, and obtaining a heat map of the key points in the head and shoulder region.

[0074] The head and shoulder region of the image to be processed in which the human body has the target behavior is obtained, and then the key points of the head and shoulder region are identified through the above-mentioned cascade pyramid model, so as to obtain a heat map of the key points of the head and shoulder region.

[0075] When building a cascade pyramid model, Fig.10 As shown, Fig.10 It is a structural diagram of an embodiment of the cascade pyramid model of the present application. The cascade pyramid model used in the present application adopts a top-down method to obtain a heat map of the key points of the head and shoulder area by identifying the key points of the head and shoulder area.

[0076] In this application, the cascade pyramid model can further improve the positioning accuracy of the key points of the entire network. The network model has good adaptability to positioning key points in images with variable backgrounds and postures.

[0077] Step S705: fusing the image features of the head-shoulder region with the heat map of the key points of the head-shoulder region to obtain fused features.

[0078] This application will use the image features obtained by the joint Trident deep residual model and the heat map of key points obtained by the cascade pyramid model for fusion processing.

[0079] Among them, Fig.11 As shown, Fig.11 The schematic diagram of the structure of the overall network of the cascade pyramid model and the joint trident deep residual model of the present application is a schematic diagram. The cascade pyramid model can extract the behavioral information of important skeletal key points of the human body and obtain the heat map of the key points of the human body. Since different human body movements will cause changes in the absolute and relative positions of different skeletal key points, the coordinates of the skeletal key points contain important information about human behavior and movements. The joint trident deep residual model is used as the backbone network, and the image features obtained are general surface feature information of the image, such as texture, color, etc. In the overall structural network, the cascade pyramid model may promote the enhancement and improvement of the joint trident deep residual model.

[0080] Optionally, in this embodiment, step S703 may be implemented by the following method, and the specific implementation steps include:

[0081] The image feature and the heat map feature are multiplied at the pixel level to obtain a fused feature, where the pixel value of the fused feature is (1+α) times the pixel value of the image feature, where α is the pixel value of the corresponding pixel in the heat map of the image feature. Fig.11 As shown in the figure, when the image to be processed of the target behavior is transmitted into the neural network, it will be transmitted into two network branches at the same time. The upper branch is a trained cascade pyramid model, which can extract a series of heat maps of key points; the lower branch is a joint trident deep residual model, which is used to extract image features. The heat map of the key points is multiplied with the image features at the pixel level. Among them, the pixel values ​​in the heat map of the key points are all between 0 and 1, and the fusion calculation formula is:

[0082] F a =(1+α)F b

[0083] The pixel value F of the fusion feature a is the image feature pixel value F b (1+α) times, where α is the value of the corresponding pixel in the heat map of the key point. That is, the higher the value of the heat map position corresponding to the key point, the closer the position feature to the key point will be.

[0084] Step S706: Re-identify the target behavior based on the fused features.

[0085] The final layer of the network is the softmax layer, which serves as the final classification layer and can identify target behaviors based on fused features.

[0086] This application obtains image features of the head and shoulders area through a combined Trident deep residual model, obtains heat maps of key points through a cascade pyramid model, and fully integrates the heat maps of key points of the human body with image features, thereby avoiding the problem of incorrect recognition caused by insufficient information of a single feature. Compared with using a single feature, the accuracy of the behavior recognition method is further enhanced.

[0087] Alternatively, if Fig.12 As shown, Fig.12 This is a flow chart of an embodiment of the training strategy of the cascade pyramid model and the joint trident deep residual model of the present application. Before using the joint trident deep residual model and the cascade pyramid model to perform secondary recognition of the target behavior for the pre-selected behavior, it is necessary to train them. For example, when identifying cheating in an examination room, the specific steps include steps S801 to S805:

[0088] Step S801: Collect cheating behavior data.

[0089] Collect surveillance video data from different examination rooms, capture video stream images according to the frame rate, extract cheating behavior fragments and annotate cheating type labels, define cheating behaviors in examination scenarios, and build a dataset for identifying cheating behaviors in examination scenarios. The cheating behavior data collection is used for statistical analysis of target detection models and training of cascade pyramid models and joint trident deep residual models.

[0090] Step S802: using the data set, separately training the cascade pyramid model.

[0091] Step S803: fixing the parameters of the cascade pyramid model for key point positioning.

[0092] Step S804: Input the images in the data set into the key point localization network with fixed parameters, and combine the obtained results with the image features obtained by the joint Trident deep residual model for training.

[0093] Step S805: until all neural network models converge to obtain the final training model.

[0094] The training strategy of the present application first trains the cascade pyramid model for fixed key point positioning, fixes its parameters, and then trains the entire network, which can effectively save the training time of the network model and improve the training efficiency of the network model.

[0095] This application further proposes a behavior recognition method, such as Fig.13 As shown, Fig.13 This is a flow chart of the fourth embodiment of the behavior recognition method of the present application. The image to be processed includes an image of a video stream collected from an examination room. The behavior recognition method further includes steps S901 to S902:

[0096] Step S901: multiple consecutive video frames in the video stream are respectively determined as images to be processed, and it is determined whether the same human body in each video frame has cheating behavior in the examination room.

[0097] Step S902: Based on the determination result of whether the same person in each video frame has cheated in the examination, if it is determined that the duration of the cheating behavior of the same person is greater than a preset duration, a warning message is triggered for the same person.

[0098] This application also builds a cheating behavior identification and alarm system. When multiple consecutive video frames in the video stream are respectively determined as images to be processed, and it is determined that the same person in each video frame has cheated in the examination room, and the duration of the person's cheating in the examination room exceeds a preset time, an early warning is issued for the same person to remind the invigilator that the person may be cheating.

[0099] Optionally, the present application further proposes an electronic device 100. Fig.14 As shown, Fig.14 This is a structural diagram of an embodiment of an electronic device 100 of the present application, wherein the electronic device 100 includes a processor 101 and a memory 102 connected to the processor 101, wherein the memory 102 stores program data, and the processor 101 executes the program data stored in the memory 102 to perform: obtaining a video stream; obtaining key point features and image features of a human body from the video stream; and recognizing human body behavior based on the key point features and image features.

[0100] The processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an electronic chip having the ability to process signals. The processor 101 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor.

[0101] The memory 102 can be a memory stick, a TF card, etc., which can store all the information in the electronic device 100, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory 102. It stores and retrieves information according to the location specified by the processor 101. With the memory 102, the electronic device 100 has a memory function and can ensure normal operation. The memory 102 of the electronic device 100 can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose, and there is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0102] Optionally, the present application further proposes a computer-readable storage medium 110. Fig.15 As shown, Fig.15 This is a schematic diagram of the structure of an embodiment of a computer-readable storage medium 110 of the present application. If the integrated units of the functional units in the various embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium 110. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer-readable storage medium 110 includes several instructions in a program instruction 111 to enable a computer device (which can be a personal computer, a system server, or a network device, etc.), an electronic device (such as MP3, MP4, etc., or a mobile terminal such as a mobile phone, a tablet computer, a wearable device, or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of various implementation methods of the present application.

[0103] Different from the prior art, the behavior recognition method of the present application can quickly and accurately identify and classify target behaviors through the head and shoulder areas and key points of the human body; wherein, by analyzing the position distribution of the head and shoulder areas and key points of the human body under normal behavior modes, detecting the head and shoulder areas and key points of the human body through a target detection model and filtering the target behavior according to preset pre-selection rules, the influence of some irrelevant targets can be effectively eliminated, and the amount of calculation during secondary recognition can be greatly reduced; in secondary recognition, by fusing image features and heat maps of key points, feature information can be fully integrated to avoid the problem of incorrect recognition caused by insufficient single feature information, and further enhance the accuracy of the behavior recognition classification method; in addition, the joint trident deep residual model used in the present application in secondary recognition can not only solve the problem of mutual occlusion between human bodies, but also reduce the amount of calculation in the behavior recognition method, and the cascade pyramid model used can also improve the accuracy of positioning key points of the human body, thereby improving the accuracy of the entire behavior recognition method in identifying target behaviors.

[0104] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A behavior recognition method, characterized in that: The behavior recognition method is applied to an examination room, and the behavior recognition method includes: Detect the image to be processed to determine the head and shoulder area and key points of the human body contained in the image to be processed; Based on the position distribution characteristics of the head-shoulder region and the key points, determining whether the human body has a target behavior; The target behavior includes cheating in an examination room, and the key points include sub-key points associated with the hands of the human body; and determining whether the human body has the target behavior based on the position distribution characteristics of the head and shoulder region and the key points includes: Determine the reference point of the head and shoulder area; and determine whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area, or determine whether the human body has cheated in the examination room based on the size information of the head and shoulder area; The cheating behavior in the examination room includes the behavior of passing objects, the reference points include the upper corner point, the upper boundary, the left shoulder key point and the right shoulder key point of the head and shoulder area; the sub-key points include the wrist key point and the elbow key point of the same hand of the human body; The determining whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area includes: Determine a second distance from the upper corner point to the corresponding wrist key point; Determine whether the second distance is greater than half of the width of the head and shoulder region, and whether the minimum horizontal distance between the wrist key point and the upper boundary is less than a second preset threshold; If the second distance is greater than half of the width of the head-shoulder region, and the minimum horizontal distance between the wrist key point and the upper boundary is less than the second preset threshold, a second reference straight line passing through the left shoulder key point and the right shoulder key point is determined, and a third reference straight line passing through the elbow key point and the left shoulder key point or the right shoulder key point on the same side as the elbow key point is determined; Based on the second reference straight line, the third reference straight line and the wrist key point, it is determined whether the human body has the behavior of transferring the item.

2. The behavior recognition method according to claim 1, characterized in that: The cheating behavior in the examination room includes lying on the table, the reference point includes the bottom center point of the head and shoulder area, and the sub-key points include at least the finger key points and the wrist key points of the same hand of the human body; The determining whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area includes: Determine a first reference straight line passing through the finger key points and the wrist key points of the same hand; Based on the distance from the bottom center point to the first reference straight line, it is determined whether the human body has the behavior of lying on the table.

3. The behavior recognition method according to claim 2, characterized in that: The determining whether the human body has the behavior of lying on the table based on the distance from the bottom center point to the first reference straight line includes: In response to the distance from the bottom center point to the first reference straight line being less than a first preset threshold, it is determined that the human body is in the behavior of lying on the table.

4. The behavior recognition method according to claim 1, characterized in that: The sub-key points further include a wrist key point of the other hand of the human body, and the determining whether the human body has the object transfer behavior based on the second reference straight line, the third reference straight line and the wrist key point includes: Calculating a cosine distance between the second reference straight line and the third reference straight line; Calculating a third distance between the wrist key point and the wrist key point of the other hand; Determine whether the cosine distance is less than a third preset threshold and whether the third distance is greater than a fourth preset threshold; In response to the cosine distance being smaller than the third preset threshold and the third distance being larger than the fourth preset threshold, it is determined that the human body has performed the item transfer behavior.

5. The behavior recognition method according to claim 1, characterized in that: The cheating behavior in the examination room includes turning around, and the size information of the head-shoulder region includes the width and height of the head-shoulder region; The determining whether the human body has cheated in the examination room based on the size information of the head and shoulder area includes: Calculating the ratio of the width of the head-shoulder region to the height of the head-shoulder region; It is determined based on the ratio whether the human body has the turning behavior.

6. The behavior recognition method according to claim 5, characterized in that: The determining, based on the ratio, whether the human body has the turning behavior comprises: In response to the ratio being less than a fifth preset threshold, it is determined that the human body has performed the turning behavior.

7. The behavior recognition method according to claim 1, characterized in that: After the step of determining whether the human body has a target behavior based on the position distribution characteristics of the head-shoulder region and the key points, the method further includes: Performing image recognition on the head and shoulder region using a joint trident deep residual model to obtain image features of the head and shoulder region; Using a cascade pyramid model to identify key points in the head and shoulder area, and obtaining a heat map of the key points in the head and shoulder area; Fusing the image features of the head-shoulder region with the heat map of the key points of the head-shoulder region to obtain fused features; The target behavior is re-identified based on the fused features.

8. The behavior recognition method according to claim 7, characterized in that: The fusing the image features of the head-shoulder region with the heat map of the key points of the head-shoulder region to obtain the fused features includes: The image feature and the heat map feature are multiplied at the pixel level to obtain the fusion feature, where: The pixel value of the fusion feature is (1+α) times the pixel value of the image feature, where α is the pixel value of the corresponding pixel in the heat map of the image feature.

9. The behavior recognition method according to any one of claims 1 to 8, characterized in that: The image to be processed includes an image of a video stream collected from an examination room, and the method further includes: Determine a plurality of continuous video frames in the video stream as the images to be processed respectively, and determine whether the same human body in each of the video frames has cheated in the examination room; Based on the determination result of whether the same human body in each of the video frames has cheated in the examination room, it is determined that the duration of the cheating behavior of the same human body in the examination room is greater than a preset duration, and then a warning message is triggered for the same human body.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory connected to the processor, wherein the memory stores program data, and the processor executes the program data stored in the memory to implement: Detect the image to be processed to determine the head and shoulder area and key points of the human body contained in the image to be processed; Based on the position distribution characteristics of the head-shoulder region and the key points, determining whether the human body has a target behavior; The target behavior includes cheating in an examination room, and the key points include sub-key points associated with the hands of the human body; and determining whether the human body has the target behavior based on the position distribution characteristics of the head and shoulder region and the key points includes: Determine the reference point of the head and shoulder area; and determine whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area, or determine whether the human body has cheated in the examination room based on the size information of the head and shoulder area; The cheating behavior in the examination room includes the behavior of passing objects, the reference points include the upper corner point, the upper boundary, the left shoulder key point and the right shoulder key point of the head and shoulder area; the sub-key points include the wrist key point and the elbow key point of the same hand of the human body; The determining whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area includes: Determine a second distance from the upper corner point to the corresponding wrist key point; Determine whether the second distance is greater than half of the width of the head and shoulder region, and whether the minimum horizontal distance between the wrist key point and the upper boundary is less than a second preset threshold; If the second distance is greater than half of the width of the head-shoulder region, and the minimum horizontal distance between the wrist key point and the upper boundary is less than the second preset threshold, a second reference straight line passing through the left shoulder key point and the right shoulder key point is determined, and a third reference straight line passing through the elbow key point and the left shoulder key point or the right shoulder key point on the same side as the elbow key point is determined; Based on the second reference straight line, the third reference straight line and the wrist key point, it is determined whether the human body has the behavior of transferring the item.

11. A computer-readable storage medium having program instructions stored therein, characterized in that: The program instructions are executed to implement: Detect the image to be processed to determine the head and shoulder area and key points of the human body contained in the image to be processed; Based on the position distribution characteristics of the head-shoulder region and the key points, determining whether the human body has a target behavior; The target behavior includes cheating in an examination room, and the key points include sub-key points associated with the hands of the human body; The determining whether the human body has a target behavior based on the position distribution characteristics of the head-shoulder region and the key points includes: Determine the reference point of the head and shoulder area; and determine whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area, or determine whether the human body has cheated in the examination room based on the size information of the head and shoulder area; The cheating behavior in the examination room includes the behavior of passing objects, the reference points include the upper corner point, the upper boundary, the left shoulder key point and the right shoulder key point of the head and shoulder area; the sub-key points include the wrist key point and the elbow key point of the same hand of the human body; The determining whether the human body has cheated in the examination room based on the position distribution between the sub-key points and the reference points and the size information of the head and shoulder area includes: Determine a second distance from the upper corner point to the corresponding wrist key point; Determine whether the second distance is greater than half of the width of the head and shoulder region, and whether the minimum horizontal distance between the wrist key point and the upper boundary is less than a second preset threshold; If the second distance is greater than half of the width of the head-shoulder region, and the minimum horizontal distance between the wrist key point and the upper boundary is less than the second preset threshold, a second reference straight line passing through the left shoulder key point and the right shoulder key point is determined, and a third reference straight line passing through the elbow key point and the left shoulder key point or the right shoulder key point on the same side as the elbow key point is determined; Based on the second reference straight line, the third reference straight line and the wrist key point, it is determined whether the human body has the behavior of transferring the item.

Images