Behavior recognition method, device, terminal equipment and storage medium

Abstract

The present application is applicable to the field of image processing technology, and provides a behavior recognition method, device, terminal device and storage medium. The recognition method includes: obtaining an image to be processed for a target person; performing human skeleton key point detection on the image to be processed to obtain a first human skeleton key point of the target person; determining first connection information of a target key point group in the first human skeleton key points based on the skeleton connection relationship between the first human skeleton key points; generating a target data fingerprint of the target person according to the first connection information of the target key point group in the first human skeleton key points; determining a behavior recognition result of the target person according to a comparison result between the target data fingerprint and a behavior data fingerprint in a behavior database. The user's behavior can be recognized according to the above scheme.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a behavior recognition method, device, terminal device and storage medium. Background Art

[0002] In campus life, most students can follow the code of conduct to ensure that they walk in a standardized manner on stairs and campus roads. However, there are a few students who cannot follow the code of conduct and run fast, chase each other, jump and walk upside down on campus roads and stairs. These dangerous behaviors can easily harm students' physical and mental health. Therefore, in order to reduce the occurrence of these dangerous behaviors and at the same time facilitate the school to focus on students with dangerous behaviors, how to identify student behavior is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a behavior recognition method, apparatus, terminal device, and storage medium, which can recognize user behavior.

[0004] A first aspect of an embodiment of the present application provides a behavior recognition method, the recognition method comprising:

[0005] Obtain an image to be processed for a target person;

[0006] Performing human skeleton key point detection on the image to be processed to obtain a first human skeleton key point of the target person;

[0007] Based on the skeletal connection relationship between the first human skeletal key points, determine first connection information of a target key point group in the first human skeletal key points, where the target key point group includes any two human skeletal key points having the connection relationship;

[0008] Generate a target data fingerprint of the target person according to first connection information of a target key point group in the first human skeleton key points;

[0009] The behavior recognition result of the target person is determined according to the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database.

[0010] A second aspect of an embodiment of the present application provides a behavior recognition device, the recognition device comprising:

[0011] An acquisition module, used for acquiring an image to be processed for a target person;

[0012] A recognition module is used to detect the key points of the human skeleton of the image to be processed to obtain the first key point of the human skeleton of the target person;

[0013] An information determination module, used to determine first connection information of a target key point group in the first human skeleton key points based on the skeleton connection relationship between the first human skeleton key points, wherein the target key point group includes any two human skeleton key points having the connection relationship;

[0014] A generating module, used for generating a target data fingerprint of the target person according to first connection information of a target key point group in the first human skeleton key points;

[0015] The determination module is used to determine the behavior recognition result of the target person according to the comparison result of the target data fingerprint and the behavior data fingerprint in the behavior database.

[0016] A third aspect of an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the behavior recognition method described in the first aspect when executing the computer program.

[0017] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the behavior recognition method described in the first aspect is implemented.

[0018] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the behavior recognition method described in the first aspect above.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application can obtain the first human skeleton key point of the target person by performing human skeleton key point detection on the image to be processed for the target person; since the connection information between the first human skeleton key points can reflect the body shape of the target person, the connection information of the target key point group in the first human skeleton key points can be determined based on the skeleton connection relationship between the first human skeleton key points; then, based on the first connection information of the target key point group in the first human skeleton key points, the target data fingerprint of the target person is generated; finally, based on the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database, the behavior recognition result of the target person is determined; the above scheme can be used to identify the behavior of the user; for example, when the user is a student, the above scheme can be used to identify the behavior of the student, so that the school can focus on students with dangerous behaviors and reduce the occurrence of dangerous behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is a flow chart of a behavior recognition method provided in Example 1 of the present application;

[0022] Figure 2 It is a schematic diagram of the key points of the first human skeleton;

[0023] Figure 3 This is a flow chart of a behavior recognition method provided in Example 2 of this application.

[0024] Figure 4 It is a schematic diagram of the key points of the second human skeleton;

[0025] Figure 5 It is a structural diagram of a behavior recognition device provided in Embodiment 3 of the present application;

[0026] Figure 6 It is a structural diagram of a terminal device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0027] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0028] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0029] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0030] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0031] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0033] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0034] In the study of behavior recognition methods, it was found that most of the current methods for behavior recognition are based on single-scene recognition, for example, recognition of jumping behavior, recognition of running behavior, or recognition of fighting behavior. All of the above are behavior recognition in a single scene. If new scenes are to be added, new recognition algorithms need to be developed. Therefore, the behavior recognition methods in the prior art have the problem of a single scene and are not conducive to quick judgment of user behavior.

[0035] The present application provides a behavior recognition method, apparatus, terminal device and storage medium. By performing human skeleton key point detection on an image to be processed of a target person, a first human skeleton key point of the target person can be obtained. Since the connection information between the first human skeleton key points can reflect the body shape of the target person, the connection information of the target key point group in the first human skeleton key points can be determined based on the skeleton connection relationship between the first human skeleton key points. Then, based on the first connection information of the target key point group in the first human skeleton key points, a target data fingerprint of the target person is generated. Finally, based on the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database, a behavior recognition result of the target person is determined. The behavior database of the recognition method provided by the present application can contain behavior data fingerprints corresponding to multiple behavior categories. Therefore, the recognition method provided by the present application can quickly judge multiple behaviors of the target person and realize behavior recognition in multiple scenarios.

[0036] In order to illustrate the technical solution of the present application, specific embodiments are provided below.

[0037] Reference Figure 1 , shows a flow chart of a behavior recognition method provided in Example 1 of the present application. Figure 1 As shown, the identification method may include the following steps:

[0038] Step 101: Obtain an image to be processed for a target person.

[0039] In the embodiment of the present application, the image to be processed for the target person can be obtained from multiple frames of images collected by the image acquisition device. For example, the image acquisition device can collect monitoring video information in real time, and the monitoring video information can be decomposed into multiple frames of images according to the time series. The image to be processed for the target person can be any frame of the multiple frames of images for the target person. The target person can be a pedestrian in the monitoring video. For example, in a campus scene, the target person can be a student in the campus monitoring video; in a hotel management scene, the target person can be a staff member in the hotel monitoring video.

[0040] Taking the campus scene as an example to identify the behavior of students, after obtaining the image to be processed for the target person, the target detection algorithm can be used to detect the target person contained in the image to be processed, and the position of the target person in the image is detected. The position is represented by a target box, and the face of the target person in the target box is detected. If it is detected that the target person is not a student in the campus, the image to be processed can be discarded. If it is detected that the target person is a student in the campus, the image to be processed can be subsequently subjected to human skeleton key point detection.

[0041] It should be understood that the image acquisition device can be any device that has the function of acquiring video or images, such as a monocular camera, a binocular camera, etc.

[0042] Step 102, performing human skeleton key point detection on the image to be processed to obtain the first human skeleton key point of the target person.

[0043] In the embodiment of the present application, the human skeleton key point detection algorithm can be used to detect the human skeleton key point of the image to be processed, such as OpenPose, DeepCut and AlphaPose algorithms, which can detect 18 human skeleton key points, such as Figure 2 1 is a schematic diagram of the first human skeleton key points shown in FIG. 1 , wherein the first human skeleton key points included in the figure are nose key point 0, neck key point 1, left shoulder key point 2, left elbow key point 3, left wrist key point 4, right shoulder key point 5, right elbow key point 6, right wrist key point 7, left hip key point 8, left knee key point 9, left ankle key point 10, right hip key point 11, right knee key point 12, right ankle key point 13, left eye key point 14, right eye key point 15, left ear key point 16, and right ear key point 17.

[0044] In a possible implementation, performing human skeleton key point detection on the image to be processed to identify the first human skeleton key point of the target person includes:

[0045] Get the target frame of the target person in the image to be processed;

[0046] Performing human skeleton key point detection on the target person in the target frame, if the number of detected human skeleton key points is greater than or equal to the number threshold, determining the detected human skeleton key point as the first human skeleton key point of the target person;

[0047] If the number of detected human skeleton key points is less than the number threshold, it is determined that the detection of the first human skeleton key point of the target person has failed.

[0048] In an embodiment of the present application, a target frame of a target person in an image to be processed is first obtained, wherein the target frame is used to frame an area of ​​the target person in the corresponding image to be processed, and for example, the target frame can be in the form of a rectangular frame. After the rectangular frame is obtained, human key point detection is performed on the target person in the rectangular frame. If there are multiple target persons in the image to be processed, there may be a situation where the target persons are mutually occluded, resulting in the number of the first human skeleton key points detected for the target person being less than 18. At this point, whether the detection of the first human skeleton key points is successful can be determined by comparing the number of the first human skeleton key points detected with a preset number threshold.

[0049] Exemplarily, assume that the preset quantity threshold is A, and A is less than or equal to 18. If the quantity of the detected first human body bone key points is B, and B < A, it indicates that the quantity of the detected first human body bone key points does not reach the preset quantity threshold, and the target person in the corresponding image to be processed is seriously occluded, and behavior recognition cannot be performed. Therefore, the recognition of the first human body bone key points of the target person fails at this time, and the corresponding image to be processed is not used as an image for behavior recognition.

[0050] If the quantity of the detected first human body bone key points is C, and C > A, it indicates that the quantity of the recognized first human body bone key points has reached the preset quantity threshold, and the target person in the corresponding image to be processed is less occluded or not occluded, and behavior recognition can be performed on the target person. At this time, it can be determined that the detected human body bone key points are the first human body bone key points of the target person.

[0051] Step 103: Determine the first connection information of the target key point group among the first human body bone key points based on the bone connection relationship between the first human body bone key points.

[0052] In the embodiment of the present application, the bone connection relationship between the first human body bone key points refers to the connection relationship between the human body bone key points obtained according to the human body structure. For example, there is a connection relationship between the left shoulder and the left elbow, and there is a connection relationship between the left elbow and the left wrist, etc. The target key point group includes any two human body bone key points with a connection relationship. For example, as Figure 2 shown, the target key point group can refer to a 5-key point group including any two human body bone key points with a connection relationship, namely the first key point group, the second key point group, the third key point group, the fourth key point group, and the fifth key point group. Among them, the first key point group can specifically refer to the key point group including the left shoulder key point and the left elbow key point, the second key point group can specifically refer to the key point group including the right shoulder key point and the right elbow key point, the third key point group can specifically refer to the key point group including the left elbow key point and the left wrist key point, the fourth key point group can specifically refer to the key point group including the right elbow key point and the right wrist key point, and the fifth key point group can specifically refer to the key point group including the left knee key point and the left ankle key point. Determining the first connection information of the target key point group among the first human body bone key points can refer to determining the first connection information between the left shoulder key point and the left elbow key point or determining the first connection information between the right shoulder key point and the right elbow key point, etc.

[0053] In a possible implementation manner, determining the first connection information of the target key point group among the first human body bone key points based on the bone connection relationship between the first human body bone key points includes:

[0054] Obtaining a human skeleton key point identifier corresponding to the first human skeleton key point;

[0055] According to the human skeleton key point identifier corresponding to the first human skeleton key node and the skeleton connection relationship between the human skeleton key points, the first connection information of the target key point group in the first human skeleton key point is determined.

[0056] In an embodiment of the present application, the first connection information may include connection length, connection direction, connection inclination, and connection angle. Among them, the connection length may refer to the length of the line between any two human skeleton key points having a connection relationship, the connection direction may refer to the direction of the line between any two human skeleton key points having a connection relationship, the connection inclination may refer to the angle between the line between any two human skeleton key points having a connection relationship and the horizontal line, the connection angle may refer to the angle between the line between any two human skeleton key points having a connection relationship and the adjacent line segment, and the adjacent line segment refers to the line segment adjacent to the line between any two human skeleton key points having a connection relationship.

[0057] Exemplarily, taking the first key point group as an example, the connection length refers to the distance between the two first human skeleton key points of the left shoulder key point and the left elbow key point. For the connection direction, taking the image to be processed as an example, assuming that the direction pointing to the top in the image to be processed is north, the direction pointing to the bottom is south, the direction pointing to the left is west, and the direction pointing to the right is east, such as when people walk upright, the connection direction of the left shoulder key point and the left elbow key point can be from north to south. For the connection inclination, taking the image to be processed as an example, a two-dimensional coordinate system is established, the horizontal direction is the x-axis, the vertical direction is the y-axis, and the lower left corner area of ​​the image to be processed is the origin, and the connection inclination of the first key point group can refer to the angle between the line connecting the left shoulder key point and the left elbow key point and the positive direction of the x-axis. For the connection angle, it refers to the angle between the line segment connecting the left shoulder key point and the left elbow key point and the adjacent line segment, wherein the adjacent line segment refers to the line segment between the left shoulder key point and another human skeleton key point having a connection relationship with the left shoulder key point.

[0058] It should be understood that determining the first connection information of the target key point group in the first human skeleton key point refers to determining the first connection information between any two skeleton key points having a connection relationship in the first human skeleton key point.

[0059] Step 104, generating a target data fingerprint of the target person according to the first connection information of the target key point group in the first human skeleton key point.

[0060] Among them, data fingerprint refers to the organization and storage of various types of collected intelligence information, making it a searchable document archive. The intelligence can include text, images, tables, data, fingerprints, sounds and other information. The document archive can also be expressed in many forms, such as data sets, data matrices, and data tables.

[0061] In the embodiment of the present application, the target data fingerprint refers to a retrievable data set generated by organizing and storing the first connection information of the target key point group in the first human skeleton key point.

[0062] In a possible implementation, generating a target data fingerprint of a target person according to first connection information of a target key point group in a first human skeleton key point includes:

[0063] Determine the first connection information of the target key point group in the first human skeleton key point as the target data fingerprint of the target person;

[0064] Correspondingly, comparing the target data fingerprint with the behavior data fingerprint in the behavior database may mean comparing the first connection information in the target data fingerprint with the data belonging to the same key point group and the same attributes in the connection information in the behavior data fingerprint to obtain a first similarity between the target data fingerprint and each behavior data fingerprint. The same attributes refer to that the two compared data have the same connection length, the same connection direction, the same connection inclination and the same connection angle.

[0065] Exemplarily, assuming that belonging to the same key point group refers to a key point group that includes both the left elbow key point and the left wrist key point, comparing the first connection information in the target data fingerprint with the data belonging to the same key point group and the same attribute in the connection information in the behavior data fingerprint means comparing the connection length between the left elbow key point and the left wrist key point in the target data fingerprint with the connection length between the left elbow key point and the left wrist key point in each behavior data fingerprint, comparing the connection direction between the left elbow key point and the left wrist key point in the target data fingerprint with the connection direction between the left elbow key point and the left wrist key point in each behavior data fingerprint, comparing the connection inclination between the left elbow key point and the left wrist key point in the target data fingerprint with the connection inclination between the left elbow key point and the left wrist key point in each behavior data fingerprint, and comparing the connection angle between the left elbow key point and the left wrist key point in the target data fingerprint with the connection angle between the left elbow key point and the left wrist key point in each behavior data fingerprint.

[0066] In a possible implementation, generating a target data fingerprint of a target person according to first connection information of a target key point group in a first human skeleton key point includes:

[0067] According to the human skeleton key point identifiers corresponding to the target key point group, the first connection information of the target key point group is stored in the corresponding position in the data set to generate the target data fingerprint of the target person.

[0068] In an embodiment of the present application, generating the target data fingerprint of the target person may also refer to storing the first connection information of the target key point group into the corresponding position in the data set, for example, storing the first connection information of the left shoulder key point and the left wrist key point in the data set for the corresponding position of the human skeleton key point being the left shoulder key point and the left wrist key point (for example, the third row in the data set is the corresponding position of the left shoulder key point and the left wrist key point, then the first connection information of the left shoulder key point and the left wrist key point is stored in the third row in the data set), until all the first connection information of all the target key point groups are stored in the corresponding positions in the data set, and the data set is determined to be the target data fingerprint of the target person.

[0069] Step 105, determining the behavior recognition result of the target person according to the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database.

[0070] In an embodiment of the present application, the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database can be reflected in the form of similarity, that is, the comparison result between the target data fingerprint and the behavior data fingerprint is the similarity between the target data fingerprint and the behavior data fingerprint. The behavior recognition result of the target person is determined based on the similarity between the target data fingerprint and each behavior fingerprint data in the behavior database.

[0071] In a possible implementation, determining the target person's behavior recognition result according to the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database includes:

[0072] Comparing the target data fingerprint with each behavior data fingerprint in the behavior database respectively to obtain a first similarity between the target data fingerprint and each behavior data fingerprint;

[0073] If the maximum value in the first similarity is greater than or equal to the preset threshold, the behavior category corresponding to the behavior data fingerprint corresponding to the maximum value is determined as the behavior recognition result of the target person.

[0074] In an embodiment of the present application, each behavior data fingerprint in the behavior database has a corresponding behavior category. If the preset threshold is 90 and the maximum value of the first similarity is 95, the behavior data fingerprint corresponding to the first similarity of 95 is determined to be the data fingerprint most similar to the target data fingerprint, and the behavior category corresponding to the behavior data fingerprint is determined as the behavior category to which the target person belongs.

[0075] In a possible implementation, determining the behavior recognition result of the target person according to the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database further includes:

[0076] If the maximum value among all the first similarities is less than a preset threshold, the target data fingerprint is stored in the category to be trained in the behavior database;

[0077] Receive the behavior category to which the image to be processed sent by the user belongs;

[0078] If the behavior category to which the image to be processed belongs exists in the behavior database, the target data fingerprint will be associated with the corresponding category in the behavior database;

[0079] If the behavior category to which the image to be processed belongs does not exist in the behavior database, the association relationship between the behavior category to which the image to be processed belongs and the target data fingerprint is stored in the behavior database.

[0080] In an embodiment of the present application, if the maximum value of all the first similarities is less than a preset threshold, it means that there is no behavior data fingerprint matching the target data fingerprint in the behavior database, and it also means that the behavior category corresponding to the target data fingerprint does not exist in the behavior database. Therefore, the target data fingerprint can be stored in the category to be trained of the behavior database to increase the number of data fingerprints in the behavior database. By classifying the target data fingerprints in the category to be trained, the behavior categories in the behavior database can be improved to increase the comprehensiveness of behavior recognition. Among them, the category to be trained includes data fingerprints that have not been specifically classified. After the target data fingerprint is stored in the category to be trained of the behavior database, the target data fingerprint can be associated with the behavior category to which the image to be processed belongs according to the behavior category to which the image to be processed sent by the receiving user belongs.

[0081] Specifically, if the behavior category to which the image to be processed sent by the user belongs exists in the behavior database, the target data fingerprint is associated with the corresponding behavior category in the behavior database; if the behavior category to which the image to be processed sent by the user belongs does not exist in the behavior database, the target data fingerprint is associated with the behavior category to which the image to be processed belongs and then stored in the behavior database. At the same time, new behavior categories are also added to the behavior database. Continuously improving the behavior categories in the behavior database can increase the comprehensiveness of behavior recognition.

[0082] In a possible implementation, comparing the target data fingerprint with each behavior data fingerprint in the behavior database to obtain a first similarity between the target data fingerprint and each behavior data fingerprint includes:

[0083] Compare the target data fingerprint with the elements at the same position in any behavior data fingerprint one by one to obtain the second similarity between the elements at the same position;

[0084] A first similarity between the target data fingerprint and the corresponding behavior data fingerprint is determined according to the second similarity between the elements located at the same position and the preset weight of the target key point group corresponding to the second similarity.

[0085] In an embodiment of the present application, since generating the target data fingerprint of the target person may refer to storing the first connection information of the target key point group into the corresponding position in the data set, it means that in the data set, the key point groups of elements in the same set are the same and the attributes are the same, and the similarity between the data fingerprints can be determined by calculating the similarity between the elements.

[0086] Specifically, the elements at the same position are compared one by one. If the elements at the same position are different (for example, one element does not exist at the same position in the target data fingerprint and the behavior data fingerprint, also known as different elements at the same position), the total similarity of the target key point group will be deducted, and each element in the comparison data set is traversed to obtain the second similarity of each target key point group. Since the first connection information between each target key point group has a different influence on human behavior, a weight should be preset for each target key point group, and the second similarity of each target key point group is multiplied by the preset weight of the target key point group corresponding to the second similarity to obtain the third similarity of each target key point group. Finally, the third similarities of each target key point group are added to obtain the first similarity between the target data fingerprint and the behavior data fingerprint.

[0087] It should be understood that the target data fingerprint should be compared with each behavior data fingerprint in the behavior database to obtain a first similarity between the target data fingerprint and each behavior data fingerprint.

[0088] In an embodiment of the present application, by performing human skeleton key point detection on the image to be processed for the target person, the first human skeleton key point of the target person can be obtained. Since the connection information between the first human skeleton key points can reflect the body shape of the target person, the connection information of the target key point group in the first human skeleton key point can be determined based on the bone connection relationship between the first human skeleton key points, and then the target data fingerprint of the target person is generated according to the first connection information of the target key point group in the first human skeleton key point. Finally, according to the comparison result of the target data fingerprint and the behavior data fingerprint in the behavior database, the behavior recognition result of the target person is determined. The above scheme can be used to identify the behavior of the user. For example, when the user is a student, the above scheme can be used to identify the behavior of the student, so that the school can focus on students with dangerous behaviors and reduce the occurrence of dangerous behaviors.

[0089] See also Figure 3 , shows a flow chart of a behavior recognition method provided in Embodiment 2 of the present application. Figure 3 As shown, the identification method may include the following steps:

[0090] Step 301: Obtain an image to be processed for a target person.

[0091] Step 302: Perform human skeleton key point detection on the image to be processed to identify the first human skeleton key point of the target person.

[0092] Steps 301 - 302 of this embodiment are the same as steps 101 - 102 of the aforementioned embodiment, and can refer to each other, and will not be described in detail in this embodiment.

[0093] Step 303, based on the symmetry between the skeletal key points of the target person, partially symmetrical first human skeletal key points are extracted from the first human skeletal key points and merged according to the central axis of the human body to obtain merged human skeletal key points, and the merged human skeletal key points and the unmerged human skeletal key points in the first human skeletal key points are determined as second human skeletal key points.

[0094] In the embodiment of the present application, since the fatness or thinness of a human body will have a certain influence on the behavior recognition of the human body, based on the symmetry between the human skeleton key points of the target person, the partially symmetrical first human skeleton key points in the first human skeleton key points can be merged according to the central axis of the human body, wherein the partially symmetrical first human skeleton key points may include the left eye key point and the right eye key point symmetrical thereto, the left ear key point and the right ear key point symmetrical thereto, the left shoulder key point and the right shoulder key point symmetrical thereto, and the left hip key point and the right hip key point symmetrical thereto. The left shoulder key point and the right shoulder key point symmetrical thereto, as well as the left hip key point and the right hip key point symmetrical thereto are merged, that is, the shoulder width and hip width of the human body are removed, so that the human behavior comparison is more standard. Since the positions of the left eye key point and the right eye key point symmetrical thereto, the left ear key point and the right ear key point symmetrical thereto have little influence on the human behavior recognition, the left eye key point and the right eye key point symmetrical thereto, the left ear key point and the right ear key point symmetrical thereto can be merged to one human skeleton key point of the nose key point, such as Figure 4 The diagram shows the key points of the second human skeleton. Figure 2 The left eye key point 14, the right eye key point 15, the left ear key point 16, and the right ear key point 17 are merged into the nose key point 0, the left shoulder key point 2 and the right shoulder key point 5 are merged into the neck key point 1, and the left hip key point 8 and the right hip key point 11 are merged into the corresponding position 18 at the central axis. The corresponding position can be called the hip key point 18. After merging, the following is obtained: Figure 4 Shown are the key points of the second human skeleton.

[0095] Step 304, based on the skeleton connection relationship between the second human skeleton key points, determine the second connection information of the target key point group in the second human skeleton key points.

[0096] In the embodiment of the present application, the method for determining the second connection information is the same as the method for determining the first connection information in the first embodiment, and they can refer to each other, and this embodiment will not be repeated here.

[0097] Step 305: Generate a target data fingerprint of the target person according to the second connection information.

[0098] In the embodiment of the present application, the method of generating the target data fingerprint is the same as the method of generating the target data fingerprint in the first embodiment, and they can refer to each other, and this embodiment will not be repeated here.

[0099] Step 306, determining the behavior recognition result of the target person according to the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database.

[0100] Step 306 of this embodiment is the same as step 105 of the above embodiment, and they can refer to each other, and will not be described in detail in this embodiment.

[0101] Compared with the first embodiment, the embodiment of the present application merges some of the symmetrical first human skeleton key points in the first human skeleton key points based on the consideration of human body shape (i.e., the fatness or thinness of the human body). Specifically, the partially symmetrical human skeleton key points may include a left eye key point and a right eye key point symmetrical thereto, a left ear key point and a right ear key point symmetrical thereto, a left shoulder key point and a right shoulder key point symmetrical thereto, and a left hip key point and a right hip key point symmetrical thereto. The left shoulder key point and the right shoulder key point symmetrical thereto, as well as the left hip key point and the right hip key point symmetrical thereto are merged. The above-mentioned first human skeleton key points have little effect on the behavior recognition of the human body. Merging the above-mentioned first human skeleton key points can not only reduce the recognition error caused by the different shapes of the human body, but also reduce the amount of data comparison in the recognition process, thereby speeding up the recognition speed.

[0102] See also Figure 5 , shows a schematic diagram of the structure of a behavior recognition device provided in Example 3 of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0103] The behavior recognition device 5 may specifically include the following modules:

[0104] An acquisition module 501 is used to acquire an image to be processed for a target person;

[0105] The recognition module 502 is used to detect the key points of the human skeleton of the image to be processed to obtain the first key points of the human skeleton of the target person;

[0106] An information determination module 503 is used to determine first connection information of a target key point group in the first human skeleton key points based on the skeleton connection relationship between the first human skeleton key points, where the target key point group includes any two human skeleton key points having a connection relationship;

[0107] A generating module 504, configured to generate a target data fingerprint of a target person according to first connection information of a target key point group in a first human skeleton key point;

[0108] The result determination module 505 is used to determine the behavior recognition result of the target person according to the comparison result between the target data fingerprint and the behavior data fingerprint in the behavior database.

[0109] In the embodiment of the present application, the behavior recognition device may further include the following modules:

[0110] An extraction module is used to extract partially symmetrical first human skeleton key points from the first human skeleton key points based on the symmetry between the human skeleton key points of the target person, merge them according to the central axis of the human body, obtain the merged human skeleton key points, and determine the merged human skeleton key points and the unmerged human skeleton key points in the first human skeleton key points as the second human skeleton key points, wherein the partially symmetrical first human skeleton key points include a left eye key point and a symmetrical right eye key point, a left ear key point and a symmetrical right ear key point, a left shoulder key point and a symmetrical right shoulder key point, and a left hip key point and a symmetrical right hip key point;

[0111] A connection determination module, used to determine second connection information of a target key point group in the second human skeleton key points based on the skeleton connection relationship between the second human skeleton key points;

[0112] Correspondingly, the generating module 504 may be specifically used for:

[0113] A target data fingerprint of the target person is generated according to the second connection information.

[0114] In the embodiment of the present application, the identification module 502 may specifically include the following submodules:

[0115] The target frame acquisition submodule is used to obtain the target frame of the target person in the image to be processed. The target frame is used to select the area of ​​the target person in the corresponding image to be processed;

[0116] A detection submodule is used to detect the human skeleton key points of the target person in the target frame, and if the number of the detected human skeleton key points is greater than or equal to the number threshold, the detected human skeleton key point is determined to be the first human skeleton key point of the target person;

[0117] The failure determination submodule is used to determine that the detection of the first human skeleton key point of the target person has failed if the number of detected human skeleton key points is less than a number threshold.

[0118] In the embodiment of the present application, the generation module 504 may specifically include the following submodules:

[0119] The first storage submodule is used to store the first connection information of the target key point group into the corresponding position in the data set according to the human skeleton key point identifier corresponding to the target key point group, so as to generate the target data fingerprint of the target person.

[0120] In the embodiment of the present application, the result determination module 505 may specifically include the following submodules:

[0121] A comparison submodule, used to compare the target data fingerprint with each behavior data fingerprint in the behavior database respectively, to obtain a first similarity between the target data fingerprint and each behavior data fingerprint;

[0122] The result determination submodule is used to determine the behavior category corresponding to the behavior data fingerprint corresponding to the maximum value as the behavior recognition result of the target person if the maximum value among all the first similarities is greater than or equal to a preset threshold.

[0123] In the embodiment of the present application, the first comparison submodule may specifically include the following units:

[0124] A comparison unit, used for comparing the target data fingerprint with the elements located at the same position in any behavior data fingerprint one by one, to obtain a second similarity between the elements located at the same position;

[0125] The similarity determination unit is used to determine the first similarity between the target data fingerprint and the corresponding behavior data fingerprint according to the second similarity between the elements located at the same position and the preset weight of the target key point group corresponding to the second similarity.

[0126] In the embodiment of the present application, the determination module 505 may further include the following submodules:

[0127] A second storage submodule, configured to store the target data fingerprint into a category to be trained in the behavior database if the maximum value among all the first similarities is less than a preset threshold;

[0128] A receiving submodule, used for receiving the behavior category to which the image to be processed belongs, sent by the user;

[0129] an association submodule, for associating the target data fingerprint with the corresponding category in the behavior database if the behavior category to which the image to be processed belongs exists in the behavior database;

[0130] The third storage submodule is used to store the association relationship between the behavior category to which the image to be processed belongs and the target data fingerprint in the behavior database if the behavior category to which the image to be processed belongs does not exist in the behavior database.

[0131] The behavior recognition device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0132] Figure 6 Schematic diagram of the structure of the terminal device provided in the fourth embodiment of the present application. Figure 6 As shown, the terminal device 600 of this embodiment includes: at least one processor 610 ( Figure 6 Only one is shown in the figure) a processor, a memory 620, and a computer program 621 stored in the memory 620 and executable on the at least one processor 610, and when the processor 610 executes the computer program 621, the steps in the embodiment of the method for identifying any of the above-mentioned behaviors are implemented.

[0133] The terminal device 600 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 610 and a memory 620. Those skilled in the art will appreciate that Figure 6 It is only an example of the terminal device 600 and does not constitute a limitation on the terminal device 600. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0134] The processor 610 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0135] In some embodiments, the memory 620 may be an internal storage unit of the terminal device 600, such as a hard disk or memory of the terminal device 600. In other embodiments, the memory 620 may also be an external storage device of the terminal device 600, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 600. Further, the memory 620 may also include both an internal storage unit of the terminal device 600 and an external storage device. The memory 620 is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory 620 may also be used to temporarily store data that has been output or is to be output.

[0136] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0139] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0143] The present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing.

[0144] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for identifying a behavior, characterized in that: The identification method comprises: Obtain an image to be processed for a target person; Performing human skeleton key point detection on the image to be processed to obtain a first human skeleton key point of the target person; Based on the skeleton connection relationship between the first human skeleton key points, determine first connection information of a target key point group in the first human skeleton key points, where the target key point group includes any two human skeleton key points having the connection relationship; Generate a target data fingerprint of the target person according to first connection information of a target key point group in the first human skeleton key points; Determine the behavior recognition result of the target person according to the comparison result of the target data fingerprint and the behavior data fingerprint in the behavior database; Before generating the target data fingerprint of the target person according to the first connection information of the target key point group in the first human skeleton key point, the method further includes: Based on the symmetry between the human skeleton key points of the target person, partially symmetrical first human skeleton key points are extracted from the first human skeleton key points and merged according to the central axis of the human body to obtain merged human skeleton key points, and the merged human skeleton key points and the unmerged human skeleton key points in the first human skeleton key points are determined as second human skeleton key points, wherein the partially symmetrical first human skeleton key points include a left eye key point and a right eye key point symmetrical thereto, a left ear key point and a right ear key point symmetrical thereto, a left shoulder key point and a right shoulder key point symmetrical thereto, and a left hip key point and a right hip key point symmetrical thereto; Determine second connection information of a target key point group in the second human skeleton key points based on the skeleton connection relationship between the second human skeleton key points; Correspondingly, generating the target data fingerprint of the target person according to the first connection information of the target key point group in the first human skeleton key point includes: A target data fingerprint of the target person is generated according to the second connection information.

2. The identification method according to claim 1, characterized in that: The step of performing human skeleton key point detection on the image to be processed to obtain a first human skeleton key point of the target person includes: Obtaining a target frame of the target person in the image to be processed, wherein the target frame is used to frame and select an area of ​​the target person in the corresponding image to be processed; Performing human skeleton key point detection on the target person in the target frame, if the number of detected human skeleton key points is greater than or equal to a quantity threshold, determining the detected human skeleton key point as the first human skeleton key point of the target person; If the number of detected human skeleton key points is less than the number threshold, it is determined that the detection of the first human skeleton key point of the target person has failed.

3. The identification method according to claim 1, characterized in that: The step of generating a target data fingerprint of the target person according to the first connection information of the target key point group in the first human skeleton key point comprises: According to the human skeleton key point identifiers corresponding to the target key point group, the first connection information of the target key point group is stored in the corresponding position in the data set to generate the target data fingerprint of the target person.

4. The identification method according to claim 3, characterized in that: Determining the behavior recognition result of the target person according to the comparison result of the target data fingerprint and the behavior data fingerprint in the behavior database includes: Comparing the target data fingerprint with each behavior data fingerprint in the behavior database respectively to obtain a first similarity between the target data fingerprint and each behavior data fingerprint; If the maximum value among all the first similarities is greater than or equal to the preset threshold, the behavior category corresponding to the behavior data fingerprint corresponding to the maximum value is determined as the behavior recognition result of the target person.

5. The identification method according to claim 4, characterized in that: The step of comparing the target data fingerprint with each behavior data fingerprint in the behavior database to obtain a first similarity between the target data fingerprint and each behavior data fingerprint includes: Comparing the target data fingerprint with the elements located at the same position in any of the behavior data fingerprints one by one to obtain a second similarity between the elements located at the same position; The first similarity between the target data fingerprint and the corresponding behavior data fingerprint is determined according to the second similarity between the elements located at the same position and the preset weight of the target key point group corresponding to the second similarity.

6. The identification method according to claim 4, characterized in that: Determining the behavior recognition result of the target person according to the comparison result of the target data fingerprint and the behavior data fingerprint in the behavior database also includes: If the maximum value among all the first similarities is less than a preset threshold, storing the target data fingerprint in the category to be trained in the behavior database; Receiving the behavior category to which the image to be processed belongs, sent by the user; If the behavior category to which the image to be processed belongs exists in the behavior database, the target data fingerprint is associated with the corresponding category in the behavior database; If the behavior category to which the image to be processed belongs does not exist in the behavior database, the association relationship between the behavior category to which the image to be processed belongs and the target data fingerprint is stored in the behavior database.

7. A device for identifying student behavior, characterized in that: The identification device comprises: An acquisition module, used for acquiring an image to be processed for a target person; A recognition module is used to detect the key points of the human skeleton of the image to be processed to obtain the first key point of the human skeleton of the target person; An information determination module, used to determine first connection information of a target key point group in the first human skeleton key points based on the skeleton connection relationship between the first human skeleton key points, wherein the target key point group includes any two human skeleton key points having the connection relationship; A generating module, used for generating a target data fingerprint of the target person according to first connection information of a target key point group in the first human skeleton key points; A determination module, used to determine the behavior recognition result of the target person according to the comparison result of the target data fingerprint and the behavior data fingerprint in the behavior database; The identification device also includes: An extraction module is used to extract partially symmetrical first human skeleton key points from the first human skeleton key points based on the symmetry between the human skeleton key points of the target person, merge them according to the central axis of the human body, obtain merged human skeleton key points, and determine the merged human skeleton key points and the unmerged human skeleton key points in the first human skeleton key points as second human skeleton key points, wherein the partially symmetrical first human skeleton key points include a left eye key point and a symmetrical right eye key point, a left ear key point and a symmetrical right ear key point, a left shoulder key point and a symmetrical right shoulder key point, and a left hip key point and a symmetrical right hip key point; A connection determination module, used for determining second connection information of a target key point group in the second human skeleton key points based on the skeleton connection relationship between the second human skeleton key points; Correspondingly, the generating module is specifically used for: A target data fingerprint of the target person is generated according to the second connection information.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images