A trajectory heat map generation method, device, equipment and medium

Abstract

The application discloses a trajectory heat map generation method, device and equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing key point detection on images in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video; determining target key points of a target object from the part key points of the candidate object according to a preset scoring index; wherein the scoring index comprises at least one of the following: key point completeness of the part key points, position information of head key points in the part key points, and appearance probability of face key points in the part key points in the to-be-processed video; and generating a trajectory heat map of the target object in the to-be-processed video according to position information of target foot key points in the target key points. The above scheme improves the richness of the scoring index, avoids the influence of a single scoring index on the determination of the target key points, improves the accuracy of the determination of the target key points, and further improves the accuracy of the trajectory heat map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a trajectory heat map generation method and device, equipment and a medium. BACKGROUND

[0002] In the process of intelligent analysis of offline classrooms, the teacher's trajectory is an important reference index. How to accurately determine the trajectory heat map of the teacher is crucial. SUMMARY

[0003] The present application provides a trajectory heat map generation method, device, equipment and medium to accurately determine the trajectory heat map of the teacher.

[0004] According to one aspect of the present application, a trajectory heat map generation method is provided, comprising:

[0005] Key point detection is performed on the images in the to-be-processed video to obtain part key points of a candidate object in the to-be-processed video;

[0006] According to a preset scoring index, target key points of a target object are determined from the part key points of the candidate object; wherein the scoring index includes at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video;

[0007] According to the position information of a target foot key point in the target key points, a trajectory heat map of the target object in the to-be-processed video is generated.

[0008] According to another aspect of the present application, a trajectory heat map generation device is provided, comprising:

[0009] A part key point determination module is configured to perform key point detection on the images in the to-be-processed video to obtain part key points of a candidate object in the to-be-processed video;

[0010] A target key point determination module is configured to determine target key points of a target object from the part key points of the candidate object according to a preset scoring index; wherein the scoring index includes at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video;

[0011] A trajectory heat map generation module is configured to generate a trajectory heat map of the target object in the to-be-processed video according to the position information of a target foot key point in the target key points.

[0012] According to another aspect of the present application, there is provided an electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein

[0015] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for generating a trajectory heat map according to any one of the embodiments of the present application.

[0016] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to implement the method for generating a trajectory heat map according to any one of the embodiments of the present application when executed by the processor.

[0017] The method for generating a trajectory heat map provided by the embodiments of the present application comprises the following steps: performing key point detection on images in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video; determining target key points of a target object from the part key points of the candidate object according to a preset scoring index; wherein the scoring index comprises at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video; and generating a trajectory heat map of the target object in the to-be-processed video according to position information of a target foot key point in the target key points. The above scheme determines the target key points through at least one of the scoring indexes, improves the richness of the scoring index, avoids the influence of a single scoring index on the determination of the target key points, improves the accuracy of the determination of the target key points, and further improves the accuracy of the trajectory heat map. Meanwhile, the trajectory heat map generated according to the position information of the foot key point in the target key points can more accurately show the motion trajectory of the target object.

[0018] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0020] Figure 1A is a flowchart of a trajectory heat map generation method provided by an embodiment of the present application;

[0021] Figure 1B is a schematic diagram of a motion trajectory of a teacher's foot key point provided by an embodiment of the present application;

[0022] Figure 1C is a schematic diagram of a motion trajectory of a teacher's foot key point generated by affine transformation provided by an embodiment of the present application;

[0023] Figure 2A is a flowchart of a trajectory heat map generation method provided by an embodiment of the present application;

[0024] Figure 2B is an eye heat map provided by an embodiment of the present application;

[0025] Figure 2C is a part affinity field of an elbow and a wrist provided by an embodiment of the present application;

[0026] Figure 3 is a structural schematic diagram of a trajectory heat map generation device provided by an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of an electronic device for implementing a trajectory heat map generation method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", "target", "candidate", "effective" and the like in the description, claims, and drawings of the application are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment one

[0031] Figure 1A is a flowchart of a trajectory heat map generation method provided by an embodiment of the application. The embodiment can be applicable to how to generate a trajectory heat map, and specifically can be applicable to how to generate a teacher's trajectory heat map in a teaching scenario. The method can be executed by a trajectory heat map generation device, which can be realized in the form of hardware and / or software and can be integrated into an electronic device, such as a server, that carries a trajectory heat map generation function.

[0032] Referring to Figure 1A the trajectory heat map generation method, the method comprises:

[0033] S110, performing key point detection on an image in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video.

[0034] The to-be-processed video refers to a video containing an image that needs to be subjected to key point detection. Optionally, the to-be-detected video can be a video of a teacher teaching collected by a monitoring camera in a classroom. The embodiment of the application does not make any limitation on the type and installation position of the monitoring camera, which can be selected by a user as needed. Illustratively, the monitoring camera can be installed on a rear wall in the classroom.

[0035] The key point detection refers to detecting key points of an image in a to-be-processed video.

[0036] The candidate object refers to an object contained in an image in a to-be-processed video. For example, in a teaching scenario of a teacher, the candidate object can be the teacher and the students in the classroom.

[0037] The part key points refer to points of parts of the candidate object. In the previous example, the part key points can be points representing body parts of the teacher and the student in the classroom, such as head key points (eyes, mouth), joint key points (shoulders, elbows, wrists), foot key points, and the like.

[0038] Specifically, a preset key point detection algorithm can be used to detect key points of each frame of image in the to-be-processed video, so as to obtain the part key points of the candidate object in each frame of image in the to-be-processed video.

[0039] In S120, a target key point of a target object is determined from the part key points of the candidate object according to a preset scoring index.

[0040] The scoring index refers to an index for screening the part key points of the candidate object to obtain the target key point of the target object. Optionally, the scoring index includes at least one of the following: a key point completeness of the part key points, position information of a head key point in the part key points, and an appearance probability of a face key point in the part key points in the to-be-processed video. The key point completeness of the part key points refers to a completeness of a number of the part key points of the candidate object for each frame of image in the to-be-processed video. The position information of the head key point in the part key points refers to position coordinates of the head key point in the part key points. The appearance probability of the face key point in the part key points in the to-be-processed video refers to a ratio between a number of appearances of the face key point (eye key point and mouth key point) of the candidate object on each frame of image in the to-be-processed video and a number of appearances of all the part key points of the candidate object.

[0041] The target object refers to an object for which a trajectory heat map needs to be generated. For example, if the trajectory of a teacher in a class is to be studied, the teacher is the target object.

[0042] The target key point refers to a part key point on the target object. In the previous example, the teacher is the target object, and the target key point is a part key point representing a body part of the teacher.

[0043] In an optional embodiment, the target key point of the target object can be determined from the part key points of the candidate object based on a key point completeness of the part key points. Specifically, for each frame of image in the to-be-processed video, for any candidate object in the frame of image, a ratio between a number of the part key points of the candidate object and a preset value is taken as the key point completeness. If the key point completeness is greater than a completeness threshold, the candidate object in the frame of image is taken as the target object, and the part key point corresponding to the target object is taken as the target key point of the target object. The present embodiment does not make any limitation on the size of the preset value and the size of the completeness threshold, which can be set according to experience or determined through a large number of experiments.

[0044] In an optional embodiment, the target key point of the target object can be determined from the part key points of the candidate object based on the position information of the head key point in the part key points. Specifically, for each frame image in the video to be processed, for any candidate object in the image, the position information of the head key point of the candidate object is determined, and if the position information is higher than a position threshold, the candidate object is taken as the target object, and the part key point corresponding to the target object is taken as the target key point of the target object. The size of the position threshold is not limited in the embodiment of the application, which can be set by technicians according to experience or determined repeatedly through a large number of experiments. It should be noted that in a teaching scene, since the teacher is often in a standing state and the students are often in a sitting state, the position information of the head key point of the target object (the teacher) is higher than that of the students (other candidate objects), and the candidate object with higher position information of the head key point is taken as the target object.

[0045] In an optional embodiment, the target key point of the target object can be determined from the part key points of the candidate object based on the position information of the head key point in the part key points. Specifically, for each frame image in the video to be processed, for any candidate object in the image, the position information of the head key point of the candidate object is determined, and if the position information is higher than a position threshold, the candidate object is taken as the target object, and the part key point corresponding to the target object is taken as the target key point of the target object. The size of the position threshold is not limited in the embodiment of the application, which can be set by technicians according to experience or determined repeatedly through a large number of experiments. It should be noted that in a teaching scene, since the teacher is often in a standing state and the students are often in a sitting state, the position information of the head key point of the target object (the teacher) is higher than that of the students (other candidate objects), and the candidate object with higher position information of the head key point is taken as the target object.

[0046] In an optional embodiment, the target key point of the target object can be determined from the part key points of the candidate object based on the position information of the head key point in the part key points. Specifically, for each frame image in the video to be processed, for any candidate object in the image, the position information of the head key point of the candidate object is determined, and if the position information is higher than a position threshold, the candidate object is taken as the target object, and the part key point corresponding to the target object is taken as the target key point of the target object. The size of the position threshold is not limited in the embodiment of the application, which can be set by technicians according to experience or determined repeatedly through a large number of experiments. It should be noted that in a teaching scene, since the teacher is often in a standing state and the students are often in a sitting state, the position information of the head key point of the target object (the teacher) is higher than that of the students (other candidate objects), and the candidate object with higher position information of the head key point is taken as the target object.

[0047] In an optional embodiment, the target key point of the target object can be determined from the part key points of the candidate object based on the position information of the head key point in the part key points. Specifically, for each frame image in the video to be processed, for any candidate object in the image, the position information of the head key point of the candidate object is determined, and if the position information is higher than a position threshold, the candidate object is taken as the target object, and the part key point corresponding to the target object is taken as the target key point of the target object. The size of the position threshold is not limited in the embodiment of the application, which can be set by technicians according to experience or determined repeatedly through a large number of experiments. It should be noted that in a teaching scene, since the teacher is often in a standing state and the students are often in a sitting state, the position information of the head key point of the target object (the teacher) is higher than that of the students (other candidate objects), and the candidate object with higher position information of the head key point is taken as the target object.

[0048] The prediction score refers to the probability that any candidate object is the target object.

[0049] Specifically, for each image in the to-be-processed video, for any candidate object in the image, the key point integrity of the part key point of the candidate object, the position information of the head key point in the part key point, and the appearance probability of the face key point in the part key point in the to-be-processed video are normalized, that is, the key point integrity of the part key point is converted into a score between 0 and 1, the position information of the head key point in the part key point is converted into a score between 0 and 1, the appearance probability of the face key point in the part key point in the to-be-processed video is converted into a score between 0 and 1, and the three scores are added to obtain a prediction score. Then, if the prediction score is greater than a set threshold, the candidate object is taken as a target object, and the part key point corresponding to the target object is taken as a target key point of the target object. The size of the set threshold is not limited in the embodiment of the present application, which can be set according to experience or determined repeatedly through a large number of tests.

[0050] It can be understood that the prediction score is determined by the key point integrity of the part key point, the position information of the head key point in the part key point, and the appearance probability of the face key point in the part key point in the to-be-processed video, which avoids the problem that the prediction score is inaccurate when the prediction score is determined according to a single score index, thereby improving the accuracy of the determination of the target key point.

[0051] S130, generating a trajectory heat map of the target object in the to-be-processed video according to the position information of the target foot key point in the target key point.

[0052] The trajectory heat map refers to a schematic diagram for representing the motion trajectory of the target object in the to-be-processed video.

[0053] In an optional embodiment, the trajectory heat map of the target object in the to-be-processed video can be generated according to the position information of the target foot key point in the target key point based on a preset trajectory heat map generation rule. For example, the trajectory corner point of the target object can be determined from the target foot key point according to the position information of the target foot key point in the target key point, and the trajectory heat map of the target object in the to-be-processed video can be generated by performing affine transformation on the target foot key point according to the trajectory corner point.

[0054] The trajectory corner point refers to a foot key point in the trajectory point of the target object in the to-be-processed video, which can represent the starting point, the end point, or the inflection point of the trajectory.

[0055] Specifically, the target foot key point corresponding to the minimum or maximum horizontal coordinate or vertical coordinate in the position information of the target foot key point in the target key point can be determined as the trajectory corner point of the target object. For example,Figure 1B The schematic diagram of the motion trajectory of the foot key points of the teacher is shown, and the trajectory corner points can be the upper left corner foot key point, the lower left corner foot key point, the upper right corner foot key point and the lower right corner foot key point of the target foot key point. Further, according to the trajectory corner points, affine transformation is performed on the target foot key point to generate the trajectory heat map of the target object in the video to be processed. Exemplarily, in the teaching scene, the teacher is the target object, and refer to FIG. 2 for a description. Figure 1C The schematic diagram of the motion trajectory of the foot key points of the teacher after affine transformation is shown.

[0056] It can be understood that the trajectory heat map generated by affine transformation makes the trajectory heat map of the target object more three-dimensional, so that the motion trajectory of the target object can be more intuitively understood.

[0057] The trajectory heat map generation scheme provided by the embodiment of the present application can obtain the part key points of the candidate object in the video to be processed by performing key point detection on the images in the video to be processed; determine the target key points of the target object from the part key points of the candidate object according to a preset scoring index; wherein the scoring index includes at least one of the following: key point completeness of the part key point, position information of the head key point in the part key point, and appearance probability of the face key point in the part key point in the video to be processed; and generate the trajectory heat map of the target object in the video to be processed according to the position information of the target foot key point in the target key point. The above scheme determines the target key points through at least one of the scoring indexes, improves the richness of the scoring index, avoids the influence of a single scoring index on the determination of the target key points, improves the accuracy of the determination of the target key points, and further improves the accuracy of the trajectory heat map. Meanwhile, according to the position information of the foot key point in the target key point, the trajectory heat map generated can more accurately show the motion trajectory of the target object.

[0058] In the embodiment, each frame of image in a complete video to be processed is analyzed to track the candidate object in the image. Specifically, the key points in each image are found, and the key points belonging to the same candidate object are framed to obtain the ROI (Region of Interest, region of interest). The ROIs of the continuous two frames of images are overlapped, and if the ROI of the intersection part is greater than a preset area threshold, it indicates that the key points on the two frames of images belong to the same candidate object, and the candidate object moves on the two frames of images; if the ROI of the intersection part is less than or equal to the preset area threshold, it indicates that the key points on the two frames of images do not belong to the same candidate object. The size of the preset area threshold is not limited in the embodiment of the present application, which can be set by the technical personnel according to experience, or can be repeatedly determined through a large number of experiments. Exemplarily, the preset area threshold can be 0.3.

[0059] Embodiment Two

[0060] Figure 2A is a flowchart of a trajectory heat map generation method provided by Embodiment Two of the present application. Based on the above-mentioned embodiment, the present embodiment further refines the operation of "detecting key points in the images in the to-be-processed video to obtain part key points of the candidate object in the to-be-processed video" into "for each frame image in the to-be-processed video, input the image into a pose estimation network to obtain a first number of key point heat maps and a second number of key point connection maps corresponding to the image; for each key point heat map, filter the key points in the key point heat map to obtain valid key points; and according to the key point connection map, match the valid key points in the to-be-processed video to obtain the part key points of the candidate object". It should be noted that the parts not described in detail in the present embodiment can be referred to the descriptions of other embodiments.

[0061] Referring to Figure 2A the trajectory heat map generation method, the method comprises the following steps.

[0062] S210, for each frame image in the to-be-processed video, input the image into a pose estimation network to obtain a first number of key point heat maps and a second number of key point connection maps corresponding to the image.

[0063] The pose estimation network refers to a network for key point estimation of the images in the to-be-processed video. Optionally, the pose estimation network comprises a backbone neural network and a convolutional neural network. The backbone neural network is used to extract an initial key point heat map and an initial key point connection map from the input image in the to-be-processed video. The convolutional neural network is used to refine the initial key point heat map and the initial key point connection map to obtain a final key point heat map and a final key point connection map.

[0064] It should be noted that the present embodiment does not make any limitation on the selection of the backbone neural network, which can be selected by the skilled person according to experience. For example, the backbone neural network can be selected as a MobileNetv1 network structure (a kind of lightweight network structure). The advantage of selecting the MobileNetv1 network structure is that the network structure is less and the running speed is faster. The present embodiment does not make any limitation on the selection of the convolutional neural network, which can be selected by the skilled person according to experience. For example, the convolutional neural network can be selected as a CPM (Convolutional Pose Machines, a kind of fully convolutional network).

[0065] The key point heat map refers to the possibility of the position key point of the candidate object in each frame image in the video to be processed, for example, the key point heat map of the head key point, that is, in the scene of the teacher teaching, the teacher and the student are taken as the candidate object, and the eyes of the student and the teacher are taken as an example, which can be seen from the eye heat map shown in Figure 2B The key point heat map is understood. The white point indicates the eye of the teacher and / or the student, the whiter the color of the point is, the greater the probability that the point is the eye is, and the lighter the color of the point is, the smaller the probability that the point is the eye is.

[0066] The key point connection line map refers to the possibility of the connection line between any two position key points, for example, the possibility of the connection line between the hand key point and the elbow key point; optionally, the key point connection line map can be a PAFS (Partial Affinity Field), which can be seen from the elbow and wrist PAFS (Partial Affinity Field) shown in Figure 2C The key point connection line map is understood. The first quantity and the second quantity in the embodiment of the application are set according to actual business requirements, and exemplarily, if 18 position key points can represent an object, the first quantity can be 18, and the second quantity can be 38.

[0067] In this embodiment, specifically, the pose estimation network can be built through the MobileNetv1 network structure and three convolutional neural networks. For each frame image in the video to be processed, the image is input into the built pose estimation network, and the first quantity of key point heat maps and the second quantity of key point connection line maps corresponding to the image are obtained. It should be noted that in the refinement stage through the convolutional neural network in the pose estimation network, two refinements are completed. The advantage of only performing two refinements is to improve the calculation speed of the pose estimation network.

[0068] In S220, for each key point heat map, the key points in the key point heat map are screened to obtain effective key points.

[0069] In this embodiment, the effective key point refers to a key point in the key point heat map that can be used for subsequent determination of the position key point.

[0070] In an optional embodiment, for each key point heat map, the key points in the key point heat map can be screened based on a preset screening rule to obtain effective key points.

[0071] In an optional embodiment, for each key point heat map, the key points in the key point heat map can be filtered according to the probability values of the key points in the key point heat map to obtain first candidate key points; for each first candidate key point, the first candidate key point and other first candidate key points in a neighboring range of the first candidate key point are compared to determine second candidate key points; and effective key points are determined from the second candidate key points of the key point heat map according to pixel distances between the second candidate key points.

[0072] The first candidate key point refers to a candidate key point obtained by filtering according to the probability values of the key points in the key point heat map. The second candidate key point refers to a candidate key point obtained by comparing according to the probability values of the first candidate key points.

[0073] Specifically, for each key point heat map, if the probability value of a key point in the key point heat map is greater than a first threshold, the key point is a first candidate key point; if the probability value of the key point in the key point heat map is less than or equal to the first threshold, the key point is not a first candidate key point. The size of the first threshold is not limited in the embodiments of the present application, which can be set according to experience or determined repeatedly through a large number of experiments. For example, the first threshold can be 0.1.

[0074] After the first candidate key points are obtained, for each first candidate key point, if the probability value of another first candidate key point in a neighboring range of the first candidate key point is greater than the probability value of the first candidate key point, the other first candidate key point is determined as a second candidate key point; if the probability value of the other first candidate key point in the neighboring range of the first candidate key point is less than or equal to the probability value of the first candidate key point, the first candidate key point is determined as a second candidate key point. The neighboring range is not limited in the embodiments of the present application, which can be set according to experience. For example, the neighboring range can be other first candidate key points above, below, left and right of any first candidate key point, and the neighboring range can also be other key points in a preset region containing any first candidate key point. The shape and size of the preset region are not limited in the embodiments of the present application, which can be a rectangle or a circle, etc.

[0075] After the second candidate key points are obtained, the effective key points are determined from the second candidate key points of the key point heat map according to the pixel distances between the second candidate key points. Specifically, if the pixel distance between two second candidate key points is less than a pixel distance threshold, the probability values of the two second candidate key points are compared, and the second candidate key point with the larger probability value is retained as the effective key point. The size of the pixel distance threshold is not limited in the embodiments of the present application, which can be set according to experience or determined through repeated tests.

[0076] It can be understood that the first candidate key points are obtained through preliminary screening, the second candidate key points are obtained through screening of the first candidate key points, and the effective key points are determined from the second key points, so that the effective key points are more accurate through layer-by-layer screening and gradual reduction of the screening range.

[0077] In S230, the effective key points in the to-be-processed video are matched according to the key point connection line map, and the part key points of the candidate object are obtained.

[0078] In an optional embodiment, the effective key points in the to-be-processed video can be matched according to the key point connection line map based on a preset matching rule, and the part key points of the candidate object are obtained.

[0079] In an optional embodiment, for each image in the to-be-processed video, the effective key points in the image are matched according to the key point connection line map, and the candidate key point combination of the candidate object in the image is obtained; the target key point combination is determined from the candidate key point combination based on the Hungarian algorithm; and the part key points of the candidate object in the image are determined according to the target key point combination.

[0080] The candidate key point combination refers to the key point combination of the effective key points that may belong to the mutually related parts of the same object. The target key point combination refers to the key point combination obtained by screening the candidate key point combination.

[0081] Specifically, the effective key points of any one part are combined with the effective key points of another part associated with the effective key points, two effective key points in any one group of combinations are connected, a preset threshold number of pixel points on the connection line are found, the probability values of the preset threshold number of pixel points are found from the key point connection graph associated with the effective key points, the mean value of the probability values of the preset threshold number of pixel points is calculated, and if the mean value of the probability values of the preset threshold number of pixel points is greater than a probability threshold value, the combination of the two effective key points is taken as a candidate key point combination. The size of the preset threshold and the probability threshold is not limited in the embodiments of the present application, which can be set by technicians according to experience, or can be repeatedly determined through a large number of experiments, for example, the preset threshold is 10 and the probability threshold is 0.5. For example, in a multi-target scene, there are multiple candidate objects in an image, for example, for the key point connection graph of the elbow and the wrist, there are multiple candidate objects, that is, there are multiple groups of elbow and wrist connections. For the effective key point corresponding to a certain elbow, the effective key point of the elbow is combined with the effective key point of the other wrist, and at least one group of the elbow and the other wrist is obtained. Then, the connection between the effective key points of the elbow and the other wrist in each group is determined, for each connection line, 10 pixel points on the connection line are found, the probability values of the 10 pixel points are found from the key point connection graph of the elbow and the corresponding wrist, the mean value of the probability values of the 10 pixel points is calculated, and if the mean value of the probability values of the 10 pixel points is greater than 0.5, the effective key point of the elbow and the effective key point of the corresponding wrist are combined as a candidate key point combination. It should be noted that one elbow corresponds to one effective key point.

[0082] In the candidate key point combination obtained above, there may be overlapping, that is, the effective key point of one part is combined with at least two effective key points of the same part associated with each other, for example, in the scene of teachers teaching, in the case of multiple people in the scene, one left eye is combined with two right eyes at the same time, and the candidate key point combination needs to be further screened. Therefore, based on the Hungarian algorithm, the target key point combination is determined from the candidate key point combination. Specifically, the effective key points of different parts having a connection relationship in the candidate key point combination are connected to form an undirected graph, and the Hungarian algorithm is used to obtain the target key point combination.

[0083] After obtaining the target key point combination, it can be determined which target key point combination belongs to the same candidate object, and then for each candidate object, the part key point in the target key point combination can be taken as the part key point of the candidate object.

[0084] It can be understood that by matching the effective key points, obtaining the candidate key point combination, and screening the target key point combination from the candidate key point combination, the screening range is gradually reduced, so that the part key points of the determined candidate object are more accurate.

[0085] In this embodiment, before the target key point combination is determined based on the Hungarian algorithm, the candidate key point combination can be further screened.

[0086] In an optional embodiment, after the effective key points in the image are matched according to the key point connection graph to obtain the candidate key point combination of the candidate object in the image, the method further includes: screening the candidate key point combination according to the distance between the key points.

[0087] Specifically, for any effective key point, a certain number of other effective key points having a connection relationship with the effective key point are obtained through the probability threshold, that is, a certain number of candidate key point combinations, the candidate key point combinations are sorted according to the distance between the two effective key points in each candidate key point combination, and the candidate key point combination in the middle position is selected, and the other candidate key point combinations in the certain number are screened according to the distance between the two effective key points in the selected candidate key point combination. For any other candidate key point combination, if the distance between the two effective key point combinations in the other candidate key point combination is greater than a set multiple of the distance between the two effective key points in the selected candidate key point combination, the other candidate key point combination is removed and no longer processed; otherwise, it is retained. It should be noted that the size of the certain number is determined according to the method of obtaining the candidate key point combination through the probability threshold. The certain number can be odd or even. For example, if the certain number is five, the middle position is three, and the candidate key point combination in the third position is selected; if the certain number is six, the middle position is three and four, and the two candidate key point combinations in the third and fourth positions are selected, and the distances of the two candidate key point combinations are averaged, and the other candidate key point combinations are screened according to the obtained distance average. The set multiple can be set by a person skilled in the art according to the actual situation. Continuing the previous example, if the obtained candidate key point combination containing the effective key point corresponding to the elbow and the effective key point corresponding to the wrist is five groups, the distance between the effective key point corresponding to the elbow and the effective key point corresponding to the wrist in each group is determined; the five candidate key point combinations are sorted according to the distance, the candidate key point combination in the middle position is determined, and the distance corresponding to the candidate key point combination in the middle position is taken as the screening distance; the other candidate key point combinations in the five candidate key point combinations are screened according to the screening distance: specifically, the candidate key point combination with a distance greater than the set multiple of the screening distance is removed from the five candidate key point combinations.

[0088] It can be understood that, by screening the candidate key point combination through the distance between the key points, the key point combination of two effective key points far apart can be avoided to be taken as a candidate key point combination for subsequent processing, the reliability of the candidate key point combination is improved, and the operation amount is reduced.

[0089] S240, determining a target key point of the target object from the part key points of the candidate object according to a preset scoring index.

[0090] S250, generating a trajectory heat map of the target object in the video to be processed according to position information of a target foot key point in the target key point.

[0091] The trajectory heat map generation scheme provided by the embodiment of the application comprises: inputting each image in the video to be processed into a pose estimation network to obtain a first number of key point heat maps and a second number of key point connection line maps corresponding to the image; screening key points in each key point heat map to obtain effective key points; matching the effective key points in the video to be processed according to the key point connection line maps to obtain part key points of a candidate object; determining a target key point of a target object from the part key points of the candidate object according to a preset scoring index; and generating a trajectory heat map of the target object in the video to be processed according to position information of a target foot key point in the target key point. The above scheme traverses all the key points on the key point heat map, avoids key point omission, makes the determination of the target key point more accurate, and improves the accuracy of the determination of the target key point.

[0092] In the embodiment, the principle of the pose estimation network is similar to that of a bottom-up detection method. Specifically, the bottom-up detection method refers to detecting part key points of all candidate objects on images in the video to be processed, and then grouping them according to candidate object instances.

[0093] In contrast to the bottom-up detection method, there is a top-down detection method. The top-down detection method refers to applying a detector, and then applying a single object pose estimation algorithm on each detected candidate object to decouple the pose estimation problem into two sub-problems.

[0094] It should be noted that, compared with the top-down detection method, the bottom-up detection method is faster when there are more candidate objects, and the bottom-up detection method detects part key points once and does not need to run the pose estimation of each candidate object again.

[0095] Embodiment three

[0096] Figure 3is a structural schematic diagram of a trajectory heat map generation device provided by Embodiment Three of the present application. The present embodiment can be applied to the case of how to generate a trajectory heat map, and specifically can be applied to the case of how to generate a trajectory heat map of a teacher in a teaching scenario. The trajectory heat map generation device can be realized in the form of hardware and / or software, and can be integrated in an electronic device that carries a function of generating a trajectory heat map, such as a server. As shown in Figure 3 the trajectory heat map generation device of the present embodiment includes a part key point determination module 310, a target key point determination module 320, and a trajectory heat map generation module 330. Among them,

[0097] The part key point determination module 310 is configured to perform key point detection on images in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video.

[0098] The target key point determination module 320 is configured to determine target key points of a target object from the part key points of the candidate object according to a preset scoring index. The scoring index includes at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video.

[0099] The trajectory heat map generation module 330 is configured to generate a trajectory heat map of the target object in the to-be-processed video according to position information of a target foot key point in the target key points.

[0100] The trajectory heat map generation scheme provided by the present embodiment includes the part key point determination module performing key point detection on images in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video, the target key point determination module determining target key points of a target object from the part key points of the candidate object according to a preset scoring index, wherein the scoring index includes at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video, and the trajectory heat map generation module generating a trajectory heat map of the target object in the to-be-processed video according to position information of a target foot key point in the target key points. The above scheme determines the target key points through at least one of the scoring indexes, improves the richness of the scoring indexes, avoids the influence of a single scoring index on the determination of the target key points, improves the accuracy of the determination of the target key points, and further improves the accuracy of the trajectory heat map. Meanwhile, the trajectory heat map generated according to the position information of the foot key point in the target key points can more accurately show the motion trajectory of the target object.

[0101] Optionally, the part key point determination module 310 includes:

[0102] The image information obtaining unit is configured to, for each frame image in the video to be processed, input the image into a pose estimation network to obtain a first number of key point heat maps and a second number of key point connection maps corresponding to the image.

[0103] The effective key point obtaining unit is configured to, for each key point heat map, screen the key points in the key point heat map to obtain effective key points.

[0104] The part key point obtaining unit is configured to, according to the key point connection maps, match the effective key points in the video to be processed to obtain part key points of the candidate object.

[0105] Optionally, the effective key point obtaining unit is specifically configured to:

[0106] For each key point heat map, the key points in the key point heat map are screened according to the probability values of the key points in the key point heat map to obtain first candidate key points.

[0107] For each first candidate key point, the first candidate key point and other first candidate key points in a neighboring range of the first candidate key point are compared to determine second candidate key points.

[0108] According to pixel distances between the second candidate key points, the effective key points are determined from the second candidate key points of the key point heat map.

[0109] Optionally, the part key point obtaining unit is specifically configured to:

[0110] For each image in the video to be processed, the effective key points in the image are matched according to the key point connection maps to obtain a candidate key point combination of the candidate object in the image.

[0111] The target key point combination is determined from the candidate key point combination based on a Hungarian algorithm.

[0112] According to the target key point combination, the part key points of the candidate object in the image are determined.

[0113] Optionally, the part key point determining module 310 further includes a key point screening unit, which is configured to:

[0114] After the effective key points in the image are matched according to the key point connection maps to obtain a candidate key point combination of the candidate object in the image, the candidate key point combination is screened according to distances between the key points.

[0115] Optionally, the target key point determining module 320 is specifically configured to:

[0116] According to the key point integrity of the part key points, position information of head key points in the part key points, and appearance probability of face key points in the part key points in the to-be-processed video, a prediction score is determined.

[0117] According to the prediction score, target key points of the target object are determined from the part key points of the candidate object.

[0118] Optionally, the trajectory heat map generation module 330 is specifically configured to:

[0119] According to the position information of the target foot key points in the target key points, trajectory corner points of the target object are determined from the target foot key points.

[0120] According to the trajectory corner points, the target foot key points are subjected to affine transformation to generate a trajectory heat map of the target object in the to-be-processed video.

[0121] The trajectory heat map generation device provided in the embodiments of the present application can execute the trajectory heat map generation method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the trajectory heat map generation method.

[0122] In the technical solution of the present application, the collection, storage, use, processing, transmission, provision and disclosure of the part key points and target key points and the like are in line with the relevant legal regulations and do not violate public order and good customs.

[0123] Embodiment Four

[0124] Figure 4 Fig. 1 is a structural schematic diagram of an electronic device 10 for implementing the trajectory heat map generation method according to Embodiment Four of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown in the figure, their connections, and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0125] As Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication. The memory stores a computer program executable by the at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0126] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0127] The processor 11 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the trajectory heat map generation method.

[0128] In some embodiments, the trajectory heat map generation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the trajectory heat map generation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the trajectory heat map generation method by any other appropriate means, such as by means of firmware.

[0129] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0130] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0131] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0133] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0134] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0135] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0136] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating a trajectory heatmap, characterized in that, The method comprises: performing key point detection on images in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video; determining target key points of a target object from the part key points of the candidate object according to a preset scoring index; wherein the scoring index comprises at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video; generating a trajectory heat map of the target object in the to-be-processed video according to position information of a target foot key point in the target key points. The method comprises: for each image in the to-be-processed video, inputting the image into a pose estimation network to obtain a first number of key point heat maps and a second number of key point connection line maps corresponding to the image; wherein the pose estimation network is a network used for key point estimation on images in the to-be-processed video; for each key point heat map, performing screening on key points in the key point heat map to obtain effective key points; matching the effective key points in the to-be-processed video according to the key point connection line maps to obtain part key points of the candidate object.

2. The method of claim 1, wherein, The method comprises: for each key point heat map, performing screening on key points in the key point heat map according to probability values of the key points in the key point heat map to obtain first candidate key points; for each first candidate key point, comparing the first candidate key point with other first candidate key points in a neighboring range of the first candidate key point to determine second candidate key points; determining effective key points from the second candidate key points in the key point heat map according to pixel distances between the second candidate key points.

3. The method of claim 1, wherein, The method comprises: for each image in the to-be-processed video, matching effective key points in the image according to the key point connection line maps to obtain a candidate key point combination of a candidate object in the image; determining a target key point combination from the candidate key point combinations based on a Hungarian algorithm; determining part key points of the candidate object in the image according to the target key point combination.

4. The method of claim 3, wherein, The method further comprises: performing screening on the candidate key point combination according to distances between key points.

5. The method of claim 1, wherein, The method comprises: determining a prediction score according to the key point completeness of the part key points, the position information of the head key point in the part key points, and the appearance probability of the face key point in the part key points in the to-be-processed video; and determining the target key points of the target object from the part key points of the candidate object according to the prediction score. According to the prediction score, a target key point of the target object is determined from the part key points of the candidate object.

6. The method of claim 1, wherein, The generating the trajectory heat map of the target object in the to-be-processed video according to the position information of the target foot key point in the target key point comprises: According to the position information of the target foot key point in the target key point, a trajectory corner point of the target object is determined from the target foot key point; According to the trajectory corner point, the target foot key point is subjected to affine transformation to generate the trajectory heat map of the target object in the to-be-processed video.

7. A device for generating a trajectory heatmap, characterized in that, Comprise: The part key point determination module is configured to perform key point detection on images in a to-be-processed video to obtain part key points of a candidate object in the to-be-processed video. The target key point determination module is configured to determine a target key point of a target object from the part key points of the candidate object according to a preset scoring index; wherein the scoring index comprises at least one of the following: key point completeness of the part key points, position information of a head key point in the part key points, and appearance probability of a face key point in the part key points in the to-be-processed video. The trajectory heat map generation module is configured to generate a trajectory heat map of the target object in the to-be-processed video according to position information of a target foot key point in the target key point. The part key point determination module comprises: The image information obtaining unit is configured to, for each frame of image in the to-be-processed video, input the image into a pose estimation network to obtain a first number of key point heat maps and a second number of key point connection line maps corresponding to the image; wherein the pose estimation network refers to a network used for key point estimation on images in the to-be-processed video. The effective key point obtaining unit is configured to, for each key point heat map, screen key points in the key point heat map to obtain effective key points. The part key point obtaining unit is configured to match the effective key points in the to-be-processed video according to the key point connection line maps to obtain part key points of the candidate object.

8. An electronic device, comprising: The electronic device comprises: At least one processor; and The memory is in communication with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the trajectory heat map generation method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the trajectory heat map generation method in any one of claims 1-6 when executed.

Images