Thermodynamic diagram track generation method, device and equipment and readable storage medium

Abstract

The invention discloses a thermodynamic diagram track generation method, apparatus and device, and a readable storage medium. The method comprises the steps of obtaining first image data and second image data in a teaching scene in real time by using a first camera and a second camera respectively; extracting a first target feature in the first image data and constructing a target feature library based on the first target feature; extracting a second target feature in the second image data, and comparing the target feature library with the second target feature to obtain an object feature of which the feature similarity exceeds a preset threshold; obtaining a thermodynamic diagram space having a mapping relationship with the second image data; and generating a thermodynamic diagram track of the to-be-measured object based on the mapping relation and the coordinates. According to the method, the target feature library is automatically identified and constructed through the first camera, the to-be-detected object is identified through the second camera, and then the thermodynamic diagram track corresponding to the motion track of the to-be-detected object is generated in the thermodynamic diagram space having the mapping relation with the second image data.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and readable storage medium for generating a thermal map trajectory. Background Art

[0002] As the country vigorously promotes the popularization and construction of intelligent teaching environment, the use of multimedia technology to assist teaching has become a trend in today's classroom teaching. Using image processing technology to evaluate teachers' behavior in the classroom, specifying quantitative indicators for teachers' teaching and providing scientific suggestions to teachers based on the analysis results is an important research field in the current intelligent classroom teaching.

[0003] In classroom teaching scenarios, teachers usually walk around the classroom, answer students' questions, and participate in classroom interactions. By analyzing the teacher's classroom patrol trajectory over a long period of time, we can effectively understand the areas with high classroom participation or teachers' attention, help teachers analyze the quality of classroom teaching, and whether they have accidentally ignored students in some areas.

[0004] In recent years, more and more classrooms have installed cameras to collect images of the teaching process in the classroom, and use computer vision technology to analyze the classroom behavior of students and teachers in the images. However, existing methods, whether using target detection, key point detection or video classification algorithms for behavior recognition, focus on the recognition of the behavior itself, and there are few methods for macro analysis of the teacher's patrol trajectory in the classroom, or methods for intuitively analyzing the teacher's patrol trajectory by generating heat maps.

[0005] When existing methods generate behavior or trajectory heat maps, they do not need to distinguish the roles or identities in the scene. However, in classroom scenes, there is usually a distinction between the roles of teachers and students. How to adaptively distinguish the target roles in the scene and output the teacher's trajectory in a targeted manner is a major difficulty. At the same time, the existing technology directly generates heat maps in the image space, which will retain the problem of low relevance to the real scene due to the perspective transformation of camera imaging (near is large and far is small). Summary of the invention

[0006] The present application provides a method, device, equipment and readable storage medium for generating a heat map trajectory, which can solve the above-mentioned problems in the prior art.

[0007] The first aspect of the present application provides a method for generating a heat map trajectory, the method comprising:

[0008] Using a first camera to acquire first image data in a teaching scene in real time, and using a second camera to acquire second image data in a teaching scene in real time; wherein the first camera and the second camera are arranged at two ends of the teaching scene, and the first image data and the second image data partially overlap;

[0009] Extracting a first target feature from the first image data to construct a target feature library based on the first target feature;

[0010] Extracting a second target feature from the second image data, comparing the target feature library with the second target feature to obtain an object feature whose feature similarity exceeds a preset threshold; wherein the object feature represents the coordinates of the object to be measured in the second image data;

[0011] Acquire a thermal map space of the teaching scene, wherein the thermal map space has a mapping relationship with the second image data;

[0012] Based on the mapping relationship and coordinates, a thermal map trajectory of the object to be tested is generated.

[0013] Furthermore, the step of extracting the first target feature in the first image data to construct a target feature library based on the first target feature includes:

[0014] Performing data classification on the first image data based on the target detection model to obtain first preset category data in the first image data;

[0015] Performing feature extraction on the first preset category data based on the feature extraction model to obtain a first target feature;

[0016] Clustering the first target feature to obtain a target feature cluster;

[0017] Construct a target feature library based on target feature clusters.

[0018] Furthermore, the step of clustering the first target feature to obtain a target feature cluster includes:

[0019] Clustering the first target feature based on the clustering model, and outputting at least two feature clusters;

[0020] The number of cluster features of at least two feature clusters is obtained, and the at least two feature clusters are sorted based on the number of cluster features, so as to use the feature cluster with the largest number of cluster features as the target feature cluster.

[0021] Further, the step of extracting a second target feature from the second image data and comparing the target feature library with the second target feature to obtain an object feature whose feature similarity exceeds a preset threshold comprises:

[0022] Performing data classification on the second image data based on the target detection model to obtain second preset category data in the second image data;

[0023] Performing feature extraction on the second preset category data based on the feature extraction model to obtain a second target feature;

[0024] Obtain any first target feature in the target feature library, and calculate the feature similarity between the first target feature and the second target feature;

[0025] In response to the feature similarity exceeding a preset threshold, a second target feature is output; wherein the second target feature is an object feature.

[0026] Furthermore, the method for generating the heat map trajectory also includes:

[0027] Generate target detection boxes based on all object features;

[0028] The center coordinates of the target detection frame are used as the coordinates of the object to be detected in the second image data.

[0029] Furthermore, the steps of obtaining the heat map space of the teaching scene include:

[0030] segmenting the second image data into a plurality of functional areas and non-functional areas based on the functional area information;

[0031] Constructing a heat map space, and dividing the heat map space into a plurality of first areas and second areas according to the functional area information; wherein the number of the first areas is the same as the number of the functional areas, the number of the second areas is the same as the number of the non-functional areas, and the first areas and the second areas are arranged in the same manner as the functional areas and the non-functional areas;

[0032] A plurality of first regions are mapped to a plurality of functional regions using a plurality of homography matrices to obtain a mapping relationship between the heat map space and the second image data.

[0033] Furthermore, based on the mapping relationship and the coordinates, the step of generating a thermal map trajectory of the object to be measured includes:

[0034] Get the functional area corresponding to the coordinates;

[0035] Based on the mapping relationship, obtaining a homography matrix between the functional area and the corresponding first area;

[0036] Acquire target coordinates of the coordinates mapped to the first area based on the homography matrix;

[0037] Generate a thermal map trajectory of the object to be tested based on the target coordinates.

[0038] A second aspect of the present application provides a heat map trajectory generating device, the heat map trajectory generating device comprising:

[0039] A first camera, used for acquiring first image data in a teaching scene in real time;

[0040] The second camera is used to obtain second image data in the teaching scene in real time; wherein the first camera and the second camera are arranged at two ends of the teaching scene, and the first image data and the second image data partially overlap;

[0041] A processing module, configured to extract a first target feature from the first image data, so as to construct a target feature library based on the first target feature;

[0042] The processing module is also used to extract the second target feature in the second image data, and compare the target feature library with the second target feature to obtain an object feature whose feature similarity exceeds a preset threshold; wherein the object feature represents the coordinates of the object to be measured in the second image data;

[0043] The processing module is also used to obtain a thermal map space of the teaching scene, wherein the thermal map space has a mapping relationship with the second image data;

[0044] The processing module is also used to generate a thermal map trajectory of the object to be tested based on the mapping relationship and coordinates.

[0045] A third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the above-mentioned method for generating a thermal map trajectory.

[0046] A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for generating a thermal map trajectory as described above is implemented.

[0047] Different from the prior art, the present application collects first image data in real time through a first camera, and automatically identifies and constructs a target feature library based on the first image data, collects second image data in real time through a second camera, and automatically identifies the object to be measured by corresponding the second image data and the first image data, and generates a heat map trajectory corresponding to the motion trajectory of the object to be measured by customizing the construction of a heat map space, and based on the mapping relationship between the second image data and the heat map space, generates a heat map trajectory corresponding to the motion trajectory of the object to be measured, thereby achieving targeted determination of the teacher's patrol trajectory, and solving the problems of near-big and far-small objects and low correlation with real scenes caused by the perspective transformation of camera imaging.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1It is a flow chart of an embodiment of a method for generating a heat map trajectory of the present application;

[0051] Figure 2 yes Figure 1 Specific flow diagram of step S12;

[0052] Figure 3 yes Figure 2 Specific flow diagram of step S123;

[0053] Figure 4 yes Figure 1 Specific flow diagram of step S13;

[0054] Figure 5 yes Figure 1 Specific flow diagram of step S14;

[0055] Figure 6 yes Figure 1 Specific flow diagram of step S15;

[0056] Figure 7 It is a flow chart of another embodiment of the method for generating a thermal map trajectory of the present application;

[0057] Figure 8 is an operational schematic diagram of an embodiment of the present application for mapping the second image data with the heat map space;

[0058] Fig. 9 It is a schematic diagram of the framework of an embodiment of a device for generating a thermal map trajectory of the present application;

[0059] Fig.10 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0060] Fig.11 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solution of the present application, the heat map trajectory generation method, device, equipment and readable storage medium provided by the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It can be understood that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0062] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0063] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0064] Since the existing technologies use target detection, key point detection or video classification algorithms for behavior recognition, they all focus on the recognition of the behavior itself. Therefore, the present application provides a heat map trajectory generation method, which collects image data in real time and generates heat map trajectories based on the image data, so as to achieve a macro and intuitive analysis of the teacher's patrol trajectory in the classroom, and avoid the problem of near big and far small caused by perspective transformation in image space.

[0065] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of a method for generating a heat map trajectory of the present application. The execution subject of the method for generating a heat map trajectory of the present application may be a heat map trajectory generating device. In some possible implementations, the method for generating a heat map trajectory may also be implemented by a processor calling a computer-readable instruction stored in a memory.

[0066] Specifically, the execution body of the heat map trajectory generation method of the present application is a heat map trajectory generation device, please refer to Fig. 9 ,like Fig. 9As shown, the heat map trajectory generating device 60 includes a first camera 61, a second camera 62 and a processing module 63, wherein the first camera 61 and the second camera 62 are used to collect image data, and the processing module 63 is connected to the first camera 61 and the second camera 62 by electrical signals or wireless communication to obtain the collected image data, and generate a corresponding heat map trajectory based on the image data. Optionally, in other embodiments, the processing chip disposed in the first camera 61 and the second camera 62 can also perform image data processing and generate a heat map trajectory. For example, after the first camera 61 completes the processing of the collected image data, it sends it to the second camera 62, and the second camera 62 generates a corresponding heat map trajectory based on the image data sent by the first camera 61 and the image data collected by itself, and so on.

[0067] The specific application scenario of this application is a teaching scenario, which is usually a scenario of taking classes in a classroom, such as Fig. 9 As shown, the first camera 61 is set at the back of the classroom and shoots forward to collect image information of the blackboard and podium area; the second camera 62 is set at the front of the classroom and shoots backward to collect image information of the student desks and the corridor area.

[0068] Since teachers usually stand on the podium and teach students, the first camera 61 can clearly capture the teacher's teaching scene, and specifically obtain the teacher's front posture, including facial features, body features, and movement features, etc. The second camera 62 can capture the teacher's walking up and down the podium and walking in the aisle to interact with students.

[0069] Specifically, the method for generating a heat map trajectory according to an embodiment of the present disclosure may include the following steps:

[0070] Step S11: using a first camera to acquire first image data in a teaching scene in real time, and using a second camera to acquire second image data in a teaching scene in real time.

[0071] Among them, the first camera 61 and the second camera 62 of this embodiment are arranged at both ends of the teaching scene, specifically at the front and back ends of the classroom, for collecting image information of different areas in the classroom. Since the first camera 61 and the second camera 62 have a certain shooting range, and based on the size of the classroom, the first image data collected by the first camera 61 and the second image data collected by the second camera 62 may partially overlap. For example, the first camera 61 and the second camera 62 can both capture image data information in front of and behind the podium.

[0072] The first camera 61 and the second camera 62 of this embodiment are used to collect the first image data and the second image data in real time, and perform corresponding image processing on the collected first image data and the second image data in real time.

[0073] Step S12: extracting the first target feature in the first image data to construct a target feature library based on the first target feature.

[0074] Among them, in this embodiment, since the specific application scenario is a teaching scenario, the first target feature extracted in this embodiment is used to characterize the teacher's identity, that is, by extracting the first target feature in the first image data, and the target feature library constructed based on the first target feature is a teacher identity feature library, which is used to complete the registration of the teacher's identity feature online.

[0075] The heat map trajectory generation method of this embodiment is different from the conventional identity recognition method. It does not need to register the teacher's face and other identity information in advance. Instead, it uses the first camera 61 to shoot the teacher's teaching position in the classroom to collect the teacher's feature data, that is, the first target feature. The target feature library is constructed through the first target feature, which can automatically complete the registration of the teacher feature library.

[0076] Step S13: extracting the second target feature in the second image data, and comparing the target feature library with the second target feature to obtain the object feature whose feature similarity exceeds a preset threshold.

[0077] Among them, in this embodiment, since the specific application scenario is a teaching scenario, and there are teachers and students in the shooting area of ​​the second camera 62, it is necessary to perform feature comparison on the collected data to determine the object features representing the identity of the teacher in the second image data.

[0078] Specifically, this embodiment extracts the second target feature in the second image data, the second target feature includes some teacher features and some student features, compares the target feature library and the second target feature, that is, compares the first target feature and the second target feature representing the teacher's identity, and when it is determined that the feature similarity exceeds a preset threshold, it is determined that the current second target feature and the first target feature are very similar, and then it is determined that the current second target feature and the first target feature represent the same target object, that is, the teacher, and the current second target feature is the object feature. Furthermore, the coordinates of the object to be measured, that is, the teacher in the second image data, can be determined by the object feature.

[0079] This embodiment can automatically identify the location information of the teacher in the teaching scene by comparing the second target feature with the target feature library.

[0080] Step S14: Acquire a thermal map space of the teaching scene, wherein the thermal map space has a mapping relationship with the second image data.

[0081] Among them, this embodiment can customize the scene distribution of the second image data and construct a heat map space that has a mapping relationship with the second image data to determine the object to be tested, that is, the specific coordinates of the teacher in the second image data mapped to the heat map space.

[0082] This embodiment can customize the scene distribution of the second image data to achieve customized construction of the heat map space, so that the heat map space is more relevant to the actual application scenario, reflecting the real teacher's classroom patrol trajectory, and avoiding the problem of directly generating a heat map in the image space, which will retain the problem of near big and far small and low correlation with the real scene due to the perspective transformation of camera imaging.

[0083] Step S15: Generate a thermal map trajectory of the object to be measured based on the mapping relationship and coordinates.

[0084] Among them, this embodiment determines the specific coordinates of the object to be tested, that is, the teacher in the thermal map space, based on the coordinates of the object to be tested obtained in step S13, and based on the mapping relationship between the second image data obtained in step S14 and the thermal map space, and generates a thermal map trajectory of the object to be tested, that is, the trajectory of the teacher's classroom patrol, from multiple specific coordinates in the thermal map space. Optionally, the thermal map trajectory generated by this embodiment can be displayed in 2D or 3D.

[0085] In this embodiment, the first camera 61 collects the first image data in real time, and automatically identifies and constructs a target feature library based on the first image data. The second camera 62 collects the second image data in real time, and automatically identifies the object to be measured by corresponding to the second image data and the first image data. The heat map space is constructed by customization, and based on the mapping relationship between the second image data and the heat map space, a heat map trajectory corresponding to the motion trajectory of the object to be measured is generated, so as to achieve targeted determination of the teacher's patrol trajectory, and solve the problems of near-big and far-small objects and low correlation with the real scene caused by the perspective transformation of camera imaging.

[0086] Optionally, the present application also provides another embodiment for implementing step S12 to construct a target feature library, wherein the application scenario of the present application is a teaching scenario, and the target feature library can specifically be a teacher feature library. Figure 2 , Figure 2 yes Figure 1 The specific flow diagram of step S12 in FIG. Specifically, the following steps are included:

[0087] Step S121: classify the first image data based on the target detection model to obtain first preset category data in the first image data.

[0088] In this embodiment, the first image data is input into the target detection model, and the target detection model is used to perform data classification on the first image data to obtain first preset category data in the first image data.

[0089] Optionally, the target detection model of this embodiment can be trained using YOLOv7 (YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors) as the target detection algorithm, and three categories of "front human body", "side human body" and "back human body" are used as the required detection categories.

[0090] Because the images taken by the first camera 61 usually include a large number of students' backs, and the people facing the first camera 61 during class are mostly teachers, this embodiment can use "front human body" as the first preset category, and only extract the human body sub-image of the "front human body" category in the first image data, that is, the first preset category data, which can reduce the computational complexity of feature extraction.

[0091] Optionally, in other embodiments, target detection algorithms including YOLO, YOLOv2-YOLOv6, FasterRCNN, etc. may be used to train the target detection model.

[0092] Step S122: extracting features of the first preset category data based on the feature extraction model to obtain a first target feature.

[0093] In this embodiment, the first preset category data extracted in step S122 is input into the feature extraction model, and the feature extraction model is used to perform feature extraction on the first preset category data to obtain the first target feature.

[0094] Optionally, this embodiment uses the reid algorithm BOT (Bag of Tricks and A Strong Baseline for Deep Person Re-identification) as a human appearance feature extraction algorithm to train a feature extraction model, and then use the feature extraction model to extract features from the first preset category data. Specifically, each human sub-image of the "frontal human body" category is used as the input of the BOT algorithm, and the array output by BOT is the feature vector corresponding to the human sub-image. Usually, a feature vector is an array composed of 512 floating point numbers.

[0095] Optionally, in other embodiments, re-identification algorithms such as AGW (derived from the paper: Deep Learning for Person Re-identification: A Survey and Outlook) and MGN (derived from the paper: Learning Discriminative Features with Multiple Granularities for Person Re-Identification) may be used to train the feature extraction model.

[0096] Step S123: clustering the first target feature to obtain a target feature cluster.

[0097] Among them, in the teaching scenario, in the picture of the first camera 61, the person who usually appears in front of the podium and the blackboard and faces the first camera 61 is most of the time the teacher giving a class. Therefore, this embodiment directly clusters the first target feature obtained in step S122 to obtain a target feature cluster, and then identifies the identity of the teacher, wherein the target feature cluster corresponds to the teacher feature.

[0098] Optionally, the clustering algorithm of this embodiment is a feature clustering algorithm. Optionally, in other embodiments, a clustering algorithm such as a hierarchical clustering algorithm, a density clustering algorithm or a spectral clustering algorithm may be used to cluster the first target features acquired by the first camera 61 .

[0099] Step S124: constructing a target feature library based on the target feature cluster.

[0100] Among them, this embodiment builds a target feature library based on the target feature cluster output in step S123, and the target feature library is the teacher identity feature library.

[0101] Optionally, the present application also provides another embodiment for implementing step S123 to obtain the target feature cluster. Figure 3 , Figure 3 yes Figure 2 Specific flow chart of step S123 in FIG.

[0102] Specifically, the following steps are included:

[0103] Step S1231: clustering the first target feature based on the clustering model, and outputting at least two feature clusters.

[0104] The clustering model of this embodiment can be obtained by training a hierarchical clustering algorithm. The first target feature extracted in step S122 is input into the clustering model, and the first target feature is clustered using the clustering model to output at least two feature clusters.

[0105] Step S1232: obtaining the number of cluster features of at least two feature clusters, and sorting the at least two feature clusters based on the number of cluster features, so as to select the feature cluster with the largest number of cluster features as the target feature cluster.

[0106] Among them, different feature clusters have different numbers of cluster features. The more cluster features there are, the more capable it is to characterize the object to be tested. Therefore, this embodiment selects a cluster with the largest number of cluster features output by clustering as the application scenario, namely, the teacher feature cluster in the teaching scenario.

[0107] Optionally, the present application also provides another embodiment for implementing step S13 to obtain object features, wherein the application scenario of the present application is a teaching scenario, and the object features may specifically be the teacher photographed by the second camera 62. Figure 4 , Figure 4 yes Figure 1 The specific flow diagram of step S13 in FIG. Specifically, the following steps are included:

[0108] Step S131: classify the second image data based on the target detection model to obtain second preset category data in the second image data.

[0109] In this embodiment, the second image data is input into the target detection model, and the target detection model is used to perform data classification on the second image data to obtain second preset category data in the second image data.

[0110] Optionally, the target detection model of this embodiment can be trained using YOLOv7 (YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors) as the target detection algorithm, and two categories of "standing posture human body" and "other posture human body" are used as the required detection categories. In other words, the first camera 61 and the second camera 62 can use the same target detection model to perform image data classification processing on the collected image data.

[0111] Since the images captured by the second camera 62 usually capture teachers walking around the classroom, that is, teachers in a standing position, or students sitting still with their backs to the second camera 62, and this embodiment is used to construct the teacher's classroom patrol trajectory, it is necessary to determine the position of the teacher in the second camera 62. Therefore, this embodiment can use "standing posture human body" as the second preset category, and only extract the human body sub-image of the "standing posture human body" category in the second image data, that is, the second preset category data, which can reduce the calculation amount of feature extraction.

[0112] Specifically, this embodiment cuts out a detection frame with a detection category of "human body in standing posture" in the second image data to obtain a human body sub-image in standing posture, which is used as data for subsequent identification of whether the human body is a teacher.

[0113] Step S132: extracting features of the second preset category data based on the feature extraction model to obtain a second target feature.

[0114] In this embodiment, the second preset category data extracted in step S122 is input into the feature extraction model, and the feature extraction model is used to perform feature extraction on the second preset category data to obtain the second target feature.

[0115] Optionally, this embodiment uses the reid algorithm BOT (Bag of Tricks and A Strong Baseline for Deep Person Re-identification) as a human appearance feature extraction algorithm to train a feature extraction model, and then use the feature extraction model to extract features from the first preset category data. Specifically, each human sub-image of the "standing posture human body" category is used as the input of the BOT algorithm, and the array output by BOT is the feature vector corresponding to the human sub-image. Usually, a feature vector is an array composed of 512 floating point numbers.

[0116] Optionally, in other embodiments, re-identification algorithms such as AGW (derived from the paper: Deep Learning for Person Re-identification: A Survey and Outlook) and MGN (derived from the paper: Learning Discriminative Features with Multiple Granularities for Person Re-Identification) may be used to train the feature extraction model.

[0117] Step S133: Obtain any first target feature in the target feature library, and calculate the feature similarity between the first target feature and the second target feature.

[0118] Among them, this embodiment randomly extracts a feature vector from the target feature library constructed in step S12, that is, from the teacher identity feature library, that is, extracts a first target feature, and compares it with the second target feature extracted in step S132 to solve the feature similarity score.

[0119] Specifically, the feature similarity score of this embodiment can be expressed as follows:

[0120]

[0121] In the above formula, A represents the first target feature randomly extracted, B represents the second target feature extracted, and score is the similarity score, whose value range is [0, 1]. The larger the value, the higher the similarity between the two features.

[0122] Step S134: In response to the feature similarity exceeding a preset threshold, outputting a second target feature.

[0123] In this embodiment, if the similarity score calculated by step S133 exceeds the preset threshold, the second target feature is considered to be very similar to the first target feature, that is, the human frame and human sub-image corresponding to the human feature are considered to be teachers, otherwise they are considered to be students, and the second target feature is an object feature. Optionally, in this embodiment, the preset threshold can be set to 0.7.

[0124] In this embodiment, all the second target features extracted in step S132 are compared with the standard feature library, so that the teacher identity recognition of all the second target features can be completed.

[0125] Optionally, the present application also provides another embodiment for implementing step S15 to achieve the construction and mapping of the heat map. Figure 5 , Figure 5 yes Figure 1 The specific flow diagram of step S14 in FIG. Specifically, the following steps are included:

[0126] Step S141: Segment the second image data into a plurality of functional areas and non-functional areas based on the functional area information.

[0127] Among them, since in the application scenario, that is, the teaching scenario, the teacher's classroom patrol trajectory refers to the teacher's walking trajectory in the aisle area of ​​the classroom, it is first necessary to divide the image space into different functional areas, that is, into aisle areas and non-aisle areas (student seat areas and wall areas). Specifically, different functional areas have corresponding functional area information. This embodiment divides the second image data into several functional areas and non-functional areas based on the functional area information, then the functional area is the aisle area, and the non-functional area is the non-aisle area.

[0128] Optionally, the division method in this embodiment can be a manual division method or a division method based on image semantic segmentation.

[0129] Further reading Figure 8 , Figure 8 FIG. 1 is an operation diagram of an embodiment of the present application for mapping the second image data to the thermal map space. Figure 8 As shown, the second image data of this embodiment can be divided into Aisle 1, Aisle 2, Aisle 3 and Aisle 4 in the functional area, and the walls, student seating area and podium in the non-functional area.

[0130] Step S142: construct a heat map space, and divide the heat map space into a plurality of first areas and second areas according to the functional area information.

[0131] Among them, this embodiment can construct a heat map space with the same custom blocks as step S141. Specifically, this embodiment divides the heat map space into a plurality of first areas and second areas according to the functional area information, wherein the number of first areas and functional areas is the same, the number of second areas and non-functional areas is the same, and the first areas and second areas are arranged in the same manner as the functional areas and non-functional areas.

[0132] like Figure 8 As shown, the heat map space of this embodiment can be divided into aisle 1, aisle 2, aisle 3 and aisle 4 in the first area, and a student seat area in the non-functional area. Wherein aisle 1, aisle 2, aisle 3 and aisle 4 in the second image data correspond one-to-one to aisle 1, aisle 2, aisle 3 and aisle 4 in the heat map space, respectively.

[0133] Step S143: Mapping the first regions to the functional regions using the homography matrices to obtain a mapping relationship between the heat map space and the second image data.

[0134] Among them, this embodiment constructs a mapping relationship from the second image data to the thermal map space, that is, constructs the following Figure 8 The mapping relationship between the four one-to-one corresponding aisle areas shown. Specifically, this embodiment abstracts the mapping relationship between the two spatial coordinates as a homography transformation from one 2D space to another 2D space, that is, this transformation is achieved through a homography matrix. Thus, four homography matrices H1, H2, H3 and H4 are constructed to represent the transformation matrices of aisle 1, aisle 2, aisle 3 and aisle 4 from the second image data to the thermal map space, respectively.

[0135] Optionally, the present application also provides another embodiment for determining the coordinates of the target object, so as to map the coordinates to the heat map space and generate a corresponding heat map trajectory. Figure 7 , Figure 7 This is a flow chart of another embodiment of the method for generating a heat map trajectory of the present application. Specifically, it includes the following steps:

[0136] Step S16: Generate a target detection box based on all object features.

[0137] In this embodiment, a target detection frame is generated based on the object features obtained in step S134. The target detection frame is used to characterize the position of the object to be detected, that is, the position of the teacher in the second image data. With the continuous data acquisition of the second camera 62 and the identity recognition of the teacher, the complete position trajectory of the teacher in the second image data, that is, the teacher's classroom patrol trajectory, is obtained.

[0138] Step S17: taking the center coordinates of the target detection frame as the coordinates of the object to be detected in the second image data.

[0139] Among them, since the target detection frame has a certain size, this embodiment further uses the center coordinates of the target detection frame as the coordinates of the object to be measured in the second image data, which facilitates the positioning of the object to be measured, that is, the teacher.

[0140] Optionally, the present application also provides another embodiment for implementing step S14 to generate a heat map trajectory based on the acquired coordinates of the object to be measured, wherein the application scenario of the present application is a teaching scenario, and the heat map trajectory can specifically be a teacher's classroom patrol trajectory. Figure 6 , Figure 6 yes Figure 1 The specific flow diagram of step S15 in FIG. Specifically, the following steps are included:

[0141] Step S151: Obtain the functional area corresponding to the coordinates.

[0142] Among them, Figure 8 As shown, in this embodiment, the coordinates of the object to be tested, that is, the teacher, in the second image data are obtained through step S13, and the coordinates of the object to be tested, that is, the teacher, can be known. Figure 8 The functional area in the second image data shown indicates the aisle in which the object is located.

[0143] Step S152: based on the mapping relationship, obtaining a homography matrix between the functional area and the corresponding first area.

[0144] Among them, this embodiment can determine the first area in the heat map space corresponding to the functional area determined in step S151 based on the mapping relationship between the heat map space determined in step S14 and the second image data. Specifically, by determining the functional area of ​​the second image data, for example, determining it as aisle 1, its corresponding homography matrix can be determined to be H1 and its corresponding first area can be determined to be aisle 1.

[0145] Step S153: Obtain target coordinates of the first region mapped with coordinates based on the homography matrix.

[0146] In this embodiment, based on the homography matrix H1 determined in step S152, the coordinates of the object to be tested in the second image data are mapped to the target coordinates in the heat map space. Specifically, in this embodiment, the coordinates in the heat map space can be obtained by left-multiplying the coordinates in the second image data by the corresponding homography matrix.

[0147] Step S154: Generate a thermal map trajectory of the object to be measured based on the target coordinates.

[0148] By analogy, after the mapping of the coordinates of all objects to be measured determined based on the second image data is completed, the trajectory of the object to be measured in the thermal map space can be obtained, that is, the classroom patrol thermal map trajectory can be determined.

[0149] This application also provides a device for generating a thermal map trajectory. Fig. 9 , Fig. 9 Schematic diagram of a heat map trajectory generating device according to an embodiment of the present invention. The heat map trajectory generating device 60 according to the present embodiment includes a first camera 61 , a second camera 62 and a processing module 63 .

[0150] Specifically, the first camera 61 is used to obtain the first image data in the teaching scene in real time, that is, it is set at the back end of the classroom to capture the image of the front end of the classroom; the second camera 62 is used to obtain the second image data in the teaching scene in real time, that is, it is set at the front end of the classroom to capture the image of the back end of the classroom. Due to the setting positions and shooting angles of the first camera 61 and the second camera 62, the first image data collected by the first camera 61 and the second image data collected by the second camera 62 partially overlap.

[0151] The processing module 63 is respectively connected to the first camera 61 and the second camera 62 by signals, which can be specifically an electrical signal connection or a wireless communication connection. The processing module 63 is used to extract the first target feature in the first image data to build a target feature library based on the first target feature; the processing module 63 is also used to extract the second target feature in the second image data, compare the target feature library and the second target feature, so as to obtain the object feature whose feature similarity exceeds a preset threshold; wherein the object feature represents the coordinates of the object to be measured in the second image data; the processing module 63 is also used to obtain the thermal map space of the teaching scene, wherein the thermal map space has a mapping relationship with the second image data; the processing module 63 is also used to generate a thermal map trajectory of the object to be measured based on the mapping relationship and the coordinates.

[0152] The thermal map trajectory generating device 60 of the present application can collect the first image data in real time through the first camera 61, and automatically identify and construct a target feature library based on the first image data, collect the second image data in real time through the second camera 62, and the processing module 63 can automatically identify the object to be measured by corresponding to the second image data and the first image data, and generate a thermal map trajectory corresponding to the motion trajectory of the object to be measured by customizing the thermal map space, and based on the mapping relationship between the second image data and the thermal map space, it can achieve targeted determination of the teacher's patrol trajectory, and solve the problems of near big and far small and low correlation with the real scene caused by the perspective transformation of camera imaging.

[0153] This application also provides an electronic device, see Fig.10 , Fig.10 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, and the processor 72 is used to execute program instructions stored in the memory 71 to implement the steps in any of the above-mentioned thermal map trajectory generation method embodiments. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 70 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0154] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above-mentioned thermal map trajectory generation method embodiments. The processor 72 can also be referred to as a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip having signal processing capabilities. The processor 72 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 72 may be implemented by an integrated circuit chip.

[0155] This application also provides a computer-readable storage medium, see Fig.11 , Fig.11 The computer readable storage medium 80 stores a computer program 81 that can be executed by a processor, and the computer program 81 is used to implement the steps in any of the above-mentioned thermal map trajectory generation method embodiments.

[0156] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0157] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0161] The above are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a heat map trajectory, characterized in that: include: Using a first camera to acquire first image data in a teaching scene in real time, and using a second camera to acquire second image data in the teaching scene in real time; The first camera and the second camera are arranged at two ends of the teaching scene, and the first image data and the second image data partially overlap; Extracting a first target feature from the first image data to construct a target feature library based on the first target feature; Extracting a second target feature from the second image data, and comparing the target feature library with the second target feature to obtain an object feature whose feature similarity exceeds a preset threshold; wherein the object feature represents the coordinates of the object to be measured in the second image data; Acquire a thermal map space of the teaching scene, wherein the thermal map space has a mapping relationship with the second image data; Based on the mapping relationship and the coordinates, a thermal map trajectory of the object to be measured is generated.

2. The method for generating a thermal map trajectory according to claim 1, characterized in that: The step of extracting the first target feature from the first image data to construct a target feature library based on the first target feature includes: Performing data classification on the first image data based on the target detection model to obtain first preset category data in the first image data; Performing feature extraction on the first preset category data based on a feature extraction model to obtain the first target feature; Clustering the first target features to obtain target feature clusters; The target feature library is constructed based on the target feature cluster.

3. The method for generating a thermal map trajectory according to claim 2, characterized in that: The step of clustering the first target feature to obtain a target feature cluster includes: Clustering the first target feature based on a clustering model, and outputting at least two feature clusters; The number of cluster features of the at least two feature clusters is obtained, and the at least two feature clusters are sorted based on the number of cluster features, so as to use the feature cluster with the largest number of cluster features as the target feature cluster.

4. The method for generating a thermal map trajectory according to claim 1, characterized in that: The step of extracting the second target feature from the second image data and comparing the target feature library with the second target feature to obtain an object feature whose feature similarity exceeds a preset threshold comprises: Performing data classification on the second image data based on the target detection model to obtain second preset category data in the second image data; Performing feature extraction on the second preset category data based on a feature extraction model to obtain the second target feature; Obtain any first target feature in the target feature library, and calculate the feature similarity between the first target feature and the second target feature; In response to the feature similarity exceeding the preset threshold, the second target feature is output; wherein the second target feature is the object feature.

5. The method for generating a thermal map trajectory according to claim 4, characterized in that: The method for generating the heat map trajectory also includes: Generate a target detection frame based on all the object features; The center coordinates of the target detection frame are used as the coordinates of the object to be detected in the second image data.

6. The method for generating a thermal map trajectory according to claim 5, characterized in that: The step of obtaining the thermal map space of the teaching scene includes: dividing the second image data into a plurality of functional areas and non-functional areas based on the functional area information; Constructing a heat map space, and dividing the heat map space into a plurality of first areas and second areas according to the functional area information; wherein the number of the first areas is the same as that of the functional areas, the number of the second areas is the same as that of the non-functional areas, and the first areas and the second areas are arranged in the same manner as the functional areas and the non-functional areas; A plurality of the first regions are mapped to a plurality of the functional regions using a plurality of homography matrices to obtain a mapping relationship between the thermal map space and the second image data.

7. The method for generating a thermal map trajectory according to claim 6, characterized in that: The step of generating a thermal map trajectory of the object to be measured based on the mapping relationship and the coordinates includes: Obtaining the functional area corresponding to the coordinates; Based on the mapping relationship, obtaining a homography matrix between the functional area and the corresponding first area; Acquire target coordinates of the first region mapped with the coordinates based on the homography matrix; A thermal map trajectory of the object to be measured is generated based on the target coordinates.

8. A device for generating a heat map trajectory, characterized in that: include: A first camera, used for acquiring first image data in a teaching scene in real time; A second camera is used to obtain second image data in the teaching scene in real time; The first camera and the second camera are arranged at two ends of the teaching scene, and the first image data and the second image data partially overlap; a processing module, configured to extract a first target feature from the first image data, so as to construct a target feature library based on the first target feature; The processing module is further used to extract a second target feature in the second image data, and compare the target feature library with the second target feature to obtain an object feature whose feature similarity exceeds a preset threshold; wherein the object feature represents the coordinates of the object to be measured in the second image data; The processing module is also used to obtain a thermal map space of the teaching scene, wherein the thermal map space has a mapping relationship with the second image data; The processing module is further used to generate a thermal map trajectory of the object to be measured based on the mapping relationship and the coordinates.

9. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the thermal map trajectory generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a thermal map trajectory according to any one of claims 1 to 7 can be implemented.

Images