Behavior recognition method, terminal equipment and computer readable storage medium

Abstract

The invention discloses a behavior recognition method, terminal equipment and a computer readable storage medium, and the method comprises the steps: obtaining a thermodynamic diagram space and image data of a to-be-recognized scene, the image data comprises a plurality of to-be-recognized regions, the thermodynamic diagram space comprises a plurality of grid regions, and the to-be-recognized regions and the grid regions have a mapping relation; and based on the mapping relationship between the to-be-identified area and the grid area, projecting the behavior identification result of the target image frame to the thermodynamic diagram space at the corresponding position, so as to represent the behavior frequency of the to-be-identified scene through the thermodynamic value of the thermodynamic diagram space, thereby facilitating the user to analyze the behaviors of all target objects in the to-be-identified scene, and improving the user experience. And the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a behavior recognition method, a terminal device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of computer technology, behavior recognition methods have broad application prospects. For example, they can be applied to intelligent video surveillance, video summarization, intelligent interface, human-computer interaction, sports video analysis, video retrieval, smart education and other scenarios.

[0003] Existing behavior recognition methods usually use related algorithms such as target detection, key point detection or video classification for behavior recognition, and usually only recognize the behavior of the target object in a single image. When the scene to be recognized includes multiple target objects, the existing behavior recognition methods cannot analyze the behavior of all target objects in the scene to be recognized, resulting in users being unable to intuitively analyze the behavior frequency of all target objects in the scene to be recognized from the behavior recognition results, resulting in a poor user experience. Summary of the invention

[0004] To solve the above technical problems, the present application provides a behavior recognition method, a terminal device and a computer-readable storage medium.

[0005] To solve the above problems, the present application provides a first technical solution: a behavior recognition method is provided, comprising: obtaining a heat map space and image data of a scene to be recognized, the image data including a number of areas to be recognized, the heat map space including a number of grid areas, and the areas to be recognized and the grid areas having a mapping relationship; in response to receiving a behavior recognition instruction, performing image acquisition on the scene to be recognized at a preset sampling frequency to obtain a number of target image frames of the scene to be recognized; performing behavior detection on the number of target image frames to obtain behavior recognition results corresponding to the target image frames; based on the mapping relationship between the area to be recognized and the grid area, projecting the behavior recognition results of the number of target image frames onto the heat map space of the corresponding position, so as to characterize the behavior frequency of the scene to be recognized by the thermal value of the heat map space.

[0006] Optionally, the behavior recognition result includes first coordinate data and behavior category data of the first target object; based on the mapping relationship between the area to be identified and the grid area, the behavior recognition results of several target image frames are projected onto the thermal map space of the corresponding position, so as to characterize the behavior frequency of the scene to be identified by the thermal value of the thermal map space, including: obtaining the second coordinate data of several areas to be identified, calculating the distance between the first coordinate data and several second coordinate data, so as to define the grid area with the smallest distance as the area to be identified where the first target object of the target image frame is located; based on the mapping relationship between the area to be identified and the grid area, obtaining the grid area corresponding to the first target object; obtaining the thermal value of the grid area corresponding to the target image frame, and updating the thermal value based on the behavior category data.

[0007] Optionally, the thermal value of the grid area corresponding to the target image frame is obtained, and the thermal value is updated based on the behavior category data, including: obtaining a first thermal value of the grid area corresponding to the first target object in the target image frame; in response to the behavior category data of the first target object including a preset behavior category, updating the thermal value of the grid area to a second thermal value, the second thermal value being the sum of the first thermal value and a first preset value.

[0008] Optionally, after obtaining the thermal value of the grid area corresponding to the target image frame and updating the thermal value based on the behavior category data, the behavior recognition method also includes: in response to receiving an end instruction, outputting the updated thermal map space of all target image frames to a preset position; or, in response to receiving an end instruction, visually rendering the updated thermal map space of all target image frames to output the rendered thermal map space to a preset position.

[0009] Optionally, a thermal map space and image data of a scene to be identified are obtained, the image data includes several areas to be identified, the thermal map space includes several grid areas, and the areas to be identified and the grid areas have a mapping relationship, including: constructing a thermal map space, and dividing the thermal map space into several grid areas according to position distribution information; obtaining image data of the scene to be identified, and performing target detection on the image data to obtain several areas to be identified of the scene to be identified and second coordinate data corresponding to the areas to be identified; sorting the several second coordinate data to bind the areas to be identified and the grid areas according to the sorting result of the second coordinate data to obtain a mapping relationship.

[0010] Optionally, the second coordinate data includes horizontal axis coordinate data and vertical axis coordinate data; sorting a number of second coordinate data to bind the area to be identified with the grid area according to the sorting result of the coordinate data to obtain a mapping relationship, including: sorting the number of second coordinate data once in ascending order of the vertical axis coordinate data to obtain multiple rows of areas to be identified according to the first sorting result; sorting the second coordinate data of each row of areas to be identified twice in ascending order of the horizontal axis coordinate data; binding the area to be identified with the grid area based on the result of the second sorting to obtain a mapping relationship.

[0011] Optionally, before the step of acquiring image data of the scene to be identified and performing target detection on the image data to obtain a plurality of areas to be identified of the scene to be identified and second coordinate data corresponding to the areas to be identified, the behavior recognition method further includes: performing image acquisition on the scene to be identified by an image acquisition device to obtain image data of the scene to be identified; or, acquiring historical transmission data of the image acquisition device, and acquiring image data of the scene to be identified from the historical transmission data.

[0012] Optionally, the area to be identified is a detection area of ​​a second target object, and the position distribution information includes distribution information of the second target object in the scene to be identified; image data of the scene to be identified is obtained, and target detection is performed on the image data to obtain several areas to be identified of the scene to be identified and second coordinate data corresponding to the areas to be identified, including: performing target detection on the image data to obtain the detection area of ​​the second target object and the second coordinate data of the center point of the detection area.

[0013] To solve the above problems, the present application provides a second technical solution: a terminal device is provided, including a processor and a memory connected to the processor, wherein program data is stored in the memory, and the processor calls the program data stored in the memory to execute the above behavior recognition method.

[0014] To solve the above problem, the present application provides a third technical solution: providing a computer-readable storage medium storing program instructions, wherein the program instructions are executed to implement the behavior recognition method as described above.

[0015] The present application provides a behavior recognition method, a terminal device, and a computer-readable storage medium. The behavior recognition method includes obtaining a heat map space and image data of a scene to be recognized, the image data includes several areas to be recognized, the heat map space includes several grid areas, and the areas to be recognized have a mapping relationship with the grid areas; in response to a behavior recognition instruction, image acquisition is performed on the scene to be recognized at a preset sampling frequency to obtain several target image frames of the scene to be recognized; behavior detection is performed on several target image frames to obtain behavior recognition results corresponding to the target image frames; based on the mapping relationship between the area to be recognized and the grid area, the behavior recognition results of several target image frames are projected onto the heat map space of the corresponding position, so as to characterize the behavior frequency of the scene to be recognized by the thermal value of the heat map space. The behavior recognition method of the present application projects the behavior recognition results of the target image frame onto the heat map space of the corresponding position, so as to characterize the behavior frequency of the scene to be recognized by the thermal value of the heat map space, so as to facilitate users to analyze the behavior of all target objects in the scene to be recognized and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a flowchart of the first embodiment of the behavior recognition method provided by the present application;

[0018] Figure 2 is a flow chart of a second embodiment of the behavior recognition method provided by the present application;

[0019] Figure 3 is a flowchart of a third embodiment of the behavior recognition method provided by the present application;

[0020] Figure 4 is a flowchart of a fourth embodiment of the behavior recognition method provided by the present application;

[0021] Figure 5 It is an operation diagram of an embodiment of the present application for binding the area to be identified with the grid area;

[0022] Figure 6 It is a structural schematic diagram of an embodiment of a terminal device provided by the present application;

[0023] Figure 7 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0026] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] In classroom teaching scenarios, how to more efficiently identify and analyze students' classroom behaviors has become a research hotspot in smart education. Specifically, a large number of interactive behaviors usually occur during classroom teaching, such as students raising their hands and standing up to participate in teaching interactions; students' interactive behaviors are an important factor in evaluating teachers' teaching quality. In existing behavior recognition methods, computer vision technology is usually used to analyze students' behaviors in images, such as raising their hands, standing up, and paying attention in class.

[0028] However, existing behavior recognition methods usually focus on identifying students' behaviors themselves. During a teaching period (such as a class), the location and frequency of students' behaviors may change. Existing behavior recognition methods cannot perform a macro analysis of all students' behaviors in the class, resulting in users being unable to intuitively understand the class participation of all students in the class from the behavior recognition results, and the user experience is poor.

[0029] In view of this, the embodiment of the present application first proposes a behavior recognition method, which can be applied to classroom teaching scenarios. For example, the behavior recognition method can be used to identify student behavior and / or teacher behavior, and by projecting the behavior recognition results into the corresponding heat map space, the user can intuitively know the behavior frequency of all target objects from the heat map space, so that the user can understand the interactive participation of students or teachers from the heat map space.

[0030] See also Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the behavior recognition method provided by this application. Figure 1 As shown, the behavior recognition method of this embodiment includes:

[0031] Step S11: Acquire the thermal map space and image data of the scene to be identified, the image data includes a number of areas to be identified, the thermal map space includes a number of grid areas, and there is a mapping relationship between the areas to be identified and the grid areas.

[0032] Specifically, the scene to be identified has a corresponding thermal map space and image data, the image data is an image acquired when an image acquisition device is used to acquire an image of the scene to be identified, and the image data can be used to reflect the distribution information of the target object in the scene to be identified. The distribution conditions of several grid areas in the thermal map space correspond to the areas to be identified in the image data, and the distribution positions of several grid areas in the thermal map space correspond to the distribution information of the target object in the scene to be identified.

[0033] Among them, the scene to be identified is the current scene of the object to be identified by the behavior recognition method. For example, when the behavior recognition method is applied to smart education, the behavior recognition method is used to identify and analyze the classroom behavior of students or teachers. Exemplarily, the scene to be identified is a classroom, and the target objects include students; because it is necessary to manage the position of students and their behavior when analyzing the classroom behavior of students, at this time, students are the first target objects, and student seats are the second target objects. The distribution position of the area to be identified and the grid area is related to the distribution information of the second target object in the scene to be identified; for example, there is a 4*4 seat distribution in the classroom. At this time, the heat map space can include a 4*4 grid area, and the image data can include a 4*4 area to be identified. Each grid area and the area to be identified are used to correspond to the student seats in the real scene.

[0034] Step S12: In response to receiving the behavior recognition instruction, image acquisition is performed on the scene to be recognized according to a preset sampling frequency to obtain a number of target image frames of the scene to be recognized.

[0035] In response to the behavior recognition instruction, the scene to be recognized is imaged at a preset sampling frequency to obtain a number of target image frames of the scene to be recognized, and the target image frames are image frames acquired sequentially in a time series. The behavior recognition instruction can be a start instruction performed by a user through a click operation or a touch operation on a display interface, or can be an instruction started when the terminal device recognizes a preset change in the scene to be recognized. For example, when the terminal device recognizes that a teacher appears in the classroom, it determines that classroom teaching has started and generates a behavior recognition instruction. No specific limitation is made here.

[0036] In an optional implementation, in response to a behavior recognition instruction, the recording function of the image acquisition device is started to record the scene to be recognized and obtain a video stream of a preset time period, and obtain a number of target image frames from the video stream at a preset sampling frequency for subsequent behavior recognition; for example, the video frame rate when recording the scene to be recognized is 25 frames / second, and the target image frames are extracted from the video stream at a sampling frequency of 5 frames / second for behavior recognition to obtain a number of target image frames. It can be understood that the number of target image frames in this embodiment is at least one image frame, and each time a target image frame is obtained at a preset sampling frequency, steps S13-S14 are executed until all target image frames complete behavior recognition.

[0037] Step S13: performing behavior detection on the target image frame to obtain a behavior recognition result corresponding to the target image frame.

[0038] After acquiring the target image frame, computer vision technology is used to perform behavior detection on the target image frame. For example, a target detection model is used to perform behavior detection on the target image frame. The target detection model is trained with training data consisting of preset scenes to be identified and target objects to obtain a behavior recognition result corresponding to the target image frame. In an optional embodiment, the behavior recognition result is the prediction result of the target detection model on the behavior category of all first target objects in the target object frame. For example, in a classroom teaching scenario, the target image frame can be an image frame obtained by taking a photo or recording a video in the classroom using an image acquisition device set at a designated location in the classroom. The behavior recognition result of the target image frame is the prediction result obtained by the target detection model for identifying the behavior of all students in the image frame.

[0039] Step S14: Based on the mapping relationship between the area to be identified and the grid area, the behavior recognition results of several target image frames are projected onto the heat map space of the corresponding positions, so as to characterize the behavior frequency of the scene to be identified through the heat value of the heat map space.

[0040] After obtaining the behavior recognition result of the target object frame, the position relationship of the first target object in the scene to be recognized in the behavior recognition result is further obtained, and based on the mapping relationship between the area to be recognized and the grid area, the behavior recognition result is projected onto the heat map space of the corresponding position. Since the area to be recognized and the grid area have a mapping relationship in coordinate position, when the behavior recognition result of the first target object is projected onto the grid area of ​​the corresponding heat map space, the thermal value of the heat map space is adjusted according to the behavior recognition result, so that the user can characterize the behavior frequency of the first target object through the thermal value of the heat map space. For example, in a classroom education scenario, the frequency of interactive behaviors of students in various positions in the classroom can be characterized by the thermal value of the heat map space, so that users can directly understand the distribution of students' classroom participation between different positions through the heat map space, thereby helping to improve teaching quality and promote educational equity.

[0041] In an embodiment of the present application, the behavior recognition method projects the behavior recognition results of the target image frame onto the heat map space of the corresponding position, and uses the thermal value of the heat map space to characterize the behavior frequency of the scene to be identified, so as to facilitate the user to analyze the behavior of all target objects in the scene to be identified and improve the user experience.

[0042] In one embodiment, see Figure 2 , Figure 2 2 is a flow chart of the second embodiment of the behavior recognition method provided by the present application. Figure 2 As shown, the behavior recognition result includes the first coordinate data and behavior category data of the first target object; step S14 includes the following steps:

[0043] Step S21: acquiring second coordinate data of a plurality of to-be-recognized areas, calculating the distances between the first coordinate data and a plurality of second coordinate data, and defining a grid area with the smallest distance as the to-be-recognized area where the first target object of the target image frame is located.

[0044] Among them, in step S13, when the behavior detection is performed on the target image frame, the behavior detection is specifically performed on at least one first target object of the target image frame to output the first coordinate data and behavior category data of all the first target objects in the target image frame, and the first coordinate data is used to indicate the position of the first target object in the target image frame. Furthermore, the target image frame of this embodiment and the image data in step S11 are image data acquired when the same image acquisition device performs image acquisition on the scene to be identified under the same acquisition perspective, that is, the coordinate axis where the first coordinate data of the target image frame and the second coordinate data of the area to be identified in the image data are located is the same coordinate axis, and the distance between the first coordinate data and a plurality of second coordinate data can be calculated to determine in which area to be identified the first target object is located.

[0045] Specifically, the distance between the first coordinate data and the plurality of second coordinate data is calculated, and the grid area with the smallest distance is selected from the plurality of distance data, and the grid area is defined as the area to be identified where the first target object is located in the target image frame. Since the target image frame may include multiple first target objects, the area to be identified where all the first target objects are located in the target image frame can be obtained by calculating the distance between the first coordinate data and the plurality of second coordinate data. In an optional embodiment, the distance between the first coordinate data and the plurality of second coordinate data may be, but is not limited to, the Euclidean distance. The behavior category data in the behavior recognition result is used to indicate the behavior of the first target object in the target image frame. Depending on the application scenario and the target object, the behavior category may include multiple types, which are not specifically limited here; for example, in a classroom teaching scenario, when the first target object is a student, the behavior category data may be used to indicate whether the student is in the state of raising his hand, standing up, sitting down, sleeping, etc. in the target image frame.

[0046] Step S22: based on the mapping relationship between the area to be identified and the grid area, obtaining the grid area corresponding to the first target object.

[0047] After the to-be-recognized area where the first target object is located in the target image frame is acquired, the grid area where the first target object is located is further acquired based on the mapping relationship between the to-be-recognized area and the grid area.

[0048] Step S23: Obtain the thermal value of the grid area corresponding to the target image frame, and update the thermal value based on the behavior category data.

[0049] The thermal value of the grid area corresponding to the target image frame is obtained, and the thermal value is updated based on the behavior category data. Specifically, the specific updating method may include: when the behavior category data includes a preset valid interactive behavior, increasing the thermal value of the grid area; when the behavior category data does not include a valid interactive behavior, keeping the thermal value of the corresponding grid area unchanged.

[0050] In an embodiment of the present application, the behavior recognition method calculates the distance between the first coordinate data and a number of second coordinate data to define the grid area with the smallest distance as the area to be identified where the first target object of the target image frame is located, and based on the mapping relationship between the area to be identified and the grid area, obtains the grid area corresponding to the first target object, and then updates the thermal value of the grid area based on the behavior category data, so that the thermal value of each grid area in the thermal map space can represent the behavior frequency of the first target object, which is convenient for users to analyze the behaviors of all first target objects in the scene to be identified and improves the user experience.

[0051] Optionally, step S23 includes: obtaining a first thermal value of a grid area corresponding to a first target object in a target image frame; in response to the behavior category data of the first target object including a preset behavior category, updating the thermal value of the grid area to a second thermal value, the second thermal value being the sum of the first thermal value and a first preset value.

[0052] Specifically, after obtaining the grid area corresponding to the first target object in the target image frame, the first thermal value of the grid area corresponding to the first target object in the target image frame is further obtained, and it is determined whether the behavior category data of the first target object includes a preset behavior category. The preset behavior category is the effective behavior that the user needs to analyze for the scene to be identified. For example, in a classroom teaching scene, the effective behavior of students (ie, the first target object) includes but is not limited to two behavior categories: students standing up and students raising their hands.

[0053] When the behavior category data of the first target object includes a preset behavior category, the thermal value of the grid area is updated to a second thermal value, or the first thermal value of the grid area is modified to a second thermal value, the second thermal value being the sum of the first thermal value and the first preset value. In an optional implementation, the plurality of target image frames acquired in step S12 have a sequential order in the sampling time sequence, and the numerical value of the first thermal value is related to the behavior recognition result of the image frame before the sampling time of the target image frame.

[0054] Exemplarily, the several target image frames acquired in step S12 may include a first image frame and a second image frame obtained by sequentially sampling along a time series. The behavior recognition method of this embodiment is used to obtain a first thermal value of a grid area corresponding to a first target object in the second image frame. At this time, the numerical value of the first thermal value is related to the behavior recognition result of the first image frame; for example, there is a first target object in the same grid area of ​​the first image frame and the second image frame, and when the behavior category data of the first target object in the first image frame does not include a preset behavior category, the numerical value of the first thermal value may be 0; when the behavior category data of the first target object in the first image frame includes a preset behavior category, the numerical value of the first thermal value may be 1. At this time, the second thermal value is the sum of the first thermal value and the first preset numerical value, and the first preset numerical value may be 1 or any set numerical value. Furthermore, in a classroom teaching scenario, if the entire behavior recognition lasts for a period of time (for example, within a course teaching time), the thermal value of each grid area in the heat map space can be used to reflect the frequency of students (first target objects) participating in teaching interactive behaviors in the grid area. For example, within a course teaching time, if the position of a grid area recognizes 5 student hand-raising behaviors and 2 student standing behaviors, the thermal value of the grid area can be 7.

[0055] In this embodiment, the behavior recognition method obtains the first thermal value of the grid area corresponding to the first target object in the target image frame, and in response to the behavior category data of the first target object including a preset behavior category, updates the thermal value of the grid area to a second thermal value, so that when the first target object in the same grid area has a preset behavior category, the thermal value of the thermal map space changes accordingly, and thus the thermal value of each grid area in the thermal map space can characterize the behavior frequency of the first target object, thereby facilitating the user to analyze the behavior of all first target objects in the scene to be identified and improving the user experience.

[0056] Furthermore, after the behavior category data of all the first target objects in each target image frame are projected into the heat map space, the behavior recognition of the target image frame is completed, and the terminal device continues to obtain the target image frame of the next sampling time series to continue analyzing the target image frame, so as to perform behavior recognition on the first target object of the real-time recognition scene within a preset time period, and enable the heat map space to characterize the changes in the behavior frequency of the first target object within a time period, thereby further improving the user experience.

[0057] Optionally, after step S23, the behavior recognition method further includes: in response to receiving an end instruction, outputting the updated heat map space of all target image frames to a preset position; or, in response to receiving an end instruction, visually rendering the updated heat map space of all target image frames to output the rendered heat map space to a preset position.

[0058] Specifically, after the behavior recognition is completed, the heat map space after the behavior recognition results of all target image frames are updated is output to the preset position. Since the behavior recognition lasts for a preset time from the start to the end, the output heat map space can be used to represent the behavior frequency of all first target objects in each grid area within the preset time period, which is convenient for users to analyze the behavior of all target objects in the scene to be identified and improve the user experience.

[0059] Among them, the end instruction is used to instruct the terminal device to stop the behavior recognition and analysis of the scene to be recognized. The generation method of the end instruction can correspond to the generation method of the behavior recognition instruction, which will not be repeated here.

[0060] In one embodiment, see Figure 3 , Figure 3 FIG. 1 is a flow chart of the third embodiment of the behavior recognition method provided by the present application. Figure 3 As shown, step S11 includes:

[0061] Step S31: construct a heat map space, and divide the heat map space into several grid areas according to the location distribution information.

[0062] Specifically, the thermodynamic space can be set in different shapes and / or sizes according to the user's usage habits, display requirements, etc. For example, the thermodynamic space can be, but is not limited to, a rectangle, a trapezoid, a pentagon, an ellipse, etc. The position distribution information can be used to indicate the distribution of the second target object in the scene to be identified, so that the regional distribution of the heat map space can correspond to the actual distribution of the scene to be identified, and the size of each grid area corresponds to the shape and size of the second target object in the real scene. For example, in a classroom teaching scene, each grid area is used to indicate a student seat, and the size of each grid area is equal or approximately equal.

[0063] Among them, the position distribution information can be distribution information pre-set by the user. For example, the terminal device stores preset position distribution information. In the classroom teaching scenario, the position distribution information is used to indicate that there are 16 student seats (second target objects) distributed in 4*4 rows and columns in the classroom (scene to be identified); or, the position distribution information can also be obtained through the heat map space generated when the behavior recognition was performed last time, for example, the heat map space generated last time is obtained, and the position distribution information is obtained based on the distribution of student seats in the heat map space last time; or, in other embodiments, step S31 can be located after step S32. At this time, the image data of the scene to be identified is first obtained, and several areas to be identified are obtained after target detection is performed on the image data, and then the position distribution information is obtained based on the distribution of the areas to be identified, and the heat map space is segmented.

[0064] Step S32: acquiring image data of the scene to be recognized, and performing target detection on the image data to obtain a plurality of regions to be recognized of the scene to be recognized and second coordinate data corresponding to the regions to be recognized.

[0065] Specifically, after acquiring image data of a scene to be identified, computer vision technology is used to perform target detection on a target image frame. For example, a target detection model is used to perform target detection on the image data. The target detection model is trained with training data consisting of preset scenes to be identified and target objects to identify a number of areas to be identified from the target image frame, and second coordinate data corresponding to the areas to be identified is obtained.

[0066] In an optional implementation, the area to be identified is an area obtained by detecting and segmenting the position of the second target object in the image data by the target detection model. By way of example, in a classroom teaching scenario, the image data may be an image obtained by taking photos or recording videos of the classroom through an image acquisition device, and the area to be identified is a detection frame segmented by the target detection model through detection of all student seats in the image data, as well as the second coordinate data corresponding to the detection frame.

[0067] Step S33: sorting a plurality of second coordinate data, and binding the area to be identified with the grid area according to the sorting result of the second coordinate data to obtain a mapping relationship.

[0068] According to the second coordinate data of the area to be identified, the distribution of all the areas to be identified in the image data can be obtained. For example, when there are 4 areas to be identified and the areas to be identified are distributed in a 2*2 array, the position of each area to be identified in the array can be determined according to the size relationship of the second coordinate data of each area to be identified, and whether it is in which row and which column. Specifically, the distribution of all the areas to be identified in the image data can be obtained by sorting a number of second coordinate data and obtaining the distribution of all the areas to be identified in the image data according to the sorting result of the second coordinate data, so as to bind the areas to be identified with the grid areas corresponding to the heat map space according to the distribution, so that the areas to be identified and the grid areas have a one-to-one mapping relationship in coordinate position.

[0069] Furthermore, the target image frame and image data are data obtained by an image acquisition device to acquire images of the scene to be identified. Due to the perspective transformation of camera imaging, the size and position of the target object presented in the target image frame and image data are different from the real space scene. This embodiment constructs a new heat map space, and sorts the second coordinate data of several areas to be identified, so as to bind the area to be identified and the grid area according to the sorting result, so that the behavior recognition results obtained by subsequent behavior recognition can be projected into the heat map space for display, avoiding the problem of large differences between the image and the real space caused by directly using the collected image to generate a heat map, so that the user can analyze the behavior frequency in the heat map space more intuitively and accurately, thereby further improving the user experience.

[0070] Optionally, the second coordinate data includes horizontal axis coordinate data and vertical axis coordinate data. Figure 4-5 , Figure 4 is a flowchart of the fourth embodiment of the behavior recognition method provided by the present application, Figure 5 FIG. 1 is an operation diagram of an embodiment of the present application for binding the area to be identified with the grid area. Figure 4 As shown, in this embodiment, step S33 further includes the following steps:

[0071] Step S41: sorting a plurality of second coordinate data in ascending order of the vertical axis coordinate data, so as to obtain a plurality of rows of to-be-identified areas according to the sorting result.

[0072] Specifically, the second coordinate data is the position data of the coordinate point on the coordinate axis established in the plane coordinate system of the image data, and the second coordinate data includes horizontal axis coordinate data and vertical axis coordinate data. After obtaining the area to be identified and its corresponding second coordinate data, all the second coordinate data in the image data are sorted in ascending order of the vertical axis coordinate data, so as to obtain multiple rows of areas to be identified according to the sorting result.

[0073] Among them, multiple rows of areas to be identified can be obtained according to the arrangement pattern of the second target object. For example, in a classroom teaching scenario, the second target object is a student seat, and the student seats are usually arranged in rows and columns. Therefore, in a sorting result, the vertical axis coordinate data of the student seats in the same row are usually close or arranged in a certain order, so multiple rows of areas to be identified can be obtained according to a sorting result; or, after performing target detection on the image data, the row and column data of several areas to be identified are obtained. For example, the row and column data indicates that there are 4 columns of seats in the image data. At this time, 1-4 areas to be identified can be directly selected from the sorting result as the first row of areas to be identified, and 5-8 areas to be identified can be selected from the sorting result as the second row of areas to be identified, until all areas to be identified are divided into multiple rows of data.

[0074] Step S42: re-sorting the second coordinate data of each row of the area to be identified in ascending order of the horizontal axis coordinate data.

[0075] The second coordinate data of each row of the to-be-recognized area is further sorted in ascending order of the horizontal axis coordinate data, so that the multiple rows of to-be-recognized areas after the secondary sorting are arranged in order from small to large in the vertical direction, so that the result of the secondary sorting corresponds to the seat distribution in the real scene. It can be understood that both the primary sorting and the secondary sorting arrange the coordinate data in ascending order from small to large.

[0076] Step S43: Binding the area to be identified with the grid area based on the result of the secondary sorting to obtain a mapping relationship.

[0077] The several areas to be identified obtained after the secondary sorting are arranged into multiple rows and columns according to the size relationship of the second coordinate data, and each area to be identified is bound to the grid area according to its distribution information in the rows and columns, so that there is a one-to-one mapping relationship between the area to be identified and the grid area.

[0078] For example, Figure 5As shown in the figure, in the classroom teaching scenario, the heat map space represents the distribution of 16 student seats through a 4*4 grid area; the target detection model detects 16 areas to be identified from the image data, and based on the result of the secondary sorting, 16 areas to be identified arranged in 4*4 rows and columns can be obtained, and each area to be identified is bound to the grid area at the corresponding position in the heat map space according to its distribution information in the rows and columns. For example, the area to be identified in the first row and the first column is bound to the grid area in the first row and the first column (i.e., grid 1), and the area to be identified in the first row and the second column is bound to the grid area in the first row and the second column (i.e., grid 2), ..., until all areas to be identified are bound to the grid areas.

[0079] Optionally, before step S32, the behavior recognition method may further include the following steps: performing image acquisition on the scene to be recognized by an image acquisition device to obtain image data of the scene to be recognized; or, obtaining historical transmission data of the image acquisition device, and obtaining image data of the scene to be recognized from the historical transmission data.

[0080] Specifically, in one embodiment, an image acquisition device can be used to pre-acquire images of the scene to be identified to obtain image data for target detection; in another embodiment, the image acquisition device can be used to acquire historical transmission data, which includes multiple image acquisition data of the scene to be identified before the current time, so that the image data of the scene to be identified can be acquired from the historical transmission data and used for target detection. Since the image data is directly obtained by acquiring images of the real space of the scene to be identified, the image data can accurately reflect the distribution information of the target object in the scene to be identified, so that when the behavior recognition results of the target image frame are subsequently projected into the thermal map space, the thermal map space can accurately reflect the distribution of the target object, which is convenient for users to analyze the behavior of all target objects in the scene to be identified and improve the user experience.

[0081] Optionally, the area to be identified is the detection area of ​​the second target object, and the position distribution information includes the distribution information of the second target object in the scene to be identified; step S32 includes the following steps: performing target detection on the image data to obtain the detection area of ​​the second target object and the second coordinate data of the center point of the detection area.

[0082] Specifically, in this embodiment, the area to be identified is the detection area where the second target object detected in the image data is located. The detection area can be an area (i.e., a detection frame) that frames the location of the second target object. The second coordinate data corresponding to the area to be identified is the second coordinate data of the center point of the detection area, so that the position of the area is represented by the center point position of the detection area.

[0083] The behavior recognition method of the embodiment of the present application can be applied to a terminal device, which can be a server or a system composed of a server and a local terminal. Accordingly, the various parts of the terminal device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or can be set in the server and the local terminal respectively.

[0084] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the behavior recognition method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0085] See also Figure 6 , Figure 6 Schematic diagram of the structure of an embodiment of the terminal device provided by the present application. Figure 6 As shown, the terminal device 100 includes a processor 101 and a memory 102 connected to the processor 101, wherein the memory 102 stores program data, and the processor 101 calls the program data stored in the memory 102 to execute the above-mentioned behavior recognition method.

[0086] Optionally, in one embodiment, the processor 101 is used to execute program data to implement the following method: obtaining a thermal map space and image data of a scene to be identified, the image data including a number of areas to be identified, the thermal map space including a number of grid areas, and the areas to be identified and the grid areas having a mapping relationship; in response to a behavior recognition instruction, performing image acquisition on the scene to be identified at a preset sampling frequency to obtain a number of target image frames of the scene to be identified; performing behavior detection on the target image frames to obtain behavior recognition results corresponding to the target image frames.

[0087] The processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an electronic chip having the ability to process signals. The processor 101 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0088] The memory 102 can be a memory stick, a TF card, etc., which can store all the information in the terminal device 100, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory 102. It stores and retrieves information according to the location specified by the processor 101. With the memory 102, the terminal device 100 has a memory function and can ensure normal operation. The memory 102 of the terminal device 100 can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose, and there is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0089] See also Figure 7 , Figure 7 Schematic diagram of the structure of an embodiment of the computer-readable storage medium provided by the present application. Figure 7 As shown, the computer-readable storage medium 110 stores program instructions 111 that can implement all the above methods.

[0090] If the integrated units of the functional units in the various embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium 110. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer-readable storage medium 110 includes several instructions in a program instruction 111 to enable a computer device (which can be a personal computer, a system server, or a network device, etc.), an electronic device (such as MP3, MP4, etc., or a mobile terminal such as a mobile phone, a tablet computer, a wearable device, or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of various implementation methods of the present application.

[0091] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media 110 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] The present application is described by flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by a computer-readable storage medium 110. These computer-readable storage media 110 can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the program instructions 111 executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0093] These computer-readable storage media 110 may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the program instructions 111 stored in the computer-readable storage medium 110 produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0094] These computer-readable storage media 110 can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing program instructions 111 executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0096] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (which can be a personal computer, server, network device or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment. The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, is also included in the patent protection scope of the present application.

[0097] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A behavior recognition method, characterized in that: include: Acquire a thermal map space and image data of a scene to be identified, wherein the image data includes a plurality of areas to be identified, the thermal map space includes a plurality of grid areas, and the areas to be identified have a mapping relationship with the grid areas; In response to receiving the behavior recognition instruction, performing image acquisition on the scene to be recognized according to a preset sampling frequency to obtain a plurality of target image frames of the scene to be recognized; Performing behavior detection on the target image frame to obtain a behavior recognition result corresponding to the target image frame; Based on the mapping relationship between the area to be identified and the grid area, the behavior recognition results of several target image frames are projected onto the heat map space at corresponding positions to characterize the behavior frequency of the scene to be identified through the thermal value of the heat map space.

2. The behavior recognition method according to claim 1, characterized in that: The behavior recognition result includes first coordinate data and behavior category data of the first target object; The method of projecting the behavior recognition results of the target image frames onto the heat map space at corresponding positions based on the mapping relationship between the area to be recognized and the grid area, so as to characterize the behavior frequency of the scene to be recognized by the heat value of the heat map space, includes: Acquire the second coordinate data of the plurality of to-be-recognized areas, calculate the distance between the first coordinate data and the plurality of the second coordinate data, and define the grid area with the smallest distance as the to-be-recognized area where the first target object of the target image frame is located; Based on the mapping relationship between the area to be identified and the grid area, acquiring the grid area corresponding to the first target object; The thermal value of the grid area corresponding to the target image frame is obtained, and the thermal value is updated based on the behavior category data.

3. The behavior recognition method according to claim 2, characterized in that: The obtaining of the thermal value of the grid area corresponding to the target image frame and updating the thermal value based on the behavior category data includes: Acquire a first thermal value of the grid area corresponding to the first target object in the target image frame; In response to the behavior category data of the first target object including a preset behavior category, the thermal value of the grid area is updated to a second thermal value, where the second thermal value is the sum of the first thermal value and a first preset value.

4. The behavior recognition method according to claim 2, characterized in that: After obtaining the thermal value of the grid area corresponding to the target image frame and updating the thermal value based on the behavior category data, the behavior recognition method further includes: In response to receiving the end instruction, outputting the updated thermal map space of all the target image frames to a preset position; or, In response to receiving the end instruction, the updated heat map space of all the target image frames is visually rendered to output the rendered heat map space to the preset position.

5. The behavior recognition method according to claim 1, characterized in that: The step of acquiring the thermal map space and image data of the scene to be identified, wherein the image data includes a plurality of areas to be identified, the thermal map space includes a plurality of grid areas, and the mapping relationship between the areas to be identified and the grid areas includes: Constructing a heat map space, and dividing the heat map space into a plurality of grid areas according to the location distribution information; Acquire image data of a scene to be identified, and perform target detection on the image data to obtain a plurality of areas to be identified of the scene to be identified and second coordinate data corresponding to the areas to be identified; A plurality of the second coordinate data are sorted, so as to bind the area to be identified with the grid area according to the sorting result of the second coordinate data, so as to obtain the mapping relationship.

6. The behavior recognition method according to claim 5, characterized in that: The second coordinate data includes horizontal axis coordinate data and vertical axis coordinate data; The step of sorting the plurality of second coordinate data to bind the to-be-identified area with the grid area according to the sorting result of the coordinate data to obtain the mapping relationship includes: Sorting the plurality of second coordinate data in ascending order of the vertical axis coordinate data, so as to obtain a plurality of rows of the to-be-identified areas according to the sorting result; Secondarily sorting the second coordinate data of the area to be identified in each row according to the ascending order of the horizontal axis coordinate data; The to-be-identified area is bound to the grid area based on the result of the secondary sorting to obtain the mapping relationship.

7. The behavior recognition method according to claim 5, characterized in that: Before the step of acquiring image data of the scene to be recognized and performing target detection on the image data to obtain a plurality of the regions to be recognized of the scene to be recognized and second coordinate data corresponding to the regions to be recognized, the behavior recognition method further includes: Capturing an image of the scene to be identified by an image acquisition device to obtain image data of the scene to be identified; or The historical transmission data of the image acquisition device is obtained, and the image data of the scene to be identified is obtained from the historical transmission data.

8. The behavior recognition method according to claim 5, characterized in that: The area to be identified is a detection area of ​​a second target object, and the position distribution information includes distribution information of the second target object in the scene to be identified; The step of acquiring image data of the scene to be identified and performing target detection on the image data to obtain a plurality of the regions to be identified of the scene to be identified and second coordinate data corresponding to the regions to be identified includes: Perform target detection on the image data to obtain a detection area of ​​the second target object and second coordinate data of a center point of the detection area.

9. A terminal device, characterized in that: It comprises a processor and a memory connected to the processor, wherein the memory stores program data, and the processor calls the program data stored in the memory to execute the behavior recognition method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: Program instructions are stored, and the program instructions are executed to implement the behavior recognition method as described in any one of claims 1-8.

Images