A data analysis method for a user group and related apparatus

Abstract

The application provides a user group data analysis method and related device, and relates to the field of data analysis. Video information of users in different areas in a user group is collected by at least one data collection terminal, face information and human body key point information of the same user in a key frame are identified, the position of the user is divided based on the face information of the user to determine target users corresponding to the same position attribute, the target users in the key frame at the current moment are matched with the target users in the key frame at the last moment to obtain user tracking results, and the action information of the target users in the user tracking results is determined based on at least one of the face information and the human body key point information of the target users at different moments. The application can avoid the influence of artificial subjective factors on the identification result and improve the data identification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data analysis, and more particularly, to a data analysis method for a user group and related apparatus. BACKGROUND

[0002] In large scenes such as concerts and training, the audience or the trained objects can be referred to as a user group. The number of user groups is generally large. In these scenes, the feelings of the user group are crucial to the effect of the concert or the training.

[0003] At present, taking a concert as an example, the feedback of the audience to the performance is obtained by relying on manual observation, such as a director located in the audience seat determining the audience with high emotions based on subjective experience, and then using a camera to collect the video of the audience with high emotions. This way of determining feedback data such as audience emotions is easily affected by subjective factors, making the accuracy of the determined feedback data of the audience low. SUMMARY

[0004] Therefore, the present application provides a data analysis method for a user group and related apparatus to solve the problem of low accuracy of determining feedback data.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] A data analysis method for a user group, comprising:

[0007] obtaining key frames in a video segment collected by at least one data collection terminal at a current time; each data collection terminal is used to collect video information of users in different areas in the user group;

[0008] identifying face information and body key point information corresponding to the same user in the key frames; the face information at least includes emotion information;

[0009] performing position division operation on the position where the user is located based on the face information of the user to determine target users corresponding to the same position attribute;

[0010] performing matching operation on the target users in the key frames at the current time and the target users in the key frames at the previous time based on the coordinates of the target users in the key frames at the current time and the coordinates of the target users in the key frames at the previous time to obtain user tracking results;

[0011] determining action information of the target users in the user tracking results based on at least one of the face information and the body key point information of the target users at different times in the user tracking results.

[0012] Optionally, the face information and the human body key point information corresponding to the same user in the key frame are identified, including:

[0013] The face information and the human body key point information of the user in the key frame are identified; the face information includes face emotion information and face key point information in a face detection frame;

[0014] The face information and the human body key point information are matched based on the relative distance between the face detection frame and a specified key point in the human body key point information, to obtain the face information and the human body key point information corresponding to the same user.

[0015] Optionally, based on the face information of the user, a position division operation is performed on the position where the user is located to determine a target user corresponding to the same position attribute, including:

[0016] Based on the center height of the face detection frame in the face information, a sorting operation is performed on the user to obtain a sorting result;

[0017] Based on the average height of the face detection frame and the mean value of the center height of the face detection frame, a position classification operation is performed on the user in the sorting result to determine a target user belonging to the same position attribute; the position attribute includes a ranking number.

[0018] Optionally, based on the coordinates of the target user in the key frame of the current moment and the coordinates of the target user in the key frame of the previous moment, a matching operation is performed on the target user in the key frame of the current moment and the target user in the key frame of the previous moment to obtain a user tracking result, including:

[0019] The difference between the horizontal coordinates of each target user in the key frame of the current moment and the horizontal coordinates of each target user in the key frame of the previous moment is calculated;

[0020] According to the difference from small to large, the target user in the key frame of the current moment and the target user in the key frame of the previous moment are matched to obtain a user matching result;

[0021] For a to-be-assigned user in the target user in the key frame of the current moment that is not located in the user matching result, based on the horizontal coordinates of the to-be-assigned user, the position information of the to-be-assigned user is determined;

[0022] Based on the user matching result and the position information of the to-be-assigned user, a user tracking result is determined.

[0023] Optionally, after determining the action information of the target user in the user tracking result, further comprising:

[0024] store the seat number, the play timestamp, the emotion information and the action information of the target user to a target area.

[0025] Optionally, after storing the seat number, the play timestamp, the emotion information and the action information of the target user to the target area, the method further comprises:

[0026] pulling the latest data stored in the target area;

[0027] if the play timestamp in the latest data is less than or equal to a preset time after the pulling time, extracting data of a specified time period stored in the target area at the preset time; the specified time period is a time period between the pulling time and the preset time.

[0028] Optionally, after extracting the data of the specified time period stored in the target area, the method further comprises:

[0029] in a case where the number of users in the latest data is greater than a preset number, determining current emotion distribution data of a user group in the latest data;

[0030] based on the current emotion distribution data, performing an emotion information filling operation on a user in the user group that does not have emotion information;

[0031] in a case where the number of users in the latest data is not greater than the preset number, using default emotion distribution data to perform an emotion information filling operation on a user in the user group that does not have emotion information.

[0032] A data analysis apparatus of a user group, comprising:

[0033] an acquisition module configured to acquire key frames in a video clip collected by at least one data collection terminal at a current time; each data collection terminal is configured to collect video information of a user in a user group located in a different area;

[0034] an identification module configured to identify face information and body key point information corresponding to a same user in the key frames; the face information at least includes emotion information;

[0035] a division module configured to perform a location division operation on a location where the user is located based on the face information of the user, to determine a target user corresponding to a same location attribute;

[0036] a matching module configured to perform a matching operation on the target user in the key frames at the current time and the target user in the key frames at a previous time based on coordinates of the target user in the key frames at the current time and coordinates of the target user in the key frames at the previous time, to obtain a user tracking result;

[0037] determine action information of the target user in the user tracking result based on at least one of the face information and the human body key point information of the target user at different time moments in the user tracking result.

[0038] An electronic device comprising at least one processor and a memory connected to the processor, wherein:

[0039] The memory is configured to store a computer program.

[0040] The processor is configured to execute the computer program to enable the electronic device to implement the user group data analysis method described above.

[0041] A computer storage medium carrying one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the user group data analysis method described above.

[0042] The present application provides a user group data analysis method and related device, in the present application, at least one data acquisition terminal is used to collect video information of users in different areas in a user group, which can cover all users in the area and ensure the comprehensiveness of user analysis. Subsequently, the face information and the human body key point information corresponding to the same user in the key frame are identified, the position of the user is divided based on the face information of the user to determine the target user corresponding to the same position attribute, the target user in the current time moment key frame and the target user in the last time moment key frame are matched based on the coordinates of the target user in the current time moment key frame and the coordinates of the target user in the last time moment key frame, and the user tracking result is obtained. The action information of the target user in the user tracking result is determined based on at least one of the face information and the human body key point information of the target user at different time moments in the user tracking result. That is, in the present application, the user's action and emotion can be automatically identified based on the above process, avoiding the influence of artificial subjective factors on the identification result and improving the data identification accuracy. In addition, in the action recognition, the target user in the user tracking result is determined based on at least one of the face information and the human body key point information of the target user at different time moments. Since the human body key points at different time moments are considered, i.e. the user's posture at different time moments is considered, the accuracy of user action determination is improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings required by the embodiments or the related art description. Obviously, the drawings in the following description are only a part of the present application, and for those skilled in the art, other drawings can be obtained based on the provided drawings without creative labor.

[0044] Figure 1 A flow chart of a user group data analysis method provided by the embodiments of the present application;

[0045] Figure 2 A flow chart of a user determination method provided by the embodiments of the present application;

[0046] Figure 3 A user tracking flow chart provided by the embodiments of the present application;

[0047] Figure 4 A data pulling flow chart provided by the embodiments of the present application;

[0048] Figure 5 A scene schematic diagram of a user group data analysis method provided by the embodiments of the present application;

[0049] Figure 6 A structure schematic diagram of a user group data analysis device provided by the embodiments of the present application;

[0050] Figure 7 A structure schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0052] In large scenes such as variety shows, concerts, training, etc., the audience or the trained object can be referred to as a user group. The number of user groups is generally large. In these scenes, the feelings of the user groups are crucial to the effect of the concert or the training effect.

[0053] At present, taking variety shows and concerts as examples, with the vigorous development of the variety show and live performance market, audience interaction experience has become a key factor to improve the program effect and commercial value. Audience feedback analysis mainly relies on manual observation, such as a director located in the audience seat determines the audience with high emotions based on subjective experience, and then the video of the audience with high emotions can be collected by using the camera, and the corresponding video can be obtained by post-editing. This way of determining audience feedback data such as audience emotions is easily affected by artificial subjective factors, making the accuracy of the determined audience feedback data lower. In addition, there are problems such as poor real-time performance, limited view, single data analysis dimension (such as only analyzing audience emotions, not analyzing audience actions), and it is difficult to fully capture the real reactions of the audience in the live show.

[0054] In recent years, the progress of artificial intelligence and cloud computing technology has provided new possibilities for audience behavior analysis. Through computer vision technology, the expressions and actions of the audience can be identified in real time, providing data support for program production. However, the existing scheme still faces many challenges: on the one hand, the view limitation of a single camera leads to serious audience occlusion problems; on the other hand, the complex live environment (such as dense audience) greatly reduces the recognition accuracy.

[0055] Therefore, in this application, in order to accurately identify audience feedback data in the case of dense audience and view limitation, a multi-terminal cooperative collection and cloud intelligent analysis scheme is adopted, image algorithm (face recognition, face key point recognition, face emotion recognition and human body key point recognition) technology is used to analyze the audience expressions / actions in the concert live show in real time, and the results are synchronized to the live page in real time. When watching the singer perform, the audience can immerse themselves in the live atmosphere. At the same time, the number and frequency of emotional resonance achieved when the singer performs can be used to score the singer's performance, which is more objective and fair than manual scoring.

[0056] Specifically, the application provides a user group data analysis method and related device. In the application, at least one data collection terminal is used to collect video information of users in different areas in a user group, which can cover all users in the areas and ensure the comprehensiveness of user analysis. Subsequently, face information and human body key point information corresponding to the same user in the key frame are identified. Based on the face information of the user, a location division operation is performed on the location where the user is located to determine target users corresponding to the same location attribute. Based on the coordinates of the target user in the current key frame and the coordinates of the target user in the key frame of the previous moment, a matching operation is performed on the target user in the current key frame and the target user in the key frame of the previous moment to obtain a user tracking result. Based on at least one of the face information and the human body key point information of the target user at different moments in the user tracking result, action information of the target user in the user tracking result is determined. That is, in the application, the user's action and emotion can be automatically identified based on the above process, which avoids the influence of artificial subjective factors on the identification result and improves the data identification accuracy. In addition, in the action recognition, the target user in the user tracking result is determined based on at least one of the face information and the human body key point information of the target user at different moments. Since the human body key points at different moments are considered, i.e., the user's posture at different moments is considered, the accuracy of user action determination is improved.

[0057] In an implementation manner, referring to Figure 1 A user group data analysis method can include:

[0058] S11, acquiring a key frame in a video slice collected by at least one data collection terminal at a current moment.

[0059] Each data collection terminal is configured to collect video information of users in different areas in a user group.

[0060] In a specific implementation, taking a concert as an example, the overall audience area is divided into multiple subareas, and the number arrangement order of the audience in each subarea is numbered and recorded to obtain a seat distribution map. Terminals are erected on site and image acquisition parameters and angles are debugged. Multiple terminals (such as mobile phones, cameras, etc.) are fixed in different areas in front of the audience area, and the sum of the coverage areas of the multiple terminals should be all the audience areas of the concert. Subsequently, the audience area picture can be started to be shot, and the angle of shooting is ensured to be parallel to the horizontal line as much as possible. The terminal transmits the video shot by using an application program APP to a live cloud server through an RTMP (Real-Time Messaging Protocol, real-time messaging protocol) protocol. In addition, in addition to using the RTMP protocol for data transmission, RTC (Real-Time Communication, real-time communication protocol), m3u8, etc. can also be used, and the specific configuration is determined according to actual conditions, so as to ensure the uniformity of stream delay and the stability of the picture.

[0061] For each terminal, the live cloud server receives the data transmitted by the terminal in real time, converts the network live stream uploaded by the terminal into a local m3u8 format video file list through a live stream conversion process, and records the timestamp Stream_start_time when the conversion starts. When the video stream is interrupted and recovered due to various reasons, the current process needs to automatically start the conversion again and record the timestamp.

[0062] The reading and analyzing process continuously reads the local m3u8 format video file list to calculate the timestamp of each video segment, and the method for calculating the timestamp is as follows:

[0063] If the writing time of the current m3u8 file list is 8:00:00, and the duration of the video segment is 2 seconds per segment, then the timestamps of the 1st / 2nd / 3rd / 4th video segments are 8:00:02 / 8:00:04 / 8:00:06 / 8:00:08 respectively. Then, the key frames in the video segment are analyzed.

[0064] It should be noted that the live stream conversion process and the reading and analyzing process in the present application are in a multi-copy mode, and one terminal corresponds to one copy.

[0065] S12, identifying the face information and the human body key point information corresponding to the same user in the key frame.

[0066] The face information at least includes emotion information, and the emotion information can be happy, sad, angry, etc.

[0067] In an implementation manner, the face information includes face emotion information and face key point information in a face detection frame.

[0068] The model can be used to detect the human key point information, the facial emotion information in the face detection frame and the face key point information. After the facial emotion information, the face key point information and the human key point information are recognized, the corresponding relationship of the facial emotion information, the face key point information and the human key point information needs to be established, that is, it is determined which facial emotion information, face key point information and human key point information belong to the same audience.

[0069] In an implementation manner, the step S12 can include:

[0070] 1) recognizing the face information and the human key point information of the user in the key frame.

[0071] The face information includes the facial emotion information in the face detection frame and the face key point information.

[0072] In the specific implementation, the face detection model can be used to detect the face region in the key frame, and the face detection frame is used to frame the face region. The face detection frame is set as a rectangular region with the face center point as the center, the height of 4 / 5 of the shoulder width and the width of 3 / 5 of the shoulder width.

[0073] Then, the face emotion model is used to recognize the emotion of the face region framed by the face detection frame, and the face key point model is used to recognize the key point of the face region framed by the face detection frame, which is called face key point information. The face key point information can be the nose, eyes, mouth and the like.

[0074] In addition, the human key point model can be used to analyze the key frame to obtain the human key point information. The human key point information can be the joint key point of the human body, such as the head center point, wrist, elbow, shoulder, knee and the like.

[0075] The model in the present application can be used after being trained on a general deep learning model.

[0076] 2) performing a matching operation on the face information and the human key point information based on the relative distance between the face detection frame and the specified key point in the human key point information, to obtain the face information and the human key point information corresponding to the same user.

[0077] The specified key point in the human key point information can be the head center point.

[0078] In the specific implementation, the center point of each face detection frame is matched with the head center point in each human key point information. If the relative distance between the center point of the face detection frame and the head center point is less than a threshold value, it is determined that the user corresponding to the face detection frame and the user corresponding to the head center point are the same user.

[0079] Wherein, the threshold value can be set as the height of the face detection frame bbox height, and the face information of the human body key point information that does not match the face information is empty. The face information of the human body key point information that does not match the face information is empty.

[0080] After this step, the face and body of the same user can be matched, so that the emotion and action of the same user can be defined.

[0081] S13, based on the face information of the user, the position of the user is divided into position classification operation to determine the target user corresponding to the same position attribute.

[0082] In this application, after matching the emotion and human body key point of the same user through the above steps, the user needs to be divided into rows. The seat number of the audience seat is divided into rows and columns. In this application, the user is divided into rows, so that it can be determined which row the user belongs to. Subsequently, it is determined which seat the user belongs to.

[0083] Wherein, the same position attribute in this application is the same order number, that is, the user in the same row is selected as the target user.

[0084] In an implementation manner, referring to Figure 2 , step S13 can include:

[0085] S21, based on the center height of the face detection frame in the face information, the user is sorted to obtain a sorting result.

[0086] In a specific implementation, after obtaining all the face detection frames in a key frame, the center point height of the face detection frame is used as a value to sort all the face detection frames to obtain a sorting result. In the sorting result, the smaller the center point height of the face detection frame, the more forward the sorting, and the larger the center point height of the face detection frame, the more backward the sorting.

[0087] In an actual scene, the center point height of the face detection frame of the user in the same row is similar, so that the center point height of the face detection frame can be used for subsequent row division operation.

[0088] S22, based on the average height of the face detection frame and the center height mean value of the face detection frame, the user in the sorting result is classified by position to determine the target user belonging to the same position attribute.

[0089] Wherein, the position attribute includes an order number.

[0090] In an actual scene, the center point heights of the face detection boxes of users in the same row are similar, and the height difference of the face detection boxes of different users in the same row does not exceed the average height of the face detection boxes of the users in the row. Therefore, based on this principle, the users are divided into rows.

[0091] In a specific implementation, the first data in the sorting result is taken out, the center point height of the face detection box of the first data is taken as the face box center height mean of the current row, and the height difference of the face detection box of the first data is taken as the face box average height bbox_y_mean of the current row.

[0092] Then the second data in the sorting result is taken out. If the difference between the center point height of the face detection box in the second data and the center_y_mean does not exceed the bbox_y_mean, it indicates that the user corresponding to the second data and the user corresponding to the first data are in the same row. Then the average of the center point height of the face detection box of the first user and the center point height of the face detection box of the second user is taken as the new face box center height mean center_y_mean of the current row, and the average of the height difference of the face detection box of the first data and the height difference of the face detection box of the second data is taken as the new face box average height bbox_y_mean of the current row.

[0093] Subsequently, the second data in the sorting result is continuously taken out, and the row division operation is performed according to the above process. If the difference between the center point height of the face detection box in the third data and the center_y_mean does not exceed the bbox_y_mean, it indicates that the user corresponding to the third data and the two previous users are in the same row. If the difference between the center point height of the face detection box in the third data and the center_y_mean exceeds the bbox_y_mean, it indicates that the user corresponding to the third data and the two previous users are not in the same row. At this time, the user corresponding to the third data is taken as a user in the row behind the two previous users, and the statistics of a new row is started. Until each data in the sorting result is analyzed, the detection data list classified by rows is obtained.

[0094] In this embodiment, by classifying the users by rows, the user can be accurately positioned on the xth row and the yth seat in the subsequent process, the user position is accurately positioned, and the user tracking at different moments is realized, and the user posture is accurately determined.

[0095] S14, based on the coordinates of the target user in the key frame of the current moment and the coordinates of the target user in the key frame of the previous moment, the target user in the key frame of the current moment and the target user in the key frame of the previous moment are matched to obtain a user tracking result.

[0096] In this embodiment, after determining the user's ranking number, the user tracking operation can be implemented based on the key frames of the two previous and subsequent moments, and the same user in the key frames at different moments can be determined, thereby determining the user's action based on the posture of the same user at different moments.

[0097] S15. Determine action information of the target user in the user tracking result based on at least one of facial information and body key point information of the target user at different moments in the user tracking result.

[0098] In this embodiment, the action information may include various action postures such as standing, swaying, and clapping.

[0099] The standing determination process involves calculating the angle between the torso and thigh, and the length ratio between the torso and thigh, based on key point information. The closer the angle is to 180 degrees, the closer the body is to being upright (or facing the camera). When standing, if the torso-thigh length ratio is relatively small, this may be due to sitting facing the camera. The angle and length ratio between the line formed by the left shoulder and left thigh root and the line formed by the left thigh root and left knee joint are calculated. If the angle is greater than 170 degrees and the length ratio is less than 2, the person is considered to be standing on the left side. The same method is used to determine if the person is standing on the right side.

[0100] It should be noted that when judging whether the audience is standing, since only the audience in the front row will block the audience in the back row, the standing judgment needs to be made in the order from the front row to the back row.

[0101] In one implementation, since the legs of back-row spectators are often obscured, a standing mask is drawn for the face and torso detection frames of spectators identified as standing. If the ratio of the intersection of the back-row spectator's torso and the standing mask to the spectator's torso area is greater than 0.05, occlusion is determined, and the spectator is considered standing. For each spectator identified as standing, the standing mask is updated, setting the standing mask for the face and torso detection frames to 1.

[0102] The process for determining whether an applause is occurring is as follows: Applause is a dynamic action. First, the wrist-elbow angle is determined to be between 30 and 60 degrees. If so, the wrist distance is calculated; otherwise, the wrist distance is returned as None. The wrist distances for the last four consecutive moments are then evaluated. If the minimum value of these consecutive non-None wrist distances is less than 1 / 4 of the shoulder length, the applause is considered occurring.

[0103] The judgment process of the swing is as follows: the position of the head center point head_center is recorded, and the angle of the head and the navel head_angle is recorded. Starting from the second key frame, the position distance change of the head center point of the adjacent frame is calculated, the head distance change and the head angle of the last 5 continuous time points are calculated, if the difference between the maximum head angle and the minimum head angle is greater than 15 degrees (there is an angle change), the head distance change is greater than 0.1 times the shoulder length and less than 1 times the shoulder length (there is a head displacement, and the displacement is not too large), it is judged as swing.

[0104] In addition, for other action postures, there are corresponding judgment logics, and action judgment can be performed by using the corresponding judgment logics.

[0105] In the embodiment, the video information of the users in different regions in the user group is collected by using at least one data collection terminal, all the users in all regions can be covered, and the comprehensiveness of user analysis is ensured. The face information and the human body key point information corresponding to the same user in the key frame are recognized, the position of the user is divided based on the face information of the user, to determine the target user corresponding to the same position attribute, the target user in the current key frame is matched with the target user in the key frame of the last time based on the coordinates of the target user in the current key frame and the coordinates of the target user in the key frame of the last time, to obtain a user tracking result, and the action information of the target user in the user tracking result is determined based on at least one of the face information and the human body key point information of the target user at different times in the user tracking result. That is, in the present application, the action and emotion of the user can be automatically recognized based on the above process, the influence of artificial subjective factors on the recognition result is avoided, and the data recognition accuracy is improved. In addition, in the action recognition, the target user is determined based on at least one of the face information and the human body key point information of the target user at different times in the user tracking result, because the human body key points at different times are considered, that is, the postures of the user at different times are considered, so that the accuracy of the determination of the user action is improved.

[0106] On the basis of any of the above embodiments, with reference to Figure 3 , the target user in the current key frame is matched with the target user in the key frame of the last time based on the coordinates of the target user in the current key frame and the coordinates of the target user in the key frame of the last time, to obtain a user tracking result, which can include:

[0107] S31, calculating the difference between the horizontal coordinates of each target user in the current key frame and the horizontal coordinates of each target user in the key frame of the last time.

[0108] If the key frame of the current time is the first key frame, there is no key frame of the last time, and at this time, the seat numbers can be assigned to each row of audience according to the left-right order based on the stored seat distribution map, and the seat positions are recorded.

[0109] If the key frame of the current time is not the first key frame, there is a key frame of the last time, and at this time, the seat numbers of each audience in the current key frame can be matched by using a seat matching method based on the minimum Euclidean distance, that is, the horizontal coordinates of the current row of audience are matched with the horizontal coordinates of the historical audience in the same row by the minimum distance principle, and then the seat numbers of each audience in the current key frame are matched. In this application, the seat numbers of each audience in the current key frame are matched because the live page needs to display the expressions / motions at the positions corresponding to the seats, so it is necessary to assign a seat number to each detection result of the expression / motion.

[0110] When the horizontal coordinates of the current row of audience are matched with the horizontal coordinates of the historical audience in the same row by the minimum distance principle, first, the horizontal coordinates of the audience in the current key frame and the horizontal coordinates of the audience in the last key frame are converted into two-dimensional NumPy arrays, respectively. In the actual scene, for the audience in the same row, there may be temporary going out, coming back, etc., so that the number of audience in the current key frame is the same as or different from the number of audience in the last key frame. The horizontal coordinates of the audience in the last key frame are represented by array a, and the value a is an N×1-dimensional array, where N is the number of audience in the last key frame. The horizontal coordinates of the audience in the current key frame are represented by array b, and the value b is an M×1-dimensional array, where M is the number of audience in the last key frame.

[0111] In an example, the examples of array a and array b can be referred to Table 1.

[0112] Table 1

[0113] a 100 200 300 500 b 220 90 490 620

[0114] In Table 1, the horizontal coordinates are sorted according to the center height of the face detection frame. In an implementation, for example, the horizontal coordinate value increases from left to right, and the height of the left user is higher than that of the right user. When sorting according to the center height of the face detection frame, the left user is sorted later than the right user, and at this time, the left user with a smaller horizontal coordinate is located behind the right user with a larger horizontal coordinate, that is, the left user 90 is located to the right of the right user 220 in Table 1.

[0115] Then, the difference between the horizontal coordinate of each target user in the key frame at the current time and the horizontal coordinate of each target user in the key frame at the last time is calculated. That is, the absolute difference of each element in the two arrays is calculated, and the formula is: D(i,j) = |a_i - b_j|, where i∈[1,N], j∈[1,M].

[0116] In the specific calculation, taking Table 2 as an example, taking 100 in array a as an example, the difference with each value in array b is calculated, that is, the difference of 100 with 220, 90, 490 and 620 is calculated, and then the difference of other values in array a, that is, 200, 300 and 500, with each value in array b is calculated. The difference specifically refers to the absolute difference. The specific calculation result is shown in Table 2 or stored in the constructed N*M dimensional distance matrix.

[0117] Table 2

[0118]

[0119] S32, according to the order of the difference from small to large, the target users in the key frame at the current time and the target users in the key frame at the last time are matched to obtain a user matching result.

[0120] Specifically, as shown in Table 2, the order of the absolute difference from small to large is 10, 10, 20, 80, ….

[0121] In the iterative matching process, the elements in array a and the elements in array b are matched according to the order of the absolute difference from small to large.

[0122] First, the position index (i,j) of the current minimum distance element is located in the distance matrix. In practice, the smallest absolute difference is 10, which is obtained by subtracting 100 in array a and 90 in array b. Therefore, 100 in array a and 90 in array b can form a seat pair, that is, (100, 90). At this time, (100, 90) belongs to the first matching result in the user matching result.

[0123] Then, the matched a_i and b_j elements are removed from the original array, that is, 100 is removed from array a and 90 is removed from array b. At this time, 200, 300 and 500 are left in array a, and 220, 490 and 620 are left in array b. Moreover, the distance matrix is updated according to the new array a and b.

[0124] Locate the position index (i, j) of the current minimum distance element in the new distance matrix, at this time, the minimum absolute difference in the new distance matrix is 10, which is obtained by subtracting 500 in array a and 490 in array b, then 500 in array a and 490 in array b can form a seat pair, that is, (500, 490).

[0125] Determine the position relationship between the new seat pair (500, 490) and (100, 90) in the user matching result, the position relationship is that 500 in (500, 490) in array a is greater than 100 in (100, 90) in array a, and 490 in (500, 490) in array b is greater than 90 in (100, 90) in array b, the position relationship verification passes, the new seat pair (500, 490) is added to the user matching result, and the seat pairs in the user matching result are arranged in ascending order of a_i.

[0126] If the position relationship verification fails, the matching ends and the iteration is exited.

[0127] Then, remove the matched a_i and b_j elements from the original array and dynamically update the distance matrix.

[0128] Locate the position index (i, j) of the current minimum distance element in the new distance matrix, at this time, the minimum absolute difference in the new distance matrix is 20, which is obtained by subtracting 200 in array a and 220 in array b, then 200 in array a and 220 in array b can form a seat pair, that is, (200, 220).

[0129] Determine the position relationship between the new seat pair (200, 220) and (100, 90), (500, 490) in the user matching result, the position relationship is:

[0130] Among (100, 90), (200, 220) and (500, 490), the size is sorted as (100, 90), (200, 220) and (500, 490), the elements 100, 200 and 500 in array a increase in turn, and 90, 220 and 190 in array b also increase in turn, which indicates that the sorting result of a_i in each seat pair (a_i, b_j) is the same as that of b_j, that is, the position relationship verification passes. At this time, the new seat pair (200, 220) is added to the user matching result, and the seat pairs in the user matching result are arranged in ascending order of a_i. That is, (100, 90), (200, 220) and (500, 490). Then repeat the above steps until the elements in any array are processed.

[0131] If the position relation check fails, it means that the matching ends and the iteration is exited.

[0132] Then, the matched a_i and b_j elements are removed from the original arrays and the distance matrix is dynamically updated.

[0133] At this time, only 300 is left in array a and only 620 is left in array b, forming (300, 620), and then the position relation check operation is performed. Among (100, 90), (200, 220), (300, 620) and (500, 490), the elements 100, 200, 300 and 500 in array a increase in turn, and 90, 220, 620 and 490 in array b do not satisfy the rule of increasing in turn, so the position relation check fails, which means that the user with horizontal coordinate 300 in array a is not the same person as the user with horizontal coordinate 620 in array b, and the user with horizontal coordinate 620 did not exist in the key frame at the previous time, which may be a temporary user returning, so the matching ends and the iteration is exited.

[0134] At this time, the user matching result obtained is (100, 90), (200, 220) and (500, 490).

[0135] S33, for the to-be-assigned user in the target users in the key frame at the current time and not in the user matching result, based on the horizontal coordinate of the to-be-assigned user, the position information of the to-be-assigned user is determined.

[0136] Specifically, the to-be-assigned user in the target users in the key frame at the current time and not in the user matching result is the user corresponding to the horizontal coordinate 620. Since 620 is greater than the maximum value 500 in array b in the user matching result, the user with horizontal coordinate 620 should be inserted after (500, 490).

[0137] S34, based on the user matching result and the position information of the to-be-assigned user, a user tracking result is determined.

[0138] Specifically, in the final user tracking result, the horizontal coordinates are sorted in ascending order as follows:

[0139] (100, 90), (200, 220), (500, 490) and 620.

[0140] After the user tracking result is determined, the body posture of each seat number, such as the seat id, can be determined, which includes standing, swinging, clapping and other action postures.

[0141] In this embodiment, the seat matching method based on the minimum Euclidean distance matches the users at the previous time and the current time, finds the same users at the same seat number, and can determine the motion posture of the user by analyzing the posture change of the user.

[0142] In an implementation, after determining the motion information of the target user in the user tracking result, the seat number, the play timestamp, the emotion information and the motion information of the target user can be stored in the target area.

[0143] In a specific implementation, the target area is a redis cache, and a data list composed of the seat number, the face emotion, the body posture and the play timestamp of the target user is placed in the redis cache to realize real-time saving of data.

[0144] In an implementation, with reference to Figure 4 After storing the seat number, the play timestamp, the emotion information and the motion information of the target user in the target area, the method further includes:

[0145] S41, pulling the latest data stored in the target area.

[0146] The data in the redis cache is pulled at a fixed time, and it is assumed that the pulling time of the current pulled data, that is, the pulling timestamp is timestamp. The timestamp of the latest data pulled from the redis cache is data_lasttime. The period of pulling data at a fixed time can be the same as the length of the video segment described above, and the period of pulling data at a fixed time is 2 seconds. In addition, the period of pulling data at a fixed time can also be configured according to actual conditions, such as 1 second.

[0147] S42, if the play timestamp in the latest data is less than or equal to a preset time after the pulling time, data stored in the target area in a specified time period is extracted at the preset time.

[0148] The specified time period is the time period between the pulling time and the preset time.

[0149] In a specific implementation, if data_lasttime is less than or equal to timestamp, it indicates that there is no updated latest data in the redis cache, which can be caused by the delay of data analysis, that is, the data is not analyzed in time and stored in the redis cache, and the current batch data is empty.

[0150] If data_lasttime is less than timestamp + the period of the timed data pulling, the period of the timed data pulling can be 2 seconds or 1 second. Taking the period of the timed data pulling as 2 seconds as an example, wait for timestamp + 1 - data_lasttime seconds. For example, if timestamp is 10 seconds, data_lasttime is 10.05 seconds, and the period of the timed data pulling is 1 second, timestamp + the period of the timed data pulling is 11 seconds, and 10.05 seconds is less than 11 seconds, wait for a certain time period, and the certain time period is timestamp + the period of the timed data pulling - data_lasttime, that is, wait for 0.5 seconds. At this time, the preset time is the time after waiting for 0.5 seconds. When the preset time is reached, the cached data between timestamp and timestamp + 1 is pulled from the redis, so that the data can be normally pulled.

[0151] In an implementation manner, after the data of the specified time period stored in the target region is extracted, the method further includes:

[0152] In a case where the number of users in the latest data is greater than a preset number, current emotion distribution data of a user group in the latest data is determined, and emotion information filling operation is performed on a user without emotion information in the user group based on the current emotion distribution data.

[0153] In a specific implementation, in order to avoid the problem that the amount of emotion or action information detected is small, resulting in incomplete display of users on the live picture, in the embodiment, data filling operation is performed.

[0154] In a specific data filling, the number of users in the latest data is directly obtained. If the number of users is greater than a preset number, the preset number can be 0.1 of the expected audience size collected by the corresponding terminal. If the number of users in the latest data is greater than the preset number, it indicates that the number of users collected in the latest data is large. At this time, the current emotion distribution data of the user group in the latest data can be determined.

[0155] The current emotion distribution data refers to the proportion of different emotions in the latest data.

[0156] For example, the number of users in the latest data is 80 in total, and the number of happy people is 20. Therefore, the proportion of happiness is 0.25, and the proportion of the remaining emotions is determined in the same way. The obtained current emotion distribution data is used as historical emotion distribution data.

[0157] Then, the seat numbers are traversed, if a seat number does not exist in the latest data, it indicates that the key frame does not detect the emotion of the user of the seat, at this time, the historical emotion distribution data can be used to fill the emotion of the user of the seat number, for example, if the emotion of 20 users is not detected, 4 of the 20 people are filled with happiness, and the specific 4 people can be randomly selected or selected according to rules.

[0158] In an implementation manner, in a case where the number of users in the latest data is not greater than a preset number, default emotion distribution data is used to perform an emotion information filling operation on a user in the user group for which emotion information does not exist.

[0159] Specifically, in a case where the number of users in the latest data is not greater than a preset number, it indicates that the number of users collected in the latest data is small, at this time, the current emotion distribution data of the user group in the latest data can not be representative, and the default emotion distribution data can be used to perform an emotion information filling operation on a user in the user group for which emotion information does not exist.

[0160] The default emotion distribution data can be configured according to actual conditions, for example, all are calm, or half are happy and half are calm.

[0161] In the embodiment, data processing and bottom operation are designed, the missing data is filled, and the integrity of the data is ensured. After the data is completed, the data is sent to the front end for use by the live page.

[0162] It should be noted that the live picture of the concert usually has a delay, which is usually several seconds to several minutes. In the embodiment, the live delay of the concert is 2.5 minutes, and the delay of the picture capture and analysis data of the audience seat is about 10 seconds. When sent to the live page, the delay time of 10 seconds is subtracted from the timestamp of the live data, and the front end searches the real-time time when searching.

[0163] As shown in FIG. 8, Figure 5 In the embodiment of the application, the above terminal is taken as a mobile phone as an example, the audience seat picture collected by the multiple mobile phones is uploaded to the cloud server, the cloud server tracks and analyzes the audience expression / action, stores the picture in the redis cache, integrates and completes the data and sends the front end, and the real-time monitoring of the audience seat state is realized, for example, 3 rows of 20 seats and 64 seats are standing up and cheering, which can be used for real-time collection and display of the audience emotion in the concert or live scene, and the immersion and interactivity of the live scene are enhanced. The embodiment of the application has the following advantages:

[0164] Enhance user experience: through the embodiment of the application, the audience's expressions / movements are displayed in real time, enhancing the sense of participation in the live performance and improving the user's click rate. When the system detects that more than 50% of the audience "stands up and cheers", it can trigger a full-field emotional resonance, and the number of emotional resonances for each singer after singing can objectively represent the audience's welcome for the song.

[0165] Reduce operating costs: this method can collect audience emotions / movements in real time, and uses terminals instead of professional cameras + cloud server solutions, which reduces the labor cost of manual operation compared to the way of manually finding and collecting audience, and creates a low-cost, high-precision, and strong real-time audience emotion and behavior analysis system, realizes multi-dimensional data collection and intelligent analysis, and becomes an important technical breakthrough point for improving the live entertainment experience.

[0166] In addition, through the multi-terminal real-time collection of audience pictures in the embodiment of the application, the data is processed by the real-time analysis and data processing background and then displayed on the live page. The audience can feel the live atmosphere when watching the live performance, increasing the sense of immersion. At the same time, each singer can be given an objective emotional resonance score according to the emotional resonance caused by each singer in each performance.

[0167] On the basis of the above-mentioned embodiment of the data analysis method of the user group, another embodiment of the application provides a data analysis device of a user group, which refers to Figure 6 may include

[0168] The acquisition module 11 is configured to acquire key frames in a video segment collected by at least one data collection terminal at a current time; each data collection terminal is configured to collect video information of users in different areas in the user group;

[0169] The identification module 12 is configured to identify face information and body key point information corresponding to the same user in the key frames; the face information at least includes emotion information;

[0170] The division module 13 is configured to perform position division operation on the position of the user based on the face information of the user, to determine target users corresponding to the same position attribute;

[0171] The matching module 14 is configured to match the target user in the key frame at the current time with the target user in the key frame at the previous time based on the coordinates of the target user in the key frame at the current time and the coordinates of the target user in the key frame at the previous time, to obtain a user tracking result;

[0172] The determination module 15 is configured to determine action information of the target user in the user tracking result based on at least one of the face information and the body key point information of the target user at different times in the user tracking result.

[0173] In one implementation, the identification module 12 is specifically configured to:

[0174] The facial information and body key point information of the user in the key frame are identified; the facial information includes facial emotion information and facial key point information in the face detection frame, and based on the relative distance between the face detection frame and the specified key point in the body key point information, the facial information and the body key point information are matched to obtain facial information and body key point information corresponding to the same user.

[0175] In one implementation, the division module 13 is specifically configured to:

[0176] Based on the center height of the face detection frame in the facial information, the users are sorted to obtain a sorting result; based on the average height of the face detection frame and the mean of the center height of the face detection frame, the users in the sorting result are position-classified to determine target users belonging to the same position attribute; the position attribute includes a sorting number.

[0177] In one implementation, the matching module 14 includes:

[0178] a calculation submodule, configured to calculate a difference between a horizontal coordinate of each target user in the key frame at the current moment and a horizontal coordinate of each target user in the key frame at the previous moment;

[0179] A matching submodule, configured to match the target user in the key frame at the current moment with the target user in the key frame at the previous moment in ascending order of the difference values, to obtain a user matching result;

[0180] an information determination submodule, configured to determine, for a user to be assigned that is not located in the user matching result among target users in the key frame at the current moment, location information of the user to be assigned based on the horizontal coordinates of the user to be assigned;

[0181] The result determination submodule is configured to determine the user tracking result based on the user matching result and the location information of the user to be assigned.

[0182] In one implementation, the method further includes:

[0183] The cache module is used to store the seat number, play timestamp, emotion information and action information of the target user in the target area.

[0184] In one implementation, the method further includes:

[0185] A pulling module, used to pull the latest data stored in the target area;

[0186] extracting the data of the specified time period stored in the target region at the preset time if the play time stamp in the latest data is less than or equal to the preset time after the pulling time; the specified time period is a time period between the pulling time and the preset time.

[0187] In an implementation manner, the method further includes:

[0188] The first filling module is configured to: in a case where the number of users in the latest data is greater than a preset number, determine current emotion distribution data of a user group in the latest data, and perform emotion information filling operation on a user without emotion information in the user group based on the current emotion distribution data;

[0189] The second filling module is configured to: in a case where the number of users in the latest data is not greater than the preset number, perform emotion information filling operation on a user without emotion information in the user group using default emotion distribution data.

[0190] In the embodiment, the video information of the users in different regions in the user group is collected by using at least one data collection terminal, so that all users in all regions can be covered and the comprehensiveness of user analysis is ensured. The face information and the human body key point information corresponding to the same user in the key frame are identified, the location of the user is divided based on the face information of the user to determine the target user corresponding to the same location attribute, the target user in the key frame of the current time is matched with the target user in the key frame of the previous time based on the coordinates of the target user in the key frame of the current time and the coordinates of the target user in the key frame of the previous time, the user tracking result is obtained, and the action information of the target user in the user tracking result is determined based on at least one of the face information and the human body key point information of the target user at different times in the user tracking result. That is, in the present application, the action and emotion of the user can be automatically identified based on the above process, the influence of artificial subjective factors on the identification result is avoided, and the data identification accuracy is improved. In addition, in the action identification, the target user at different times is determined based on at least one of the face information and the human body key point information of the target user in the user tracking result. Since the human body key points at different times are considered, that is, the postures of the user at different times are considered, the accuracy of the determination of the action of the user is improved.

[0191] It should be noted that the working processes of the modules and sub-modules in the embodiments are described above, and will not be described herein again.

[0192] In the embodiments of the present application, an electronic device is also provided, which includes at least one processor and a memory connected with the processor, wherein:

[0193] The memory is configured to store a computer program.

[0194] The processor is configured to execute the computer program to enable the electronic device to implement the user group data analysis method described above.

[0195] Reference Figure 7 As shown in the figure, a structural schematic diagram of an electronic device suitable for use in the embodiments of the present application is shown. The electronic device in the embodiments of the present application can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, and the like. Figure 7 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0196] As Figure 7 shown, the electronic device can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. In the state that the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0197] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a memory card, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices can be implemented or provided instead.

[0198] The embodiments of the present application also provide a computer program product including computer readable instructions, which, when executed on an electronic device, enable the electronic device to implement any one of the user group data analysis methods provided by the embodiments of the present application.

[0199] The embodiment of the present application further provides a computer readable storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can realize any one of the data analysis methods for user groups provided by the embodiment of the present application.

[0200] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for analyzing user group data, characterized in that: include: Obtaining key frames in video segments collected by at least one data collection terminal at a current moment; Each of the data collection terminals is used to collect video information of users in different areas of the user group; Identifying facial information and body key point information corresponding to the same user in the key frame; the facial information at least includes emotional information; Based on the facial information of the user, performing a location division operation on the user's location to determine target users corresponding to the same location attribute; Based on the coordinates of the target user in the key frame at the current moment and the coordinates of the target user in the key frame at the previous moment, a matching operation is performed on the target user in the key frame at the current moment and the target user in the key frame at the previous moment to obtain a user tracking result; Based on at least one of facial information and body key point information of the target user in the user tracking result at different moments, action information of the target user in the user tracking result is determined.

2. The method for analyzing user group data according to claim 1, characterized in that: Identifying facial information and body key point information corresponding to the same user in the key frame, including: Identify the user's facial information and body key point information in the key frame; the facial information includes facial emotion information and facial key point information in the face detection frame; Based on the relative distance between the face detection frame and the designated key point in the human body key point information, a matching operation is performed on the face information and the human body key point information to obtain the face information and the human body key point information corresponding to the same user.

3. The method for analyzing user group data according to claim 1, wherein: Based on the facial information of the user, a location segmentation operation is performed on the location of the user to determine target users corresponding to the same location attribute, including: performing a sorting operation on the users based on the center heights of the face detection frames in the face information to obtain a sorting result; Based on the average height of the face detection frame and the average center height of the face detection frame, a position classification operation is performed on the users in the sorting result to determine target users belonging to the same position attribute; the position attribute includes a sorting number.

4. The method for analyzing user group data according to claim 1, wherein: Based on the coordinates of the target user in the key frame at the current moment and the coordinates of the target user in the key frame at the previous moment, a matching operation is performed on the target user in the key frame at the current moment and the target user in the key frame at the previous moment to obtain a user tracking result, including: Calculating the difference between the horizontal coordinate of each target user in the key frame at the current moment and the horizontal coordinate of each target user in the key frame at the previous moment; Matching the target user in the key frame at the current moment with the target user in the key frame at the previous moment in ascending order of the difference values ​​to obtain a user matching result; For a user to be assigned that is not in the user matching result among the target users in the key frame at the current moment, determining the position information of the user to be assigned based on the horizontal coordinate of the user to be assigned; A user tracking result is determined based on the user matching result and the location information of the user to be assigned.

5. The method for analyzing user group data according to claim 1, wherein: After determining the action information of the target user in the user tracking result, the method further includes: The seat number, play timestamp, emotion information and action information of the target user are stored in the target area.

6. The method for analyzing user group data according to claim 5, characterized in that: After storing the target user's seat number, play timestamp, emotion information, and action information in the target area, the method further includes: Pulling the latest data stored in the target area; If the playback timestamp in the latest data is less than or equal to a preset time after the pulling time, at the preset time, data of a specified time period stored in the target area is extracted; the specified time period is the time period between the pulling time and the preset time.

7. The method for analyzing user group data according to claim 6, wherein: After extracting the data of the specified time period stored in the target area, the method further includes: When the number of users in the latest data is greater than a preset number, determining current emotion distribution data of the user group in the latest data; Based on the current emotion distribution data, performing an emotion information filling operation on users in the user group for whom emotion information does not exist; When the number of users in the latest data is not greater than a preset number, default emotion distribution data is used to perform an emotion information filling operation on users in the user group for whom emotion information does not exist.

8. A data analysis device for a user group, characterized in that: include: An acquisition module, configured to acquire key frames in a video segment acquired by at least one data acquisition terminal at a current moment; Each of the data collection terminals is used to collect video information of users in different areas of the user group; A recognition module, configured to identify facial information and body key point information corresponding to the same user in the key frame; The facial information at least includes emotional information; a segmentation module, configured to perform a location segmentation operation on the user's location based on the user's facial information, so as to determine target users corresponding to the same location attribute; a matching module, configured to match the target user in the key frame at the current moment with the target user in the key frame at the previous moment based on the coordinates of the target user in the key frame at the current moment and the coordinates of the target user in the key frame at the previous moment, to obtain a user tracking result; The determination module is configured to determine the action information of the target user in the user tracking result based on at least one of the facial information and the body key point information of the target user at different moments in the user tracking result.

9. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so that the electronic device can implement the user group data analysis method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the user group data analysis method as described in any one of claims 1 to 7.

Images