Social activity level detection system and method

Abstract

A social activity level detection system is disclosed, which includes a data capturing device and a server. a server executes following actions: receiving a audio signal and an image from the data capturing device; performing speech-to-text processing and word count calculation on the audio signal to generate a speech word count, and generating a speech vector based on a word count threshold and the speech word count; performing face detection and person count calculation on the image to generate a field person count, and generating a personnel vector based on a person count threshold and the field person count; performing face recognition, emotion recognition, and summation scale adjustment on the image to generate a collective emotion vector; and performing element-wise addition on the speech vector, the personnel vector, and the collective emotion vector to generate a field emotion vector, and converting the field emotion vector into a social temperature value.

Description

[Technical Field]

[0001] This disclosure relates to a technique for monitoring social interactions, and more particularly to a social activity detection system and method. [Previous Technology]

[0002] While current technologies employ numerous methods to track and analyze the behavioral patterns, emotional changes, and social interactions of human groups, these technologies typically focus on measuring individual emotional or physiological responses (e.g., heart rate variability, skin conductivity, or semantic analysis), often failing to effectively combine and quantify this data with social activity. Particularly in the field of social interaction monitoring, there is no clear method to measure or quantify social activity, making it difficult for current monitoring systems to accurately reflect the intensity of social interaction or the emotional atmosphere within human groups. Therefore, how to accurately quantify the social activity of human groups in the same field is a problem that those skilled in the art urgently need to solve. [Summary of the Invention]

[0003] The main purpose of this disclosure is to provide a social activity detection system and method that can accurately quantify the social activity of human groups’ behavior patterns, emotional changes and social interactions.

[0004] To achieve the above objectives, this disclosure proposes a social activity detection system, comprising: a data acquisition device disposed in a field and configured to acquire an audio signal in the field and capture an image of the field; and a server connected to the data acquisition device and configured to perform the following actions: action a) receiving the audio signal and the image from the data acquisition device; action b) performing speech-to-text processing and word count calculation on the audio signal to generate a speech word count, and generating a speech vector based on a word count threshold and the speech word count; action c) performing face detection and people count calculation on the image to generate a field number of people among multiple people in the field, and generating a people vector based on a people threshold and the field number of people. Action d) Perform facial recognition and expression recognition on the image to generate individual emotion vectors for each of the multiple individuals in the field, and perform a summation and scaling adjustment on the multiple individual emotion vectors to generate an overall emotion vector; and action e) perform element-wise addition on the speech vector, the individual vector, and the overall emotion vector to generate a field emotion vector, and convert the field emotion vector into a social temperature value, which indicates the social activity of the multiple individuals in the field.

[0005] To achieve the above objectives, this disclosure proposes a social activity detection method, comprising: step a) receiving, via a server, a sound signal captured in a field and an image captured in the field from a data acquisition device; step b) using the server to perform speech-to-text processing and word count calculation on the sound signal to generate a speech word count, and generating a speech vector based on a word count threshold and the speech word count; step c) using the server to perform face detection and people count calculation on the image to generate a field number of people among multiple people in the field, and generating a people vector based on a people threshold and the field number of people. Step d) Using the server, perform facial recognition and expression recognition on the image to generate individual emotion vectors for each of the multiple people in the field, and perform summation and scaling adjustment on the multiple individual emotion vectors to generate an overall emotion vector; and Step e) Using the server, perform element-wise addition on the voice vector, the person vector, and the overall emotion vector to generate a field emotion vector, and convert the field emotion vector into a social temperature value, which indicates the social activity of the multiple people in the field.

[0006] Compared to previous technologies, this invention combines various speech algorithms and computer vision algorithms to quantify vectors corresponding to the number of people in a field, the number of words in the dialogue, and emotions from speech signals and images, and then converts the vectors into social temperature values. In this way, this invention can accurately quantify the social activity of a human group in the same field.

Implementation Method

[0007] Referring to FIG1A, FIG1A illustrates a block diagram of a social activity detection system 100 disclosed in some embodiments. As shown in FIG1A, in this embodiment, the social activity detection system 100 includes a data acquisition device 110 and a server 120. The server 120 is connected to the data acquisition device 110.

[0008] In some embodiments, the data acquisition device 110 can be implemented by any device with recording and photography functions (e.g., smartphone, digital camera, wide-angle camera, digital video camera, action camera, laptop computer, etc.). In this embodiment, the data acquisition device 110 is set in a field and used to capture sound signals in the field and capture images of the field. In other words, a user can use the data acquisition device 110 set in a specific field to simultaneously capture sound signals in the field and capture images of the field.

[0009] In some embodiments, the data acquisition device 110 can simultaneously acquire sound signals and capture images of the area during the detection period. In some embodiments, the detection period can be preset by the user (e.g., preset to 10 seconds). In some embodiments, the data acquisition device 110 can be placed at any location in the area that can capture the entire area. In some embodiments, the area can be any socially active place where people may be present (e.g., residence, conference room, supermarket, department store, social hall, or networking hall, etc.).

[0010] The data acquisition device 110 in the field will be described below with a practical example. Referring to FIG1B, FIG1B shows a schematic diagram of the field F1 disclosed in some embodiments. As shown in FIG1B, in this embodiment, the data acquisition device 110 may be a wide-angle camera with sound recording function, and is set in the field F1 at a position that can capture the entire field F1. Suppose that there are currently 4 people P1 to P4 having a meeting in the field F1, the data acquisition device 110 captures the sound signal corresponding to the voices of people P1 to P4, and captures the image of the field F1 where people P1 to P4 are present.

[0011] Referring also to FIG1C, FIG1C illustrates a schematic diagram of the field F1 disclosed in some other embodiments. As shown in FIG1C, in this embodiment, compared to the embodiment of FIG1B, the field F1 may be provided with multiple other data acquisition devices 1101~110N. The other data acquisition devices 1101~110N may be arranged at other locations in the field F1 that can capture the entire field F1, where N can be any positive integer. In this way, the data acquisition devices 110, 1101~110N can capture images of personnel P1~P4 from more shooting angles and record audio from personnel P1~P4 from more sound receiving angles. In other embodiments, some of the data acquisition devices (e.g., data acquisition devices 110, 1101-1103) may be devices that only have sound recording function (e.g., microphone array), while other data acquisition devices (e.g., data acquisition devices 1104-110N) may be devices that only have photography function (e.g., digital camera).

[0012] Based on this, performing various image and audio processing procedures as described in the following paragraphs on more images and more audio signals will greatly improve the accuracy and efficiency of these image and audio processing procedures. It is worth noting that, to more clearly explain how to implement the social activity detection method in the following paragraphs, the following paragraphs will continue to use a single data acquisition device 110 as an example. Furthermore, although this example uses four people P1 to P4, in reality, there may be more people in the field F1.

[0013] Returning to Figure 1A, in some embodiments, the data acquisition device 110 includes a sound receiving circuit 111 and a camera circuit 112. The sound receiving circuit 111 is used to detect sound in the field to generate a sound signal. The camera circuit 112 is used to capture images of the field. In some embodiments, the sound receiving circuit 111 may be implemented by an audio amplifier, a wireless audio receiver, a microphone array, or a combination thereof. In some embodiments, the camera circuit 112 may be implemented by a wide-angle camera circuit, a digital camera circuit, an optical camera circuit, a stereo camera circuit, a thermal imaging camera circuit, a radar imaging camera circuit, or a combination thereof.

[0014] In this embodiment, the server 120 receives audio signals and images from the data acquisition device 110, and uses the audio signals and images to perform the social activity detection method described in the following paragraphs (details to follow). In some embodiments, the server 120 can be implemented by any server with high data processing capabilities (e.g., cloud server, virtual server, or rack server).

[0015] Referring also to FIG2, FIG2 illustrates a flowchart of a social activity detection method disclosed in some embodiments, which is applicable to the social activity detection system 100 shown in FIG1A-FIG. As shown in FIG2, the social activity detection method includes steps S210-S250.

[0016] First, in step S210, the server 120 receives the sound signal captured in the field F1 and the image captured in the field F1 from the data acquisition device 110. In step S220, the server 120 performs speech-to-text processing and word count calculation on the sound signal to generate the number of speech words, and generates a speech vector based on the word count threshold and the number of speech words.

[0017] In some embodiments, speech-to-text (STT) processing can be implemented by any algorithm (e.g., the whisper-small model proposed by OpenAI, the hidden Markov model (HMM), the Gaussian mixture model (GMM), or the long short-term memory model (LSTM model)) used to accurately map language units (e.g., syllables or words) in speech to their corresponding text forms. In some embodiments, the word count threshold can be preset by the user or obtained statistically from multiple experiments (e.g., social activity is usually high when all participants in field F1 speak more than 15 words within 10 seconds, therefore, the word count threshold can be set to 15).

[0018] In some embodiments, the server 120 performs speech-to-text processing on the sound signal to generate multiple speech-text pairs, and counts the multiple speech-text pairs to generate a speech word count. Then, the server 120 compares the speech word count with a word count threshold, and generates a speech vector from the two-dimensional emotion model based on the comparison result. In some embodiments, when the comparison result indicates that the speech word count is greater than the word count threshold, the server 120 sets the high arousal vector on the arousal coordinate axis of the two-dimensional emotion model as the speech vector. Conversely, when the comparison result indicates that the speech word count is not greater than the word count threshold, the server 120 sets the low arousal vector on the arousal coordinate axis of the two-dimensional emotion model as the speech vector.

[0019] In some embodiments, the two-dimensional emotion model can be implemented by any two-dimensional emotion dimension model (e.g., Russell's circular model of emotion or Plutchik's wheel of emotions model, etc.). In some embodiments, the two-dimensional emotion model is a two-dimensional space established by a vertical arousal coordinate axis and a horizontal valence coordinate axis. In some embodiments, emotions corresponding to vectors in the two-dimensional emotion model that point more positively towards the valence coordinate axis have higher valence, while emotions corresponding to vectors in the two-dimensional emotion model that point more negatively towards the valence coordinate axis have lower valence. In some embodiments, emotions corresponding to vectors in the two-dimensional emotion model that point more positively towards the arousal coordinate axis have higher arousal, while emotions corresponding to vectors in the two-dimensional emotion model that point more negatively towards the arousal coordinate axis have lower arousal.

[0020] In some embodiments, the two-dimensional emotion model includes multiple feature vectors, each corresponding to a different emotion characteristic (e.g., happy, surprised, fear, angry, disgusted, sad, and neurotic). In some embodiments, the vectors in the two-dimensional emotion model have a vector angle relative to the negative direction of the pleasure coordinate axis, and this vector angle is proportional to the social temperature value of the vector (i.e., the social temperature value in Russell's theory). The multiple quadrants in the two-dimensional emotion model correspond to multiple temperature ranges (e.g., 0–10 degrees, 10–20 degrees, 20–30 degrees, and 30–40 degrees).

[0021] The following uses a practical example to illustrate the two-dimensional emotion model. Referring also to Figure 3, Figure 3 shows a schematic diagram of the two-dimensional emotion model 300 disclosed in some embodiments. As shown in Figure 3, the two-dimensional emotion model 300 is Russell's emotion ring model. The X-axis of the two-dimensional emotion model 300 is the pleasure coordinate axis, and the Y-axis is the arousal coordinate axis. The four quadrants in the two-dimensional emotion model 300 correspond to 0~10 degrees, 10~20 degrees, 20~30 degrees, and 30~40 degrees, respectively. The two-dimensional emotion model 300 includes multiple feature vectors V1~V7, which correspond to seven emotion features: happy, startled, afraid, angry, nauseous, sad, and neutral.

[0022] Furthermore, the starting point of feature vectors V1 to V7 is the origin, the coordinates of the endpoint of feature vector V1 are (0, 0), the coordinates of the endpoint of V2 are (1, 0), the coordinates of the endpoint of feature vector V3 are (0, 1), the coordinates of the endpoint of feature vector V4 are, the coordinates of the endpoint of feature vector V5 are, the coordinates of the endpoint of feature vector V6 are (-1, 0), and the coordinates of the endpoint of feature vector V7 are. The vector angle (i.e., the included angle) between eigenvector V1 and the -X-axis is 0 degrees, the vector angle between eigenvector V2 and the -X-axis is 180 degrees, the vector angle between eigenvector V3 and the -X-axis is 270 degrees, the vector angle between eigenvector V4 and the -X-axis is 300 degrees, the vector angle between eigenvector V5 and the -X-axis is 330 degrees, the vector angle between eigenvector V6 and the -X-axis is 360 degrees, and the vector angle between eigenvector V7 and the -X-axis is 30 degrees.

[0023] Since the vector angle is proportional to the social temperature value of the vector, with the -X axis as the reference, the social temperature value of feature vector V1 can be set to 0 degrees, the social temperature value of feature vector V2 can be set to 20 degrees, the social temperature value of feature vector V3 can be set to 30 degrees, the social temperature value of feature vector V4 can be set to 33.33 degrees, the social temperature value of feature vector V5 can be set to 36.67 degrees, the social temperature value of feature vector V6 can be set to 40 degrees, and the social temperature value of feature vector V7 can be set to 3.33 degrees.

[0024] It is worth noting that these emotional features and their corresponding emotional vectors may each correspond to multiple elements in the emotion recognition vector described in the paragraph (e.g., 7 emotional features correspond to 7 elements, and 7 corresponding feature vectors also correspond to 7 elements). In some embodiments, these emotional features and their corresponding emotional vectors may also be preset by the user or obtained statistically from multiple experiments. In other embodiments, in addition to the aforementioned emotional features and their corresponding emotional vectors, other emotional features (e.g., excitement) and other corresponding feature vectors (e.g., the feature vector corresponding to excitement starts at the origin and ends at the endpoint) may also be set in the two-dimensional emotion model 300.

[0025] On the other hand, when the comparison result indicates that the number of words in the speech is greater than the word count threshold, the server 120 can set the vector (i.e., the high arousal vector) on the Y-axis of the two-dimensional emotion model 300, with the origin as the starting point and the endpoint at (0, 1), as the speech vector. Conversely, when the comparison result indicates that the number of words in the speech is not greater than the word count threshold, the server 120 sets the vector (i.e., the low arousal vector) on the Y-axis of the two-dimensional emotion model 300, with the origin as the starting point and the endpoint at (0, -1), as the speech vector. In this way, the frequency of the dialogue of the people in the field F1 can be quantified into a vector for use in calculating the social temperature value of subsequent segments.

[0026] Returning to Figure 2, in step S230, the server 120 performs face detection and people calculation on the image to generate the number of people in the field F1, and generates a people vector based on the people threshold and the number of people in the field.

[0027] In some embodiments, face detection can be implemented by any algorithm used to identify face objects from an image (e.g., the YOLOv8n model, the faster region-based convolutional neural network (faster R-CNN) model, or the histogram of oriented gradients (HOG) algorithm, etc.). In some embodiments, the number of people threshold can also be preset by the user or obtained statistically from multiple experiments (e.g., social activity only increases when the number of people in field F1 is greater than 1 within 10 seconds, therefore, the number of people threshold can be set to 1).

[0028] In some embodiments, the server 120 performs face detection on the image to identify multiple face objects corresponding to multiple people in the image, and counts the multiple face objects to generate the number of people in the field. Next, the server 120 compares the number of people in the field with a number threshold, and generates a person vector from the two-dimensional emotion model 300 based on the comparison result. In some embodiments, when the comparison result indicates that the number of people in the field is greater than the number threshold, the server 120 sets a high arousal vector (e.g., a vector starting at the origin and ending at (0, 1)) on the arousal coordinate axis (i.e., the aforementioned Y-axis) in the two-dimensional emotion model 300 as the person vector. Conversely, when the comparison result indicates that the number of people in the field is not greater than the number threshold, the server 120 sets a low arousal vector (e.g., a vector starting at the origin and ending at (0, -1)) on the arousal coordinate axis in the two-dimensional emotion model 300 as the person vector. In this way, the number of people in field F1 can be quantified into a vector for use in calculating the social temperature value of subsequent paragraphs.

[0029] In step S240, the server 120 performs face recognition and expression recognition on the image to generate individual emotion vectors of multiple people in the field, and performs summation and scale adjustment on the multiple individual emotion vectors to generate an overall emotion vector.

[0030] In some embodiments, face recognition can be implemented by any algorithm for face-body recognition (e.g., Dlib model, FaceNet model, or ArcFace model, etc.). In some embodiments, expression recognition can be implemented by any algorithm for emotion recognition (e.g., VGG-face model, Deep Residual Network (ResNet) model, or Multi-Task Learning (MLT) model, etc.). In some embodiments, the sum scaling includes element-wise addition and scaling based on the vector dimension (e.g., performing element-wise addition on multiple vectors of dimension 7 to produce a vector sum, and then dividing all 7 elements of the vector sum by 7).

[0031] In some embodiments, the server 120 performs facial recognition on the images to generate identity information for multiple individuals in the field F1. Next, the server 120 performs facial expression recognition on the images to generate emotion recognition vectors corresponding to each identity information. Then, the server 120 converts the emotion recognition vectors corresponding to each identity information into individual emotion vectors for each individual in the two-dimensional emotion model 300.

[0032] In some embodiments, the server 120 pre-stores multiple facial feature sets and candidate identity information corresponding to each facial feature set. Then, it performs facial recognition on the image based on these facial features and candidate identity information to generate identity information for multiple individuals in the field F1 and corresponding facial regions in the image. In this way, the server 120 can determine which individuals are currently engaging in social activities in the field F1 and whether there are any unrecorded individuals engaging in social activities in the field F1.

[0033] Next, the server 120 further performs expression recognition on these facial regions to generate emotion recognition vectors corresponding to each identity information. Multiple elements in each emotion recognition vector correspond to multiple emotion features, and these emotion features correspond to multiple feature vectors in the two-dimensional emotion model 300 (e.g., an emotion recognition vector has 7 elements, each corresponding to one of the 7 emotion features in Figure 3, which in turn correspond to one of the 7 feature vectors in the two-dimensional emotion model 300 of Figure 3). In some embodiments, the value of each element in an emotion recognition vector indicates the probability that the expression of the person corresponding to that emotion recognition vector belongs to the emotion feature corresponding to that element (e.g., when the value of the element corresponding to happiness in an emotion recognition vector is 0.5, the probability that the expression of the person corresponding to that emotion recognition vector is happy is 0.5).

[0034] In some embodiments, the server 120 performs a weighted average of multiple feature vectors based on multiple elements in the emotion recognition vector corresponding to each identity information to generate a personal emotion vector for each person corresponding to each identity information. In some embodiments, the server 120 uses multiple elements in the emotion recognition vector corresponding to each identity information as multiple weights to perform a weighted average with multiple feature vectors, and uses the calculation result as the personal emotion vector for each person corresponding to each identity information.

[0035] For example, based on the example in Figure 3, server 120 can use the 7 elements of the emotion vector of one person corresponding to one of the identity information as 7 weights corresponding to 7 feature vectors in the two-dimensional emotion model 300, and multiply these 7 weights by the 7 feature vectors to generate 7 product vectors. Then, server 120 can perform element-wise addition on these 7 product vectors to generate a vector sum, and divide the vector sum by 7 to generate the personal emotion vector of the person corresponding to one of the identity information. In this way, the expressions of all people in field F1 can be integrated and quantized into a vector for use in calculating the social temperature value of subsequent paragraphs.

[0036] In step S250, the server 120 performs element-wise addition on the voice vector, the personnel vector, and the overall emotion vector to generate a field emotion vector, and converts the field emotion vector into a social temperature value indicating the social activity of multiple personnel in field F1. In some embodiments, when the social temperature value is higher, there will be more personnel in field F1 or the emotions of multiple personnel in field F1 will be more active. Conversely, when the social temperature value is lower, there will be fewer personnel in field F1 or the emotions of multiple personnel in field F1 will be more subdued. In some embodiments, the social temperature value has a preset temperature range (for example, the temperature range of the social temperature value may be the same as the overall temperature range of the two-dimensional emotion model 300 (i.e., both are 0~40 degrees)).

[0037] In other embodiments, the server 120 can further multiply the three preset weights with the voice vector, the personnel vector, and the overall emotion vector to generate a weighted voice vector, a weighted personnel vector, and a weighted overall emotion vector, and then perform element-wise addition on the weighted voice vector, the weighted personnel vector, and the weighted overall emotion vector to generate a field emotion vector. In some embodiments, these preset weights may also be pre-set by the user or obtained statistically from multiple experiments.

[0038] In some embodiments, the server 120 generates a vector angle based on the negative direction of the pleasure coordinate axis in the two-dimensional emotion model 300 and the field emotion vector (i.e., the angle between the field emotion vector and the negative direction of the pleasure coordinate axis is taken as the vector angle). Then, the server 120 determines which quadrant of the multiple quadrants in the two-dimensional emotion model 300 the vector angle is located in, and calculates the social temperature value based on the determination result and the vector angle.

[0039] In some embodiments, the server 120 calculates the angle difference between the vector angle and the starting angle of the quadrant indicated by the judgment result (for example, the starting vectors of the first to fourth quadrants in the two-dimensional emotion model 300 are on the X-axis, Y-axis, -X-axis, and -Y-axis, respectively, and the angles between the X-axis, Y-axis, -X-axis, and -Y-axis and -X-axis are 180, 270, 0, and 90 degrees, respectively, which can be used as the starting angles of the first to fourth quadrants), and calculates the social temperature value based on the number corresponding to the quadrant indicated by the judgment result (for example, the first to fourth quadrants correspond to 0, 10, 20, and 30, respectively) and the angle difference. In some embodiments, the social temperature value is represented by the following formula (1). ... Formula (1)

[0040] As shown in formula (1), T is the social temperature value, D is the vector angle, and the angle subtracted from the vector angle is the starting angle of the quadrant in which the vector angle is located. The number added to the product in formula (1) (for example, ) is the number corresponding to the quadrant in which the vector angle is located. Specifically, when the vector angle is in the first quadrant, the angle difference is , the number corresponding to the first quadrant is 0, and the social temperature value is equal to . When the vector angle is in the second quadrant, the angle difference is , the number corresponding to the second quadrant is 10, and the social temperature value is equal to . When the vector angle is in the third quadrant, the angle difference is , the number corresponding to the third quadrant is 20, and the social temperature value is equal to . When the vector angle is in the fourth quadrant, the angle difference is , the number corresponding to the fourth quadrant is 30, and the social temperature value is equal to .

[0041] In some embodiments, when the social temperature value is greater than a temperature threshold, the server 120 can obtain emotional features corresponding to all elements greater than the emotional threshold from each person's personal emotional vector, and display (e.g., on a display device (not shown) or a monitoring device (not shown) of the server 120) these emotional features corresponding to all elements greater than the emotional threshold. For example, when the display device or monitoring device is a laptop computer set in the field F1, the display device or monitoring device can receive these emotional features from the server 120 for display. In this way, the user can observe what emotions the people in the field F1 will have under conditions of high social activity. In some embodiments, the temperature threshold and the emotional threshold may also be preset by the user or obtained statistically from multiple experiments.

[0042] Through the above steps, this disclosure can quantify the number of people in field F1, the frequency of their conversations, and their emotions into multiple vectors, and then convert these vectors into social temperature values. Therefore, this disclosure can use social temperature values ​​to quantify the number of people in field F1 and the degree of their emotional activity, allowing users to understand the current social activity level in field F1 and to conduct research in the field of social interaction monitoring.

[0043] In summary, the social activity detection system and method proposed in this disclosure combine various speech algorithms and computer vision algorithms to quantify vectors corresponding to the number of people, the number of words in the dialogue, and emotions in the field from speech signals and images, and then convert the vectors into social temperature values. Therefore, the social activity detection system and method proposed in this disclosure can determine the current social activity level of the field from the social temperature values. Furthermore, the social activity detection system and method proposed in this disclosure utilize a two-dimensional emotion model commonly used in social research to convert the vectors into temperature values ​​to obtain the required social temperature values. In this way, this disclosure can use social temperature values ​​to allow users to know the current social activity level in field F1 and to conduct research in the field of social interaction monitoring. In addition, the social activity detection system and method proposed in this disclosure allow users to further observe the results of facial expression recognition when the social temperature value is high, in order to know what emotions the people in the field are experiencing when the social temperature value is high.

[0044] Although this disclosure has been disclosed above with reference to embodiments, it is not intended to limit this disclosure. Anyone skilled in the art may make some modifications and refinements without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0045] Figure 1A illustrates a block diagram of a social activity detection system in some embodiments.

[0046] Figure 1B illustrates a schematic diagram of the field disclosed in some embodiments.

[0047] FIG1C illustrates a schematic diagram of the field disclosed in some other embodiments.

[0048] Figure 2 illustrates a flowchart of a social activity detection method disclosed in some embodiments.

[0049] Figure 3 illustrates a schematic diagram of a two-dimensional emotion model in some embodiments.

Claims

1. A social activity detection system, comprising: A data acquisition device, disposed in a field, configured to acquire an audio signal and capture an image of the field; and a server, connected to the data acquisition device, configured to perform the following actions: Action a) receiving the audio signal and the image from the data acquisition device; Action b) performing speech-to-text processing and word count calculation on the audio signal to generate a speech word count, and generating a speech vector based on a word count threshold and the speech word count, wherein Action b) includes: Action b1) performing speech-to-text processing on the audio signal to generate multiple speech characters, and counting the multiple speech characters to generate the speech word count; and Action b2) comparing the speech word count with the word count threshold, and generating the speech vector from a two-dimensional emotion model based on a comparison result; Action c) performing face detection and people calculation on the image to generate a field population of multiple people in the field, and generating a people vector based on a people threshold and the field population. Action d) Perform facial recognition and expression recognition on the image to generate individual emotion vectors for each of the multiple individuals in the field, and perform a summation and scaling adjustment on the multiple individual emotion vectors to generate an overall emotion vector; and action e) perform element-wise addition on the speech vector, the individual vector, and the overall emotion vector to generate a field emotion vector, and convert the field emotion vector into a social temperature value, which indicates the social activity of the multiple individuals in the field.

2. The social activity detection system as claimed in claim 1, wherein in action b2), the server is configured to perform the following actions: when the comparison result indicates that the number of words in the speech is greater than the number of words threshold, setting a high arousal vector on an arousal coordinate axis in the two-dimensional emotion model as the speech vector; and when the comparison result indicates that the number of words in the speech is not greater than the number of words threshold, setting a low arousal vector on the arousal coordinate axis in the two-dimensional emotion model as the speech vector.

3. The social activity detection system as claimed in claim 1, wherein in action c), the server is configured to perform the following actions: action c2) performing face detection on the image to identify multiple face objects in the image corresponding to the multiple people, and counting the multiple face objects to generate the number of people in the field; and action c2) comparing the number of people in the field with the number threshold, and generating the person vector from a two-dimensional emotion model based on a comparison result.

4. The social activity detection system as described in claim 3, wherein in action c2), the server is configured to perform the following actions: when the comparison result indicates that the number of people in the field is greater than the number threshold, setting a high arousal vector on an arousal coordinate axis in the two-dimensional emotion model as the person vector; and when the comparison result indicates that the number of people in the field is not greater than the number threshold, setting a low arousal vector on the arousal coordinate axis in the two-dimensional emotion model as the person vector.

5. The social activity detection system as claimed in claim 1, wherein in action d), the server is configured to perform the following actions: action d1) performing facial recognition on the image to generate identity information for each of the plurality of persons in the field; action d2) performing facial expression recognition on the image to generate an emotion recognition vector corresponding to each of the identity information; and action d3) converting the emotion recognition vector corresponding to each of the identity information into a two-dimensional emotion model for the individual emotion vector corresponding to each of the identity information.

6. The social activity detection system as claimed in claim 5, wherein multiple elements in the emotion recognition vector corresponding to each of the identity information correspond to multiple emotion features, and the multiple emotion features correspond to multiple feature vectors in the two-dimensional emotion model, wherein in action d3), the server is configured to perform the following action: weighted average the multiple feature vectors according to the multiple elements in the emotion recognition vector corresponding to each of the identity information to generate the personal emotion vector of the person corresponding to each of the identity information.

7. The social activity detection system as claimed in claim 1, wherein in action e), the server is configured to perform the following actions: action e1) generating a vector angle based on a negative direction of a pleasure coordinate axis in a two-dimensional emotion model according to the field emotion vector; and action e2) determining which quadrant of the multiple quadrants in the two-dimensional emotion model the vector angle is located in, and calculating the social temperature value based on the determination result and the vector angle.

8. The social activity detection system as claimed in claim 7, wherein in action e2), the server is configured to perform the following actions: calculate an angle difference between the vector angle and a starting angle of the quadrant indicated by the judgment result, and calculate the social temperature value based on a number corresponding to the quadrant indicated by the judgment result and the angle difference.

9. A method for detecting social activity, comprising: Step a) Using a server, a sound signal captured in a field and an image captured in the field are received from a data acquisition device; Step b) Using the server, the sound signal is processed for speech-to-text conversion and word count is calculated to generate a speech word count, and a speech vector is generated based on a word count threshold and the speech word count, wherein step b) includes: Step b1) The sound signal is processed for speech-to-text conversion to generate multiple speech characters, and the multiple speech characters are counted to generate the speech word count; and Step b2) The speech word count is compared with the word count threshold, and the speech vector is generated from a two-dimensional emotion model based on a comparison result; Step c) Using the server, face detection and people counting are performed on the image to generate a field number of people for multiple people in the field, and a people vector is generated based on a people threshold and the field number of people. Step d) Using the server, perform facial recognition and expression recognition on the image to generate individual emotion vectors for each of the multiple people in the field, and perform summation and scaling adjustment on the multiple individual emotion vectors to generate an overall emotion vector; and Step e) Using the server, perform element-wise addition on the voice vector, the person vector, and the overall emotion vector to generate a field emotion vector, and convert the field emotion vector into a social temperature value, which indicates the social activity of the multiple people in the field.

Images