Artificial intelligence-based ip network visual intercom emergency emotion recognition method and system

Abstract

The application discloses an IP network visual intercom emergency emotion recognition method and system based on artificial intelligence, and comprises the following steps: when a user triggers an alarm instruction of an IP network visual intercom emergency device, starting a camera to collect video data and determining a current face image, then constructing an emotion feature correlation matrix of the face image, and determining a corresponding emotion type according to the matrix. According to the method, the emotion of the user in an emergency situation can be more accurately recognized by constructing the correlation matrix, thereby providing strong support for taking targeted emergency measures subsequently.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to an IP network visual intercom emergency emotion recognition method and system based on artificial intelligence. BACKGROUND

[0002] In an emergency scenario, timely and accurate identification of user emotions is crucial for taking appropriate emergency measures. Traditional emergency handling methods often lack precise judgment of user emotions, resulting in insufficient pertinence and effectiveness of emergency response. SUMMARY

[0003] The present application relates to the field of artificial intelligence, in particular to an IP network visual intercom emergency emotion recognition method and system based on artificial intelligence.

[0004] In a first aspect, the present application provides an IP network visual intercom emergency emotion recognition method based on artificial intelligence, comprising:

[0005] In response to the alarm instruction of the IP network visual intercom emergency device triggered by the current user, starting the camera of the IP network visual intercom emergency device;

[0006] Collecting current video data through the camera and determining the current face image from the video data;

[0007] Building an emotion feature correlation matrix of the current face image, and determining the emotion type corresponding to the current face image according to the emotion feature correlation matrix.

[0008] In a second aspect, the present application provides a server system, comprising a server, which is used to execute the method of the first aspect.

[0009] Compared with the prior art, the present application provides the following beneficial effects: by using the IP network visual intercom emergency emotion recognition method and system based on artificial intelligence disclosed in the present application, after the user triggers the alarm instruction of the IP network visual intercom emergency device, the camera is started to collect video data and determine the current face image, then the emotion feature correlation matrix of the face image is built, and the corresponding emotion type is determined according to the matrix. This method can more accurately identify the user's emotions in emergency situations by building a correlation matrix, providing strong support for subsequent targeted emergency measures. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those of ordinary skill in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0011] Figure 1 The step flowchart of the IP network visual intercom emergency emotion recognition method based on artificial intelligence provided by the embodiments of the present application is shown in the figure.

[0012] Figure 2 The structural schematic block diagram of the computer device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0014] The specific embodiments of the present application will be described in detail in combination with the drawings.

[0015] In order to solve the technical problems in the foregoing background art, Figure 1 The flowchart of the IP network visual intercom emergency emotion recognition method based on artificial intelligence provided by the embodiments of the present application is shown in the figure, and the IP network visual intercom emergency emotion recognition method based on artificial intelligence will be described in detail.

[0016] Step S201, in response to the alarm instruction of the IP network visual intercom emergency device triggered by the current user, starting the camera of the IP network visual intercom emergency device;

[0017] Step S202, collecting current video data through the camera, and determining the current face image from the video data;

[0018] Step S203, constructing the emotion feature correlation matrix of the current face image, and determining the emotion type corresponding to the current face image according to the emotion feature correlation matrix.

[0019] In an embodiment of the present application, for example, in a certain cell of a city, a gas leakage accident suddenly occurs in a resident's home. After the resident (i.e. the current user) senses the danger, he quickly runs to the IP network video intercom emergency device installed at the entrance and presses the alarm button on the device, triggering an alarm instruction. This alarm instruction is quickly transmitted to the server through the network within the cell. After receiving the alarm instruction, the server immediately sends an instruction to start the camera to the IP network video intercom emergency device. After receiving the instruction from the server, the IP network video intercom emergency device quickly starts its built-in camera. At this time, the camera begins to prepare to collect relevant video data to provide a basis for subsequent analysis and processing. After the camera is started, it begins to collect current video data. In the scenario of gas leakage, the camera captures the resident standing in front of the emergency device with an anxious and slightly fearful expression. These video data are transmitted to the server in the form of digital signals through the network. After receiving the video data, the server analyzes and processes the video data using advanced image recognition algorithms. The server first scans each frame of image in the video data to detect whether there is a face in the image through face recognition technology. In this process, the server determines whether it is a face according to the features of the face, such as the positions and shapes of the eyes, nose, and mouth. When the server accurately identifies a face in a certain frame of image, it extracts the image segment containing the face and determines it as the current face image. For example, the server identifies the resident's face from the 10th to 20th frames of images in the video data, so it determines the image composed of these 10 frames as the current face image. After obtaining the above-determined current face image, the server extracts multiple images from the image. For example, it selects the 12th and 18th frames of images from different frames of the current face image. Then, the server divides these two images into two image sets according to the features of the images, such as the subtle differences in facial expressions and light conditions. Suppose the resident's eyes are slightly wide open in the 12th frame of image, which is divided into an image set representing the "slightly nervous" feature; the resident's furrowed brows in the 18th frame of image are divided into an image set representing the "more nervous" feature. The server selects representative target images from the above two image sets, such as selecting the 18th frame of image as the target image from the image set representing the "more nervous" feature. Then, the server uses deep learning algorithms to enhance the image feature extraction of the target image. The server convolves the target image through a convolutional neural network (CNN) to extract the edge, texture, and other features in the image, obtaining the first image feature. For example, in the 18th frame of image, the CNN detects the texture feature of the resident's furrowed brows and the feature of the slightly congested eyes, which are extracted as the first image feature. The server assigns a first emotion label to the current face image according to the pre-set emotion classification standard and the overall performance of the current face image.In the scenario of gas leakage, the server assigns the first emotion label of "nervous fear" to the current face image based on the features such as furrowed brows and anxious expressions. Then, the server extracts semantic features from the first emotion label. Through natural language processing (NLP) technology, the server decomposes the emotion label "nervous fear" into semantic units such as "nervous" and "fear", and further analyzes the relevant features of each semantic unit. For example, "nervous" may be related to physiological reactions such as increased heart rate and muscle tension, and "fear" may be related to psychological reactions such as perception of danger and avoidance tendency. These analyzed features form the first semantic features. The server fuses the first image features and the first semantic features to obtain the first multi-dimensional features. This first multi-dimensional feature contains multiple dimensional feature information, for example, one dimension corresponds to the facial expression feature in the image, such as the degree of furrowed brows; another dimension corresponds to the psychological reaction in the semantic feature, such as the degree of fear. Assuming that in this example, the first multi-dimensional feature contains multiple dimensional features such as "the degree of furrowed brows is 8 (full score 10)" in the facial expression dimension and "the degree of fear is 7 (full score 10)" in the psychological reaction dimension. The server obtains the pre-set enhanced image feature matching pool and semantic feature matching pool. The enhanced image feature matching pool stores a large number of correspondence relationships between pending face images and their corresponding enhanced image features, and the semantic feature matching pool stores a large number of correspondence relationships between pending face images and their corresponding semantic features. The server matches the target enhanced image feature in the first multi-dimensional feature and the first semantic feature with the two matching pools respectively. For example, the server compares the target enhanced image feature (such as the texture feature of furrowed brows) in the first multi-dimensional feature with the enhanced image features of each pending face image in the enhanced image feature matching pool, and compares the first semantic feature (such as the semantic feature related to "nervous fear") with the semantic features of each pending face image in the semantic feature matching pool. Through comparison, the server finds at least two target pending face images close to the current face image. Assuming that the two target pending face images are found from the images of the parties in the previous recorded other gas leakage events and fire events. Then, the server determines the reference face image from the target pending face images based on the image matching coefficients between the at least two target pending face images and the current face image. Specifically, for each target pending face image matched by each matching strategy, the server confirms the matching coefficient between the current face image and each target pending face image. For example, for the matching strategy based on facial expression features, the server calculates the matching coefficient between the current face image and the first target pending face image as 0.85, and the matching coefficient between the current face image and the second target pending face image as 0.82. Then, for each target pending face image, the server confirms the average matching coefficient between the target pending face image and the current face image in each matching strategy.Assuming that the average matching coefficient of the first target face image under multiple matching strategies is 0.84, and the average matching coefficient of the second target face image is 0.80. Finally, the server determines the reference face image from the target face image according to the average matching coefficient. In this example, the first target face image with a higher average matching coefficient is determined as the reference face image. Perform feature integration operation on at least two first image features to obtain target enhanced image features: the server performs feature integration operation on at least two first image features extracted from the current face image. For example, the server fuses the first image feature representing "frowning" and the first image feature representing "slightly bloodshot eyes" to obtain a comprehensive target enhanced image feature that more comprehensively reflects the characteristics of the current face image. Synchronize the target enhanced image features to the preset semantic feature domain through the pre-trained feature synchronization network to obtain the first image semantic identifier set, and perform feature mapping on the second image features corresponding to each reference face image to obtain the second image semantic identifier set corresponding to each reference face image: the server uses the pre-trained feature synchronization network to synchronize the target enhanced image features to the preset semantic feature domain. For example, the "frowning" feature in the target enhanced image feature is mapped to the "nervous" semantic identifier in the semantic feature domain, and the "slightly bloodshot eyes" feature is mapped to the "excited" semantic identifier, thereby obtaining the first image semantic identifier set. At the same time, the server performs similar feature mapping operation on the second image features corresponding to the reference face image to obtain the second image semantic identifier set corresponding to the reference face image. Obtain the collection location of the current face image and the voice content recognized by the voice data of the current face image: the server obtains the collection location information of the current face image from the IP network visual intercom emergency device. In this example, the collection location is a certain unit in a certain building in a community. At the same time, the server identifies the voice content of the resident as "quickly help me, there is a gas leak at home" by recognizing the voice data attached to the current face image. Generate emergency scene information of the current face image according to the background image of the current face image: the server analyzes the background image of the current face image, such as seeing smoke and other gas leakage in the background, and generates the emergency scene information of the current face image by combining related knowledge and algorithms, and determines that it is an emergency scene of gas leakage. Generate the first emotion identifier according to the collection location, the voice content and the emergency scene information: the server integrates the collection location, the voice content and the emergency scene information to further confirm and refine the first emotion identifier. In this example, according to the scene of gas leakage at home and the voice content of the resident, the first emotion identifier is again confirmed as "nervous and fearful".According to the first semantic identification for distinguishing different information in the same person's face image, and the second image semantic identification set, the second emotion identification and the emotion target value corresponding to the same reference face image, an emotion feature correlation parameter is established to obtain at least two emotion feature correlation parameters corresponding to at least two reference face images: the server establishes an emotion feature correlation parameter according to the first semantic identification (such as the identification of distinguishing different expression features in the current face image), the second image semantic identification set of the reference face image, the second emotion identification (the emotion identification corresponding to the reference face image) and the emotion target value (the emotion quantization value preset for the reference face image). For example, for the first reference face image, its second image semantic identification set contains "nervousness", "worry" and other identifications, the second emotion identification is "high nervousness and fear", and the emotion target value is 9 (full score 10). The server establishes an emotion feature correlation parameter according to these information and the first semantic identification of the current face image. The at least two emotion feature correlation parameters are combined to obtain an initial emotion correlation matrix, wherein at least two emotion feature correlation parameters in the initial emotion correlation matrix are separated by the second semantic identification for distinguishing information of different face images: the server combines the emotion feature correlation parameters corresponding to multiple reference face images to form an initial emotion correlation matrix. In the merging process, the second semantic identification is used to distinguish the information of different face images, ensuring that the data in the matrix is clear and orderly. According to the first semantic identification, the first image semantic identification set and the first emotion identification, an emotion classification identification is established: the server establishes an emotion classification identification according to the first semantic identification, the first image semantic identification set and the first emotion identification. This emotion classification identification is used to classify and identify the emotion of the current face image, for example, the established emotion classification identification is "nervous and fearful emotion in gas leakage scene". According to the initial emotion correlation matrix and the emotion classification identification, the emotion feature correlation matrix is generated: the server fuses the initial emotion correlation matrix and the emotion classification identification to generate the final emotion feature correlation matrix. This matrix comprehensively reflects the emotion feature correlation relationship between the current face image and the reference face image. The server performs emotion element feature conversion on the emotion feature correlation matrix. For example, the server standardizes various emotion feature parameters in the matrix according to the requirements of the target value recognition model, converts feature values of different dimensions into a unified format and range, and obtains an emotion feature set. The data in this emotion feature set meets the input requirements of the target value recognition model, so as to perform accurate emotion type recognition subsequently. The server inputs the emotion feature set into the target value recognition model. The target value recognition model is obtained through careful training. It optimizes the model parameters by training the emotion feature set on the basis of keeping the basic model parameters of the pre-set multi-modal model unchanged, and adds an attention mechanism to enhance the attention to key features.For example, in this gas leakage scene, the attention mechanism pays more attention to features related to the "tense fear" emotion, such as the tension of facial expressions, the urgency of the tone in the voice, etc. The target value recognition model analyzes and processes the input set of emotional features to generate the target emotion target value of the current face image. In this example, the server obtains the target emotion target value of the current face image from the target value recognition model as 8 (full score 10), indicating that the resident's emotion is at a high level of tense fear. The server determines the emotion type corresponding to the current face image according to the size of the target emotion target value and the pre-set emotion type division standard. For example, when the target emotion target value is between 7-10, the corresponding emotion type is "high tense fear". Therefore, the server determines that the current resident's emotion type is "high tense fear". In this way, the server successfully identifies the emotion type of the current user in the emergency situation, providing an important basis for subsequent adoption of appropriate emergency measures.

[0020] In the embodiments of the present application, the following implementation modes are also provided.

[0021] In the case where the emotion type belongs to a first-level emotion type, the intercom function of the IP network visual intercom emergency device is started to realize manual real-time reply interaction;

[0022] In the case where the emotion type belongs to a second-level emotion type, the current voice data from the current user is loaded into the first generative dialogue model of the integrated model to obtain the basic target of the current user; the emergency degree of the second-level emotion type is greater than that of the first-level emotion type in the emergency scene;

[0023] The past voice content of the full user in the preset period range and the basic target are loaded into the second generative dialogue model of the integrated model to obtain the final target of the current user;

[0024] The final target is adapted to a plurality of preset emergency strategies in a preset emergency strategy database to obtain a target emergency strategy suitable for the current user, and the target emergency strategy is provided to the current user.

[0025] In an embodiment of the present application, it is assumed that a customer in a large shopping mall accidentally collides with someone during shopping, and his mood is slightly excited, but not very urgent. At this time, the server determines that the customer's emotion type belongs to the first level emotion type through the previous emotion recognition method. The server immediately sends an instruction to the IP network visual intercom emergency device to turn on its intercom function after identifying the emotion type. The IP network visual intercom emergency device receives the instruction and establishes a communication connection with the customer service center of the shopping mall. The staff of the customer service center can immediately interact with the customer in real time through the intercom device. For example, the customer says through the emergency device: "I was just hit by someone, and I feel a little uncomfortable." The customer service staff responds through the intercom device: "I'm very sorry to hear that you have such a situation. Are you injured now?" Through this real-time interaction, the customer service staff can further understand the customer's specific situation and calm the customer's emotions to avoid further escalation. If the customer subsequently raises some reasonable demands, such as finding the person who collided with him to negotiate a solution, the customer service staff can also assist in handling it in a timely manner. In a factory workshop, a worker operating a machine suddenly discovers that the machine has a serious malfunction, which poses a significant safety hazard, making him extremely panicked and anxious. The server determines that the worker's emotion type belongs to the second level emotion type. At this time, the worker sends the current voice data to the server through the IP network visual intercom emergency device, and he says: "The machine suddenly lost control, what should I do?" The server receives this voice data and loads it into the first generative dialogue model of the integrated model. The first generative dialogue model analyzes and understands the voice data. It first converts the voice data into text form, and then analyzes the semantic information in the text through deep learning algorithms and a large amount of training data. In this example, the model identifies the key information described by the worker as "machine failure" and "seeking a solution", so it determines that the worker's basic goal is to solve the current machine failure problem and ensure production safety. After obtaining the worker's basic goal, the server retrieves all user past voice content in the pre-set period (e.g., the past month) from the database. These past voice content may include descriptions of other workers when they encounter similar machine failures, solutions, and their expectations, etc. Assuming that some of the past voice content contains cases of the same type of machine failure, some workers mention the need to repair the machine as soon as possible to restore production, and some emphasize the need to ensure personnel safety first. The server loads these past voice content and the worker's basic goal into the second generative dialogue model of the integrated model. The second generative dialogue model analyzes these information comprehensively, combines the current actual situation, and further explores the worker's potential needs and final expectations.For example, after model analysis, it is found that the production task of the current worker is relatively urgent, and the malfunctioning machine has a greater impact on the entire production process, so the final goal of the worker is determined to not only solve the machine malfunction, but also to restore production as soon as possible in the shortest time, while ensuring the safety of himself and other colleagues. After determining the final goal of the worker, the server will adapt it to multiple preset emergency strategies in the preset emergency strategy database. The preset emergency strategy database stores various coping strategies for different types of emergency situations, including machine failure, fire, personnel injury, etc. For this type of emergency situation of machine failure, there may be multiple emergency strategies in the database, such as immediately stopping the machine from running, notifying professional maintenance personnel, starting backup equipment, etc. The server matches the final goal of the worker with these preset emergency strategies one by one through a matching algorithm. In this example, according to the final goal of the worker who wants to restore production as soon as possible and ensure safety, the server adapts the following target emergency strategy from the preset emergency strategy database: first, immediately stop the operation of the malfunctioning machine to avoid further expansion of the danger; at the same time, notify professional maintenance personnel to quickly arrive on site with relevant tools and spare parts for emergency repair; during the repair process, arrange other workers to do a good job of on-site safety protection to prevent accidents; in addition, start the backup equipment to temporarily maintain normal production. After determining the target emergency strategy, the server sends it to the worker through the IP network visual intercom emergency equipment. After receiving the target emergency strategy, the worker operates according to the guidance of the strategy, thereby effectively dealing with the current emergency situation and minimizing losses. Through the above steps, the server can provide targeted emergency strategies according to the emotional type and specific needs of the user, improving the efficiency and effectiveness of emergency handling.

[0026] In the embodiment of the present application, the emotion feature correlation matrix of the current face image is constructed, and the emotion type corresponding to the current face image is determined according to the emotion feature correlation matrix. The following example can be used for implementation.

[0027] The current face image is obtained, and multi-dimensional feature extraction is performed on the current face image to obtain first multi-dimensional features. The first multi-dimensional features include multiple dimension features corresponding to multiple dimensions, respectively.

[0028] At least two reference face images close to the current face image are matched according to the first multi-dimensional features. Each reference face image is pre-provisioned with an emotion target value.

[0029] The emotion feature correlation matrix is established according to the second multi-dimensional features corresponding to each reference face image, the pre-provisioned emotion target value of each reference face image, and the first multi-dimensional features of the current face image.

[0030] According to the emotion feature correlation matrix, an emotion target value of the current face image is recognized.

[0031] According to the recognition result of the emotion target value, an emotion type corresponding to the current face image is determined.

[0032] In an embodiment of the present application, an exemplary IP network visual intercom emergency device is installed in a school building in a campus environment. One day, a student suddenly feels unwell in the classroom and triggers an alarm instruction in front of the emergency device. After receiving the alarm instruction, the server starts the camera of the emergency device and collects video data containing the student's face, and then determines the current face image. The server starts to extract multi-dimensional features from the current face image. First, the image dimension is extracted. The server uses advanced computer vision algorithms to analyze the features of each part of the face in the current face image. For example, it is observed that the student's eyebrows are slightly furrowed, the eyes are slightly dull, and the lips are slightly pale. Through the analysis of these facial details, the server extracts features related to facial expressions, such as the furrowing degree of the eyebrows quantified as 3 (full score 10), the dullness of the eyes quantified as 4 (full score 10), etc. At the same time, the server also extracts features from the speech dimension. The emergency device may also synchronously collect the student's weak voice when collecting video data, such as the student's hoarse voice and weak tone. The server converts the voice into text through speech recognition technology and further analyzes the features such as tone, speed, and volume in the voice. For example, the student's speech speed is slow, the volume is small, and the tone is flat. The server quantifies these speech features, such as speech speed quantified as 2 (full score 10), volume quantified as 3 (full score 10), etc., as features of the speech dimension. In addition, the server also considers the features of the behavior dimension. From the current face image, the student's body posture and action are analyzed, and it is found that the student stands unsteadily and the body slightly shakes. The server quantifies these behavior features, such as the shaking degree of the body quantified as 5 (full score 10), etc. By integrating the features of multiple dimensions such as image, speech, and behavior, the server obtains the first multi-dimensional feature. This first multi-dimensional feature contains multiple dimension features corresponding to multiple dimensions, and comprehensively reflects the current student's state at the emergency moment. The server obtains a pre-established baseline face image library, which stores a large number of face images in different scenes and their corresponding multi-dimensional features and pre-set emotion target values. The server matches and searches in the baseline face image library according to the features of each dimension in the first multi-dimensional feature. For example, the server first searches for face images with similar features such as eyebrow furrowing degree and eye dullness in the baseline face image library according to the features of the facial expression dimension. After finding some tentative baseline face images that meet the conditions, further screening is performed in combination with the features of the speech dimension and the behavior dimension. Assume that there are two baseline face images in the baseline face image library that are close to the first multi-dimensional feature of the current student.One is a video of another student who was sick before, and its facial expressions, voice, and behavioral characteristics have a high degree of similarity with the current student. The pre-set emotional target value is 7 (out of 10), indicating a state of weakness and discomfort. The other is a video of the student after being slightly startled, and some of its characteristics match those of the current student. The pre-set emotional target value is 6 (out of 10), indicating a state of mild nervousness. For the two reference face videos found, the server obtains their corresponding second multi-dimensional features. Taking the sick student's reference face video as an example, its second multi-dimensional features may include a more obvious pale face feature quantization value of 8 (out of 10) in the facial expression dimension, a cough sound feature quantization value of 6 (out of 10) in the voice dimension, and a slow walking feature quantization value of 7 (out of 10) in the behavior dimension. The server associates the second multi-dimensional features of the reference face videos, the pre-set emotional target values, and the first multi-dimensional features of the current face video. First, arrange the feature values of each dimension correspondingly to form a preliminary framework of a matrix. For example, place the first multi-dimensional feature values of the current face video in the first row, the second multi-dimensional feature values of the sick student's reference face video in the second row, and the second multi-dimensional feature values of the startled student's reference face video in the third row. Then, add a column in the matrix to record the emotional target values, and fill the pre-set emotional target values of the two reference face videos into the corresponding rows. At the same time, in order to more clearly represent the relationship between the features, the server may assign weights to the elements in the matrix according to the categories and importance of the features. For example, the feature weight of the facial expression dimension is set to 0.3, the feature weight of the voice dimension is set to 0.3, and the feature weight of the behavior dimension is set to 0.4. In this way, the server establishes an emotional feature association matrix. This matrix comprehensively shows the association between the current face video and the reference face videos in each dimension and the corresponding emotional target values. The server uses complex data analysis algorithms and machine learning models to analyze and process the emotional feature association matrix to identify the emotional target value of the current face video. First, the server calculates the similarity between the first multi-dimensional features of the current face video and the second multi-dimensional features of the reference face videos according to the weights of the features in the matrix. For example, in the facial expression dimension, the similarity between the current student and the sick student's reference face video is 0.8, and the similarity between the current student and the startled student's reference face video is 0.6; in the voice dimension, the similarities are 0.7 and 0.5, respectively; in the behavior dimension, the similarities are 0.9 and 0.4, respectively. Then, the server multiplies the similarity of each dimension by the corresponding weight and adds them up to get the comprehensive similarity. For the sick student's reference face video, the comprehensive similarity is: (0.8×0.3+0.7×0.3+0.9×0.4)=0.81; for the startled student's reference face video, the comprehensive similarity is: (0.6×0.3+0.5×0.3+0.4×0.4)=0.49.Next, the server calculates the emotion target value of the current facial image based on the comprehensive similarity and the preset emotion target value of the baseline facial image, using methods such as linear interpolation. Assume the calculated emotion target value for the current student's facial image is 6.8 (out of 10). The server determines the corresponding emotion type of the current facial image based on the recognized emotion target value according to pre-defined emotion type classification rules. For example, the emotion type classification rules set in the server are: emotion target values ​​between 0 and 3 represent a calm state, 3 to 6 represent a mildly uneasy state, 6 to 9 represent a moderately uncomfortable or tense state, and 9 to 10 represent a highly urgent state. Since the current student's facial image has an emotion target value of 6.8, falling within the 6-9 range, the server determines that the student's current emotion type is moderately uncomfortable or tense. After determining the emotion type, the server can take subsequent actions according to appropriate strategies, such as activating the intercom function to inquire and reassure the student, or calling upon relevant emergency resources to provide assistance. Through the detailed steps described above, the server can accurately construct an emotion feature association matrix and determine the emotion type corresponding to the current facial image based on the matrix, providing an important basis for appropriate response in emergency situations.

[0033] In this embodiment of the invention, the first multidimensional feature includes a first image feature, and the second multidimensional feature includes a second image feature; the step of establishing the emotion feature association matrix based on the second multidimensional feature corresponding to each of the reference face images, the preset emotion target value of each of the reference face images, and the first multidimensional feature of the current face image can be implemented through the following example.

[0034] The first image features are subjected to feature mapping to obtain a first image semantic identifier set, and the second image features corresponding to each of the reference face images are subjected to feature mapping to obtain a second image semantic identifier set corresponding to each of the reference face images.

[0035] Obtain the first emotion identifier corresponding to the current face image and the second emotion identifier corresponding to each of the reference face images;

[0036] The emotion feature association matrix is ​​established based on the second image semantic identifier set, the second emotion identifier, and the emotion target value corresponding to each of the reference face images, as well as the first image semantic identifier set and the first emotion identifier.

[0037] In an embodiment of the present application, in an emergency room scene of a hospital, a patient comes to the IP network visual intercom emergency equipment due to sudden abdominal pain to trigger an alarm instruction. The server starts the camera to collect video data and determines the current facial image, and then extracts the first multi-dimensional feature from the current facial image, which contains the first image feature. The server uses a pre-trained feature mapping model to map the first image feature. For example, the first image feature extracted from the current facial image shows that the patient's face has a frown and a slightly sweaty forehead. The server's feature mapping model maps the image feature of "frown" to the semantic space to obtain the corresponding semantic identifier "pain"; and maps the image feature of "slightly sweaty forehead" to "unwell". Through the one-by-one mapping of multiple first image features, the server obtains a first image semantic identifier set, such as {"pain", "unwell"}. At the same time, the server finds two reference facial images close to the current facial image in the reference facial image library. One of the reference facial images is the image of another patient with abdominal pain, and its second image feature includes pale face and chapped lips. The server also maps these second image features, maps "pale face" to "weak", and "chapped lips" to "dehydration", thereby obtaining the second image semantic identifier set corresponding to the reference facial image, such as {"weak", "dehydration"}. The other reference facial image is the image of a patient with physical discomfort caused by anxiety, and its second image feature includes wandering eyes and frequent blinking. After feature mapping, the second image semantic identifier set is {"anxiety", "uneasy"}. The server determines the first emotion identifier corresponding to the current facial image according to the overall performance of the current facial image and the first image semantic identifier set, combined with the pre-set emotion classification rule. In this emergency room scene, the server determines the first emotion identifier as "pain and discomfort" by combining the patient's frown, forehead sweat, and the first image semantic identifier set {"pain", "unwell"}. For the two reference facial images found by the server, the server also determines the second emotion identifier according to their respective second image semantic identifier sets and overall performance. For the reference facial image of the patient with abdominal pain, combined with its second image semantic identifier set {"weak", "dehydration"} and the overall appearance of the sickly face, the server determines the second emotion identifier as "suffering from illness". For the reference facial image of the patient with physical discomfort caused by anxiety, according to the second image semantic identifier set {"anxiety", "uneasy"} and the overall performance of the nervous expression, the server determines the second emotion identifier as "anxious and uneasy". The server starts to construct the emotion feature association matrix. First, the first image semantic identifier set and the first emotion identifier of the current facial image are taken as a row of the matrix. For example, the first row of data is {"pain", "unwell", "pain and discomfort"}. Then, the second image semantic identifier set, the second emotion identifier and the emotion target value corresponding to the first reference facial image (the image of the patient with abdominal pain) are taken as another row of the matrix.Assuming that the baseline face image preset emotion target value is 8 (full score 10), this row of data is {"weak", "dehydrated", "sick and tortured", 8}. The relevant information corresponding to the second baseline face image (anxiety patient image) is taken as another row of the matrix. Assuming that its emotion target value is 7 (full score 10), the data in this row is {"anxiety", "uneasy", "anxious and uneasy", 7}. In the process of constructing the matrix, the server also considers the importance and relevance of each feature and assigns appropriate weights to different elements. For example, the weight of image semantic identification is 0.3, the weight of emotion identification is 0.4, and the weight of emotion target value is 0.3. In this way, the server successfully establishes the emotion feature correlation matrix. This matrix comprehensively reflects the correlation between the current face image and the baseline face image in terms of image semantics, emotion identification, and emotion target value. Through further analysis and processing of the matrix, the server can more accurately identify the current patient's emotional state and provide more powerful support for subsequent emergency handling. For example, according to the matrix information, the server can judge that the current patient's emotion is closer to the baseline image of the abdominal pain patient, and thus prioritize abdominal pain-related emergency handling measures, such as arranging medical personnel for preliminary examination, preparing appropriate treatment equipment, etc.

[0038] In the embodiment of the present application, the number of first image features includes at least two; the first image feature mapping obtains a first image semantic identification set, which can be implemented by the following examples.

[0039] Performing feature integration operation on at least two first image features obtains target enhanced image features;

[0040] Synchronize the target enhanced image features to the preset semantic feature domain through the pre-trained feature synchronization network to obtain the first image semantic identification set.

[0041] In an embodiment of the present application, for example, in a busy train station waiting hall, a passenger suddenly feels unwell and triggers an alarm instruction in front of an IP network visual intercom emergency device. After receiving the alarm instruction, the server starts the camera to collect video data and determines the current face image from it, then extracts the multi-dimensional features of the image to obtain a plurality of first image features. Suppose two key first image features are extracted from the current face image: one is the feature of furrowed brows in facial expression, and the other is the feature of slightly pale face. The server first performs feature integration operation on the two first image features. In the execution of the feature integration operation, the server uses advanced image processing algorithms and deep learning technology. For the feature of furrowed brows, the server analyzes the degree, angle and duration of the furrowed brows, and converts them into a set of quantitative data, such as a furrowed brow degree of 7 (full score of 10), an angle of 30 degrees, etc. For the feature of slightly pale face, the server also analyzes the degree and distribution area of the paleness, and converts them into corresponding quantitative data, such as a paleness degree of 6 (full score of 10), mainly distributed in the cheek and forehead area, etc. Then, the server fuses and processes these quantitative data. For example, by using the weighted average method, different weights are assigned according to the importance of different features in reflecting the unwellness. Suppose the weight of the furrowed brow feature is 0.4, and the weight of the pale face feature is 0.6. Then, after calculation, a comprehensive target enhanced image feature value is obtained. The calculation process is as follows: target enhanced image feature value = furrowed brow feature quantitative value x weight + pale face feature quantitative value x weight = 7 x 0.4 + 6 x 0.6 = 2.8 + 3.6 = 6.4 (full score of 10), which comprehensively reflects the unwellness degree embodied by the two first image features of furrowed brows and pale face, and it is more comprehensive and accurate than a single feature in describing the current passenger's state, becoming a target enhanced image feature. After obtaining the target enhanced image feature, the server synchronizes it to the preset semantic feature domain through the pre-trained feature synchronization network. This feature synchronization network is trained by a large amount of data, and it can understand the correspondence between image features and semantic information. In the train station scenario, the server inputs the target enhanced image feature value 6.4 into the feature synchronization network. The feature synchronization network first analyzes and interprets this value. It knows that in the previous training data, similar values are usually associated with which semantic information. For example, when the target enhanced image feature value is between 6-8, the corresponding semantic information may include "unwellness", "possible health problems", etc. Then, the feature synchronization network maps the target enhanced image feature to the preset semantic feature domain according to these association relationships. In the preset semantic feature domain, there is a series of pre-defined semantic identifiers.For the current target enhanced image feature value 6.4, the feature synchronization network maps it to the semantic labels of "unwell" and "possible health problem". Finally, the server obtains these mapped semantic labels from the feature synchronization network to form a first image semantic label set, that is, {"unwell", "possible health problem"}. This first image semantic label set can more intuitively express the semantic information embodied in the current face image, and provides an important basis for subsequent establishment of an emotion feature correlation matrix. Through such a process, the server successfully converts the plurality of first image features extracted from the current face image into a first image semantic label set with clear semantic meaning, making the analysis and judgment of the passenger's emotional state more accurate and in-depth, and providing strong support for subsequent appropriate emergency measures. For example, according to the first image semantic label set, the server can preliminarily judge that the passenger may need medical assistance, so as to timely contact the medical personnel of the station to assist.

[0042] In the embodiment of the present application, the establishment of the emotion feature correlation matrix according to the second image semantic label set, the second emotion label and the emotion target value corresponding to each of the reference face images, and the first image semantic label set and the first emotion label can be implemented by the following examples.

[0043] According to the first semantic label for distinguishing different information in the same face image, and the second image semantic label set, the second emotion label and the emotion target value corresponding to the same reference face image, an emotion feature correlation parameter is established to obtain at least two emotion feature correlation parameters corresponding to at least two reference face images;

[0044] The at least two emotion feature correlation parameters are combined to obtain an initial emotion correlation matrix, wherein at least two emotion feature correlation parameters in the initial emotion correlation matrix are separated by a second semantic label for distinguishing information of different face images;

[0045] According to the first semantic label, the first image semantic label set and the first emotion label, an emotion classification label is established.

[0046] According to the initial emotion correlation matrix and the emotion classification label, the emotion feature correlation matrix is generated.

[0047] In the embodiment of the present application, an exemplary IP network visual intercom emergency device is triggered by an employee in an office area of an office building. After receiving the alarm instruction, the server obtains a first image semantic identifier set and a first emotion identifier of the current face image through a series of processes, and also finds two reference face images close to the current face image and their respective second image semantic identifier sets, second emotion identifiers and emotion target values. Assuming that the first semantic identifier is used to distinguish different information such as facial expressions and body postures in the same face image. For the first reference face image, its second image semantic identifier set is {"tired", "dull eyes"} and the second emotion identifier is "work tired" with an emotion target value of 7 (full score of 10). The server associates these information according to the first semantic identifier. For example, for the facial expression related information, "tired" is associated with "facial state" in the first semantic identifier, and "dull eyes" is also associated with "facial state"; for the overall emotional state, "work tired" is associated with "emotional state" in the first semantic identifier; and the emotion target value 7 is associated with "emotional quantification". In this way, an emotion feature association parameter is established, which can be represented as {(facial state: {"tired", "dull eyes"}), (emotional state: "work tired"), (emotional quantification: 7)}. For the second reference face image, its second image semantic identifier set is {"frown", "slight tremor"} and the second emotion identifier is "excessive pressure" with an emotion target value of 8 (full score of 10). Similarly, according to the first semantic identifier, the established emotion feature association parameter is {(facial state: {"frown", "slight tremor"}), (emotional state: "excessive pressure"), (emotional quantification: 8)}. In this way, the server obtains two emotion feature association parameters corresponding to the two reference face images. After obtaining the two emotion feature association parameters, the server combines them to form an initial emotion association matrix. Assuming that the second semantic identifier is used to distinguish different face images, and "reference image 1" and "reference image 2" are used to represent the two different reference face images. The server arranges the two emotion feature association parameters in a certain order and separates them using the second semantic identifier. For example, the initial emotion association matrix can be represented as:

[0048] ;

[0049] In this way, the emotional feature association parameters of different reference face images are clearly separated by the second semantic identification, forming an ordered initial emotional association matrix. After obtaining the first semantic identification, the first image semantic identification set, and the first emotional identification, the server begins to establish an emotional classification identification. Assuming that the first image semantic identification set of the current face image is {"slightly tired", "eyes slightly dull"}, and the first emotional identification is "mild fatigue". The server integrates and classifies these information according to the first semantic identification. For example, according to the "facial state" in the first semantic identification, "slightly tired" and "eyes slightly dull" in the first image semantic identification set are classified into one category; according to the "emotional state", the first emotional identification "mild fatigue" is classified into one category. Then, the server establishes an emotional classification identification according to these classification information, which can be represented as {(facial state: {"slightly tired", "eyes slightly dull"}), (emotional state: "mild fatigue")}. This emotional classification identification can clearly reflect the characteristics and emotional state of the current face image. After the server has the initial emotional association matrix and the emotional classification identification, it fuses them to generate the final emotional feature association matrix. The server adds the emotional classification identification as a new row to the initial emotional association matrix, and uses a specific identification such as "current image" to represent that this row corresponds to the information of the current face image. The generated emotional feature association matrix is as follows:

[0050] ;

[0051] Here, "-" represents that the emotional target value of the current face image has not been determined and needs to be further analyzed and calculated according to this emotional feature association matrix. In this way, the server successfully establishes the emotional feature association matrix, which integrates the relevant information of the reference face images and the current face image, providing an important data basis for accurately identifying the emotional target value and emotional type of the current face image in the future. For example, the server can use specific algorithms and models to calculate the emotional target value of the current face image according to the relationships and differences between the elements in the matrix, and then determine its emotional type, so as to take appropriate emergency measures.

[0052] In the embodiments of the present application, the first emotional identification corresponding to the current face image and the second emotional identification corresponding to each reference face image can be implemented by the following examples.

[0053] Obtain the collection location of the current face image and the voice content recognized by the voice data of the current face image;

[0054] Generate emergency scene information of the current face image according to the background image of the current face image;

[0055] generating the first emotion label according to the collection site, the voice content, and the emergency scene information.

[0056] In an embodiment of the present application, for example, in a large shopping mall, IP network visual intercom emergency devices are distributed in various areas. At this time, a customer suddenly triggers the alarm instruction of the emergency device in the clothing area of the mall. After receiving the alarm instruction, the server immediately begins to process the relevant information. First, the server obtains the collection location information of the current face image from the IP network visual intercom emergency device. The emergency device is built-in with a positioning system, which can accurately send the location information of itself to the server. In this example, the server obtains the collection location as the clothing area of the third floor of the mall near a certain store. At the same time, the server identifies the voice data attached to the current face image. The emergency device also synchronously collects the sound information around when collecting video data. This customer may have some voice expressions when triggering the alarm instruction, such as saying, “I suddenly feel dizzy, and my heart is very uncomfortable.” After receiving these voice data, the server converts them into text form using advanced voice recognition technology, i.e., “I suddenly feel dizzy, and my heart is very uncomfortable,” thereby obtaining the voice content. After obtaining the current face image, the server analyzes the background image to generate emergency scene information. In this shopping mall scenario, the background of the current face image shows that there are many clothes racks and clothing display racks around, the store has normal lighting, the customer flow is moderate, but there are no obvious signs of chaos or danger. The server interprets the background image through image recognition algorithms and pre-trained scene analysis models. It identifies that this is a normal clothing sales area, and in combination with the “dizziness, discomfort” mentioned in the customer's voice content, the server determines that the current emergency scene is that the customer suddenly has a physical discomfort during shopping in the mall. In order to generate more detailed emergency scene information, the server may also analyze other details in the background image. For example, it is observed that there are no obvious obstacles or dangerous items around the customer, and the other customers around do not show abnormal behavior, etc. Integrating these information, the emergency scene information generated by the server is: “In the clothing area of the third floor of the mall, a customer suddenly feels dizzy and uncomfortable during normal shopping, and there are no obvious dangerous factors in the surrounding environment.” After obtaining the collection location, voice content, and emergency scene information, the server begins to integrate these information to generate the first emotion identifier. The collection location is the clothing area of the mall, which is a relatively safe and normal shopping environment and generally does not cause people to have strong emotions such as fear and panic. The voice content indicates that the customer clearly states “dizziness, heart very uncomfortable,” which reflects the customer's current poor physical condition and may be accompanied by some anxiety and anxiety. The emergency scene information indicates that there are no obvious dangerous factors in the surrounding environment, and the main problem is the customer's physical discomfort. The server integrates and judges these information according to pre-set emotion classification rules and algorithms. In this example, according to the customer's voice content and emergency scene information, the server determines that the customer's current main emotion is anxiety and anxiety due to physical discomfort.Therefore, the server generates a first emotion identifier as "anxiety caused by physical discomfort". The first emotion identifier accurately summarizes the emotional state of the current customer in a specific collection location and a specific emergency scene. After generating the first emotion identifier, the server can further take corresponding measures according to the identifier. For example, the server can start the intercom function of the IP network visual intercom emergency device, contact the staff or medical personnel of the shopping mall to go to the collection location to provide help and support for the customer, and at the same time, calm the customer's emotion through voice and inform the customer that the relevant personnel are on the way. In this way, the server can more effectively deal with the emergency and protect the safety and health of the customer.

[0057] In the embodiment of the present application, the first multi-dimensional feature includes target enhanced image features and first semantic features; at least two reference facial images close to the current facial image can be matched according to the first multi-dimensional feature, which can be implemented by the following examples.

[0058] A preset enhanced image feature matching pool and a semantic feature matching pool are obtained, the enhanced image feature matching pool includes a corresponding relationship between each undetermined facial image and the enhanced image feature corresponding to the undetermined facial image, and the semantic feature matching pool includes a corresponding relationship between each undetermined facial image and the semantic feature corresponding to the undetermined facial image;

[0059] The enhanced image feature matching pool and the semantic feature matching pool are matched according to the target enhanced image feature respectively, and the enhanced image feature matching pool and the semantic feature matching pool are matched according to the first semantic feature respectively, to obtain at least two target undetermined facial images close to the current facial image;

[0060] The reference facial image is determined from the at least two target undetermined facial images according to an image matching coefficient between the at least two target undetermined facial images and the current facial image.

[0061] In the embodiments of the present application, for example, in an intelligent community, the server stores a large amount of face image data for assisting emergency emotion recognition. Among them, there are pre-set enhanced image feature matching pool and semantic feature matching pool. The enhanced image feature matching pool is a huge database that stores a large number of pending face images and their respective enhanced image features. For example, a pending face image shows the state of a resident after participating in an intense movement in the community activity, and its corresponding enhanced image features may include a higher degree of facial redness (quantitative value is 8, full score is 10), obvious sweat on the forehead (quantitative value is 7, full score is 10), etc. These features are obtained by advanced image processing algorithms and feature extraction techniques, which can more accurately describe the detailed information of the face image. The semantic feature matching pool is also a rich data set that records the correspondence between each pending face image and the corresponding semantic feature. For example, for the face image of the resident after participating in the intense movement, its corresponding semantic features may be "tired after exercise" and "body has certain consumption". These semantic features are obtained by analyzing and understanding the behavior, expression and surrounding environment of the image. When the server needs to perform face image matching, it will first obtain the data of the two matching pools from the storage system to prepare for the subsequent matching work. On a certain day, a resident in the community triggers the alarm instruction of the IP network video intercom emergency device at his own door. After receiving the instruction, the server starts the camera to collect video data and determines the current face image, and then extracts the target enhanced image features and the first semantic features from it. Assuming that the target enhanced image features in the current face image show that the resident's face is slightly red (quantitative value is 5, full score is 10) and there are a small amount of sweat on the forehead (quantitative value is 4, full score is 10). The server will match these target enhanced image features with the enhanced image features of each pending face image in the enhanced image feature matching pool one by one. During the matching process, the server will calculate the similarity between them. For example, through a certain similarity calculation algorithm, it is found that the similarity with the enhanced image features of the face image of the resident after participating in the intense movement is 0.6 (the similarity value range is 0-1, the higher the value, the more similar). At the same time, the server will also match according to the first semantic features of the current face image. Assuming that the first semantic feature is "slight discomfort", the server will match this semantic feature with the semantic features of each pending face image in the semantic feature matching pool. When the semantic feature "tired after exercise" of the face image of the resident after participating in the intense movement is matched, it is found that there is a certain relevance in semantics, and the semantic similarity is calculated by a semantic analysis algorithm, which is 0.5. Combining the matching results of the target enhanced image features and the first semantic features, the server will screen out the target pending face image close to the current face image.In addition to the above-mentioned resident face image after participating in intense exercise, another pending face image can be found, such as a resident face image after working outdoors for a period of time. The similarity of the enhanced image features of this image to the target enhanced image features of the current face image is 0.55, and the semantic similarity of the semantic feature "fatigue after work" to the first semantic feature "slight discomfort" of the current face image is 0.45. In this way, the server obtains two target pending face images close to the current face image. After obtaining two target pending face images, the server needs to further determine the reference face image. This requires calculating the image matching coefficient between the target pending face image and the current face image. The calculation of the image matching coefficient will consider the similarity of the target enhanced image features and the similarity of the first semantic features. The server will assign different weights to these two similarities, assuming that the weight of the target enhanced image feature similarity is 0.6 and the weight of the first semantic feature similarity is 0.4. For the resident face image after participating in intense exercise, the image matching coefficient is calculated as follows: image matching coefficient = target enhanced image feature similarity × weight + first semantic feature similarity × weight = 0.6 × 0.6 + 0.5 × 0.4 = 0.36 + 0.2 = 0.56. For the resident face image after working outdoors for a period of time, the image matching coefficient is calculated as follows: image matching coefficient = 0.55 × 0.6 + 0.45 × 0.4 = 0.33 + 0.18 = 0.51. By comparing the image matching coefficients of the two target pending face images, the server finds that the image matching coefficient (0.56) of the resident face image after participating in intense exercise is higher. Therefore, the server will determine the resident face image after participating in intense exercise as the reference face image. After determining the reference face image, the server can use the relevant information of the reference face image, such as the pre-set emotion target value, to further analyze the emotional state of the current face image, providing more accurate basis for subsequent emergency handling. For example, if the emotion target value of the reference face image is expressed as "moderate fatigue", the server can combine the specific circumstances of the current face image to preliminarily judge that the emotional state of the current resident is also inclined to moderate fatigue, and then take appropriate measures, such as contacting community medical personnel for simple examination, etc.

[0062] In the embodiment of the present application, the at least two target pending face images include at least two target pending face images matched by each matching strategy; and the reference face image is determined from the at least two target pending face images according to the image matching coefficients between the at least two target pending face images and the current face image. The implementation can be performed by the following examples.

[0063] For each matching strategy, the at least two target pending face images are matched to determine the matching coefficient of the current face image and each target pending face image.

[0064] For each target pending face image, confirming a matching coefficient mean value of the target pending face image in each matching strategy with the current face image;

[0065] According to the matching coefficient mean value, determining the reference face image from the at least two target pending face images.

[0066] In the embodiment of the present application, for example, in a large shopping center, the server is responsible for processing various information transmitted by the IP network visual intercom emergency equipment. One day, a customer suddenly feels unwell in the mall, triggering the alarm instruction of the emergency equipment. After receiving the instruction, the server starts the relevant process, obtains the target enhanced image feature and the first semantic feature through the previous steps, and matches multiple target pending face images in the enhanced image feature matching pool and the semantic feature matching pool through different matching strategies. Here we assume that there are two matching strategies: the matching strategy based on image feature similarity and the matching strategy based on semantic feature correlation. The matching strategy based on image feature similarity mainly focuses on the similarity between the target enhanced image feature and the enhanced image feature of the pending face image; the matching strategy based on semantic feature correlation focuses on the correlation between the first semantic feature and the semantic feature of the pending face image. Through the matching strategy based on image feature similarity, the server finds two target pending face images. The first target pending face image shows a customer who looks slightly tired after a long walk in the mall, and the similarity between its enhanced image feature and the target enhanced image feature of the current face image after calculation is 0.7 (the similarity value range is 0-1, and the higher the value, the more similar). The second target pending face image is a customer who looks after a heatstroke in the mall, and the similarity between its enhanced image feature and the target enhanced image feature of the current face image is 0.6. Through the matching strategy based on semantic feature correlation, the server also finds another two target pending face images. One of them is a customer who looks anxious because he cannot find his child in the mall, and the correlation between its semantic feature and the first semantic feature of the current face image is determined to be 0.4 after analysis (the correlation value range is 0-1, and the higher the value, the stronger the correlation). The other is a customer who looks tired after shopping in the mall, and the correlation between its semantic feature and the first semantic feature of the current face image is 0.5. In this way, the server confirms the matching coefficient of the current face image and each target pending face image matched by each matching strategy. After determining the matching coefficient of each target pending face image and the current face image under different matching strategies, the server needs to calculate the average matching coefficient of each target pending face image in all matching strategies. For the customer's face image who looks slightly tired after a long walk, its matching coefficient under the matching strategy based on image feature similarity is 0.7, and since it is not matched in the matching strategy based on semantic feature correlation, its matching coefficient under this strategy is defaulted to 0. Then its average matching coefficient is calculated as follows: average matching coefficient = (0.7 + 0) / 2 = 0.35.For the customer face image after heatstroke, the matching coefficient is 0.6 under the matching strategy based on image feature similarity, and the matching coefficient is 0 by default under the matching strategy based on semantic feature correlation, and the average matching coefficient is: average matching coefficient=(0.6+0) / 2=0.3. For the customer face image who is anxious to find the child, the matching coefficient is 0.4 under the matching strategy based on semantic feature correlation, and the matching coefficient is 0 by default under the matching strategy based on image feature similarity, and the average matching coefficient is: average matching coefficient=(0+0.4) / 2=0.2. For the customer face image who feels tired after shopping, the matching coefficient is 0.5 under the matching strategy based on semantic feature correlation, and the matching coefficient is 0 by default under the matching strategy based on image feature similarity, and the average matching coefficient is: average matching coefficient=(0+0.5) / 2=0.25. Through such calculation, the server obtains the average matching coefficient of each target face image in all matching strategies. After calculating the average matching coefficient of each target face image, the server determines the reference face image according to the average matching coefficient. The higher the average matching coefficient, the higher the overall matching degree between the target face image and the current face image. In the above example, the average matching coefficient of the customer face image who is slightly tired after a long walk is 0.35, which is the highest among the four target face images. Therefore, the server determines the customer face image who is slightly tired after a long walk as the reference face image. After determining the reference face image, the server can further analyze the emotional state of the current customer by using the related information of the reference face image, such as the preset emotional target value. For example, if the preset emotional target value of the reference face image is represented as "mild fatigue", the server can preliminarily judge that the emotional state of the customer who triggers the alarm instruction is also inclined to mild fatigue in combination with the specific circumstances of the current face image. Based on this judgment, the server can take appropriate emergency measures, such as sending a notification to the staff of the shopping mall through the IP network visual intercom emergency device, informing them that there is a customer who may be unwell and needs help, such as guiding the customer to rest in the rest area, providing drinking water, etc. At the same time, the server can also send some warm tips and soothing information to the customer through the emergency device, informing the customer that the relevant personnel are on their way, and asking the customer not to worry too much.

[0067] In this way, the server can more accurately determine the reference face image, thereby more effectively analyzing the emotional state of the current face image and providing strong support for emergency handling.

[0068] In the embodiment of the application, the multi-dimensional feature extraction of the current face image to obtain the first multi-dimensional feature can be implemented by the following example.

[0069] extracting at least two images from the current face image, and segmenting the at least two images into at least two image sets;

[0070] selecting a target image from the at least two image sets, and performing enhanced image feature extraction on the target image to obtain a first image feature;

[0071] obtaining a first emotion label corresponding to the current face image, and performing semantic feature extraction on the first emotion label to obtain a first semantic feature;

[0072] obtaining the first multi-dimensional feature according to the first image feature and the first semantic feature.

[0073] In an embodiment of the present application, an exemplary passenger triggers an alarm instruction of the IP network visual intercom emergency device in the departure hall of an airport. After receiving the alarm instruction, the server starts the camera to collect video data and determines the current facial image from it. The server starts to extract images from the current facial image. Assuming the duration of the facial image is 10 seconds, the server extracts two images from this image at certain time intervals, which are the 3rd second and 7th second images. These two images capture the passenger's state at different times. Next, the server segments these two images to form at least two image sets. The segmentation is based on the passenger's facial expressions and body postures and other features in the images. In the 3rd second image, the passenger's facial expression shows slight anxiety, with a slightly furrowed brow; the body posture is relatively relaxed, without obvious signs of tension. According to these features, the server divides this part of the image into an image set representing "mild anxiety but relaxed body". In the 7th second image, the passenger's facial expression becomes more anxious, with a more furrowed brow and a hint of unease in the eyes; at the same time, the body posture becomes somewhat rigid, with both hands involuntarily clenched. According to these features, the server divides this image into another image set representing "increased anxiety and body tension". Through such segmentation operations, the server classifies the two extracted images into two different image sets, providing a more targeted data basis for subsequent feature extraction and analysis. After completing the segmentation of the image sets, the server needs to select the target image from these image sets. In this example, since the "increased anxiety and body tension" image set reflects the passenger's state more prominently and critically, the server selects the 7th second image in this set as the target image. After selecting the target image, the server starts to extract enhanced image features. The server uses advanced image processing algorithms and deep learning techniques to analyze the target image from multiple aspects. First, the server focuses on the passenger's facial features in the target image. Through detailed analysis of the face, the server detects that the passenger's furrowed brow reaches a degree of 6 (out of 10), the pupils are slightly dilated with a quantization value of 5 (out of 10), and the lips are slightly trembling with a quantization value of 4 (out of 10). These facial feature quantization values reflect the passenger's anxious and uneasy emotional state. Next, the server analyzes the passenger's body posture features in the target image. It finds that the passenger's body is slightly leaning forward with a quantization value of 3 (out of 10), and the hands are tightly clenched with a quantization value of 7 (out of 10). These body posture features further indicate the passenger's nervous emotion. Based on the above analysis, the server extracts the first image features from the target image, which are presented in the form of quantized data, such as {(furrowed brow degree: 6), (pupil dilation degree: 5), (lip trembling degree: 4), (body leaning degree: 3), (hand clenching degree: 7)}. These first image features can more accurately and detailedly describe the passenger's state at the current time.After obtaining the first image feature, the server also needs to obtain the first emotion identifier corresponding to the current face image. According to the previous analysis of the image and the scene in which the passenger may be (waiting for a flight in the airport waiting hall), the server comprehensively judges that the first emotion identifier of the current passenger is "flight-related anxiety emotion". After determining the first emotion identifier, the server extracts the semantic features thereof. The server uses natural language processing technology and semantic analysis algorithm to deeply analyze the identifier "flight-related anxiety emotion". From the semantic level, the server analyzes that "flight-related" means that this anxiety emotion is closely related to the situation of the flight, and may be worried about flight delay, luggage loss and other problems; "anxiety emotion" reflects the current psychological state of the passenger as uneasy and worried. The server further refines and quantifies these semantic information, for example, the semantic feature of "flight-related" can be quantified as {(flight delay worry: 0.8), (luggage problem worry: 0.2)}, indicating that the main worry of the passenger is flight delay; the semantic feature of "anxiety emotion" can be quantified as {(uneasy degree: 0.7), (worry degree: 0.9)}. Through such semantic feature extraction, the server obtains the first semantic feature, such as {(flight delay worry: 0.8), (luggage problem worry: 0.2), (uneasy degree: 0.7), (worry degree: 0.9)}. These first semantic features describe the emotional state of the passenger from the semantic point of view. After obtaining the first image feature and the first semantic feature, the server fuses them to obtain the first multi-dimensional feature. The first multi-dimensional feature is an information set that comprehensively combines image features and semantic features, and can more comprehensively and deeply reflect the passenger state represented by the current face image. In this example, the server combines the first image feature and the first semantic feature to obtain the first multi-dimensional feature as follows: {(eyebrow furrowing degree: 6), (pupil dilation degree: 5), (lips trembling degree: 4), (body leaning forward degree: 3), (both hands tightly holding degree: 7), (flight delay worry: 0.8), (luggage problem worry: 0.2), (uneasy degree: 0.7), (worry degree: 0.9)}. Through such a first multi-dimensional feature, the server can more accurately understand the current state of the passenger, and provide rich and detailed data support for subsequent matching of the reference face image, determination of the emotion type and taking of corresponding emergency measures. For example, the server can further analyze the severity of the passenger's emotion according to these features, and if it is judged that the emotion is relatively serious, airport staff can be arranged in time to assist the passenger, understand the specific situation and provide corresponding help and comfort.

[0074] In the embodiment of the present application, the emotion target value of the current face image can be identified according to the emotion feature association matrix through the following example.

[0075] The mood element feature conversion is performed on the mood feature association matrix to obtain a mood feature set, and the mood element feature is standardized data corresponding to a target value recognition model;

[0076] The mood feature set is loaded into the target value recognition model, and the target value recognition model is obtained by maintaining a basic model parameter of a preset multi-modal large model unchanged, and optimizing an updated model parameter of the multi-modal large model according to a training mood feature set, the updated model parameter being related to an attention mechanism added to the multi-modal large model;

[0077] The target mood target value of the current face image generated by the target value recognition model is obtained.

[0078] In an embodiment of the present application, an exemplary IP network visual intercom emergency device is triggered by a passerby in a bustling commercial district. After a series of complex processing, the server has constructed an emotion feature correlation matrix. This matrix contains various feature information of the current face image and multiple reference face images, such as facial expression, semantic feature, and emotion target value, etc. Now, the server needs to perform emotion element feature conversion on this emotion feature correlation matrix. Suppose the elements in the emotion feature correlation matrix contain feature values of different dimensions, such as facial expression quantization value (0-10), semantic feature correlation degree (0-1), and emotion target value (0-10), etc. The value range and data type of these feature values are different, and need to be standardized to meet the input requirements of the target value recognition model. The server first standardizes the facial expression quantization value in the matrix. For example, the facial expression quantization value of the current face image is 8, and the server converts it to standardized data that the target value recognition model can accept through a standardization algorithm. Suppose the standardization algorithm is to subtract the mean value and divide by the standard deviation, and after calculation, the value 8 is converted to 0.6 (this is just an example, the actual calculation will be based on the specific mean value and standard deviation). For the semantic feature correlation degree, the server also performs similar processing. For example, the correlation degree of the current face image with a certain semantic feature is 0.7, which is converted to 0.5 after standardization. The emotion target value is also standardized accordingly. For example, the emotion target value of a certain reference face image is 6, which is standardized to 0.4. After the server performs such emotion element feature conversion on each element in the emotion feature correlation matrix, it arranges the converted data into an emotion feature set. All data in this emotion feature set are standardized data corresponding to the target value recognition model, such as {(facial expression standardized value: 0.6), (semantic feature correlation degree standardized value: 0.5), (emotion target value standardized value: 0.4)}. This data form is more conducive to the subsequent processing and analysis of the target value recognition model. After obtaining the emotion feature set, the server loads it into the target value recognition model. This target value recognition model is obtained through careful training and optimization. The target value recognition model is based on a pre-set multi-modal large model. The multi-modal large model itself has strong processing capability and can handle multiple types of data such as images and texts. During the training of the target value recognition model, the server maintains the basic model parameters of the multi-modal large model unchanged, which is to preserve the general knowledge and feature representation ability learned by the multi-modal large model on large-scale data. At the same time, the server optimizes the updated model parameters of the multi-modal large model according to the training emotion feature set. The training emotion feature set is obtained by a large number of face image data with known emotion target values through a similar feature extraction and conversion process as before.By having the multi-modal large model learn the data patterns and rules in these training emotion feature sets, the model parameters are adjusted and updated so that it can more accurately identify the emotion target value. In this process, the attention mechanism added to the multi-modal large model plays a key role. The attention mechanism enables the model to pay more attention to important features related to the emotion target value. For example, in the current business district scenario, for the pedestrian's facial image, the attention mechanism may pay more attention to the key parts of the facial expression that can reflect the emotional state, such as eye contact and furrowed brows, as well as the key information in the semantic features related to the current scene, such as whether it mentions physical discomfort or shows signs of anxiety. When the server loads the emotion feature set into the target value recognition model, the model uses its updated model parameters and attention mechanism to perform in-depth analysis and processing on the input emotion feature set. It assigns different attention weights based on the importance of each feature, focusing more on those features that play a key role in identifying the emotion target value. After processing and analysis by the target value recognition model, the server obtains the target emotion target value generated by the model for the current facial image. In the business district scenario, the target value recognition model considers various standardized data in the emotion feature set, as well as the key features focused on by the attention mechanism. For example, the model notices that the facial expression standardization value of the current facial image is high, indicating that the facial expression shows strong emotions; the semantic feature correlation degree standardization value also shows that there is a high correlation between some negative emotions and semantic features. Based on these analyses, the target value recognition model generates the target emotion target value for the current facial image. Assuming that the generated target emotion target value is 8.5 (out of 10), this value indicates that the pedestrian's current emotion is in a state of high tension or anxiety. After obtaining this target emotion target value, the server can determine the emotion type corresponding to the current facial image based on the pre-set emotion type classification rules. For example, if 7-10 is set as the high tension and anxiety type, the server can determine that the current emotion type of the pedestrian is high tension and anxiety. Based on this judgment result, the server can take appropriate emergency measures. For example, through the IP network visual intercom emergency device, the server can send a notification to nearby security personnel or medical personnel, informing them that there is a pedestrian in a state of high tension and anxiety, and they need to go there immediately to provide help. At the same time, the server can also send some calming information to the pedestrian through the emergency device, informing them that relevant personnel are on their way and asking them to remain calm as much as possible. Through this way, the server uses the emotion feature correlation matrix to perform emotion element feature conversion, loads it into the target value recognition model, and obtains the target emotion target value, which can accurately identify the current facial image's emotional state, providing strong support for timely and effective emergency handling.

[0079] In the embodiments of the present application, the training step of the target value recognition model can be implemented by the following example.

[0080] obtaining a first training enhanced image feature corresponding to a first training face image, a first training emotion label and a first training emotion target value, and a second training enhanced image feature corresponding to a second training face image, a second training emotion label and a second training emotion target value;

[0081] performing a feature synchronization operation on the first training enhanced image feature and the second training enhanced image feature according to an initial feature synchronization network trained in advance, obtaining a first training image label set and a second training image label set, and establishing a training emotion feature correlation matrix according to the first training image label set, the first training emotion label and the first training emotion target value, and the second training image label set and the second training emotion label;

[0082] performing an emotion element feature conversion on the training emotion feature correlation matrix to obtain the training emotion feature set;

[0083] adding the attention mechanism to the multi-modal large model, so that the multi-modal large model adds the updated model parameters through the attention mechanism;

[0084] maintaining the basic model parameters of the multi-modal large model unchanged, and optimizing the updated model parameters of the multi-modal large model according to the training emotion feature set and the second training emotion target value to obtain the target value recognition model.

[0085] The embodiment of the present application provides a computer device 100, which comprises a processor and a non-volatile memory storing computer instructions, and when the computer instructions are executed by the processor, the computer device 100 executes the aforementioned IP network visual intercom emergency emotion recognition method based on artificial intelligence. As shown in the figure, Figure 2 Figure 2 The computer device 100 provided by the embodiment of the present application is a structural block diagram. The computer device 100 comprises a memory 111, a processor 112 and a communication unit 113. In order to realize the transmission or interaction of data, the memory 111, the processor 112 and the communication unit 113 are directly or indirectly electrically connected with each other.

[0086] The foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments are chosen and described in order to best explain the principles of the disclosure and its practical application to thereby enable others skilled in the art to best utilize the disclosure and various embodiments with various modifications as are suited to the particular use contemplated.​

Claims

1. An AI-based IP network visual intercom emergency emotion recognition method, characterized in that, The method comprises the following steps: in response to the alarm instruction of the IP network visual intercom emergency device triggered by the current user, starting the camera of the IP network visual intercom emergency device; collecting current video data through the camera, and determining the current face image from the video data; constructing an emotion feature correlation matrix of the current face image, and determining the emotion type corresponding to the current face image according to the emotion feature correlation matrix; the method further comprises the following steps: obtaining the current face image, extracting at least two images from the current face image, and dividing the at least two images into at least two image sets; selecting a target image from the at least two image sets, and obtaining a first image feature by enhancing the image feature of the target image; obtaining a first emotion identifier corresponding to the current face image, and obtaining a first semantic feature by extracting the semantic feature of the first emotion identifier; obtaining a first multi-dimensional feature according to the first image feature and the first semantic feature, wherein the first multi-dimensional feature comprises a plurality of dimension features corresponding to a plurality of dimensions respectively; matching at least two reference face images close to the current face image according to the first multi-dimensional feature, wherein each reference face image is preloaded with an emotion target value; establishing the emotion feature correlation matrix according to the second multi-dimensional feature corresponding to each reference face image, the emotion target value preloaded in each reference face image, and the first multi-dimensional feature of the current face image; performing emotion element feature conversion on the emotion feature correlation matrix to obtain an emotion feature set, wherein the emotion element feature is standardized data corresponding to a target value recognition model; loading the emotion feature set into the target value recognition model; obtaining a target emotion target value of the current face image generated by the target value recognition model; determining the emotion type corresponding to the current face image according to the recognition result of the emotion target value.

2. The method of claim 1, wherein, The method further comprises the following steps: in the case that the emotion type belongs to a first-level emotion type, starting the intercom function of the IP network visual intercom emergency device to realize manual real-time reply interaction; in the case that the emotion type belongs to a second-level emotion type, loading the current voice data from the current user into a first generative dialogue model of an integrated model to obtain a basic target of the current user; the emergency degree of the second-level emotion type is greater than that of the first-level emotion type in an emergency scene; loading the past voice content of the full user in the preset period range and the basic target into a second generative dialogue model of the integrated model to obtain a final target of the current user; adapting the final target to a plurality of preset emergency strategies in a preset emergency strategy database to obtain a target emergency strategy suitable for the current user, and providing the target emergency strategy to the current user.

3. The method of claim 1, wherein, The first multi-dimensional features include first image features, and the second multi-dimensional features include second image features, and the number of the first image features includes at least two; the emotion feature association matrix is established according to the second multi-dimensional features corresponding to each of the reference face images respectively, the emotion target values preset for each of the reference face images, and the first multi-dimensional features of the current face image, and the emotion feature association matrix includes: performing a feature integration operation on the at least two first image features to obtain a target enhanced image feature; synchronizing the target enhanced image feature to a preset semantic feature domain through a feature synchronization network trained in advance to obtain a first image semantic identifier set, and performing feature mapping on the second image features corresponding to each of the reference face images respectively to obtain a second image semantic identifier set corresponding to each of the reference face images; obtaining a collection location of the current face image, and voice content recognized from voice data of the current face image; generating emergency scene information of the current face image according to a background image of the current face image; generating the first emotion identifier according to the collection location, the voice content, and the emergency scene information; establishing emotion feature association parameters according to a first semantic identifier used to distinguish different information in the same face image, the second image semantic identifier set, a second emotion identifier, and the emotion target value corresponding to the same reference face image, to obtain at least two emotion feature association parameters corresponding to at least two reference face images; merging the at least two emotion feature association parameters to obtain an initial emotion association matrix, wherein the at least two emotion feature association parameters in the initial emotion association matrix are separated by a second semantic identifier used to distinguish information of different face images; establishing an emotion classification identifier according to the first semantic identifier, the first image semantic identifier set, and the first emotion identifier; generating the emotion feature association matrix according to the initial emotion association matrix and the emotion classification identifier.

4. The method of claim 1, wherein, The first multi-dimensional features include a target enhanced image feature and a first semantic feature; matching at least two reference face images close to the current face image according to the first multi-dimensional features includes: obtaining a preset enhanced image feature matching pool and a semantic feature matching pool, the enhanced image feature matching pool including a corresponding relationship between each pending face image and an enhanced image feature corresponding to the pending face image, and the semantic feature matching pool including a corresponding relationship between each pending face image and a semantic feature corresponding to the pending face image; matching the enhanced image feature matching pool and the semantic feature matching pool according to the target enhanced image feature respectively, and matching the enhanced image feature matching pool and the semantic feature matching pool according to the first semantic feature respectively to obtain at least two target pending face images close to the current face image; determining the reference face images from the at least two target pending face images according to an image matching coefficient between the at least two target pending face images and the current face image.

5. The method of claim 4, wherein, The at least two target undetermined face images include at least two target undetermined face images matched by each matching strategy; and determining the reference face image from the at least two target undetermined face images according to the image matching coefficients between the at least two target undetermined face images and the current face image, comprises: For each matching strategy, confirming the matching coefficients of the current face image and each target undetermined face image matched by the at least two target undetermined face images; For each target undetermined face image, confirming the average matching coefficients of the target undetermined face image in each matching strategy and the current face image; Determining the reference face image from the at least two target undetermined face images according to the average matching coefficients.

6. A server system, characterized by A server is included, and the server is configured to execute the method in any one of claims 1-5.

Images