Production management emotion analysis system based on multi-dimensional indexes

Abstract

The present application relates to a production management emotion analysis system based on multi-dimensional indicators, relates to the field of emotion recognition and production management optimization, comprising: according to the key facial image sequence and the voice sequence, the user emotion recognition is carried out, and the first and second user emotion grades are obtained;According to the key voice sequence, the environmental noise intensity is identified, the environmental noise intensity sequence is obtained, the face weight and the voice weight are configured;The first and second user emotion grades are weighted to obtain the weighted user emotion grade, the voice quality influence coefficient is corrected to obtain the user emotion grade for management. Through the present application, the technical problems that the traditional method cannot efficiently identify the emotional fluctuation of employees in a complex production environment, and the emotional evaluation efficiency, accuracy and reliability are insufficient can be solved;The emotional change of employees in a complex production environment can be accurately perceived in real time, the efficiency, accuracy and reliability of emotional evaluation are significantly improved, and decision support is provided for intelligent production management of enterprises.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition and production management optimization, and in particular to a production management emotion analysis system based on multi-dimensional indicators. Background Technology

[0002] In modern enterprise production processes, employee emotion management and monitoring have become crucial for improving production efficiency, optimizing the work environment, and enhancing employee satisfaction. Emotional fluctuations not only affect employee work efficiency but may also negatively impact the stability and quality of the entire production process. Therefore, timely and accurate identification and assessment of employee emotional states are essential for enterprise production management.

[0003] Traditional methods for identifying employee emotions mainly rely on single sensing technologies, such as facial expression recognition, voice analysis, or self-report questionnaires. These methods often have certain limitations and cannot comprehensively and accurately perceive changes in employee emotions. Especially in complex production environments, due to interference from external factors, these methods cannot efficiently and accurately identify employee emotional fluctuations. Summary of the Invention

[0004] This invention addresses the technical problem that traditional methods for identifying employee emotions cannot efficiently identify employee emotional fluctuations in complex production environments, resulting in insufficient efficiency, accuracy, and reliability in emotion assessment. It provides a production management emotion analysis system based on multi-dimensional indicators to solve this problem.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: This invention provides a production management emotion analysis system based on multi-dimensional indicators, including: a key information acquisition module, used to collect facial image sequences and voice sequences of users in the production space, extract key facial image sequences from the facial image sequences according to the production cycle, and index key voice sequences; a user emotion recognition module, used to perform user emotion recognition based on the key facial image sequences and voice sequences to obtain a first user emotion level and a second user emotion level; a weight calculation and configuration module, used to perform environmental noise intensity recognition based on the key voice sequences to obtain an environmental noise intensity sequence, and configure facial weights and voice weights according to the magnitude of each environmental noise intensity, wherein a voice quality influence coefficient is obtained through voice quality influence analysis for weight calculation and configuration; and a user emotion level acquisition module, used to perform weighted calculation of the first user emotion level and the second user emotion level based on the facial weights and voice weights to obtain a weighted user emotion level, and correct it using the voice quality influence coefficient to obtain the user emotion level for production management.

[0006] Optionally, the production management sentiment analysis system based on multi-dimensional indicators is further configured to: collect facial images and voice signals of users within a continuous time period in the production space, arrange them chronologically, and obtain facial image sequences and voice sequences; acquire production cycles within the production space, extract time frames between two adjacent production cycles as key frames, and obtain a cycle key frame sequence; extract facial images under multiple cycle key frames within the cycle key frame sequence to obtain a cycle key facial image sequence; perform similarity analysis and filtering on adjacent cycle key facial images within the cycle key facial image sequence to obtain a key facial image sequence; and extract voice signals under time frames corresponding to multiple key facial images within the key facial image sequence to obtain a key voice sequence.

[0007] Optionally, the production management sentiment analysis system based on multi-dimensional indicators is further configured to: select a first periodic key facial image within the periodic key facial image sequence and retain it as a key facial image; select a second periodic key facial image within the periodic key facial image sequence, identify the similarity with the first periodic key facial image, and obtain facial image similarity; determine whether the facial image similarity is greater than or equal to an image similarity threshold; if so, delete the second periodic key facial image and select a third periodic key facial image for similarity analysis; if not, retain the second periodic key facial image as a key facial image and continue similarity analysis and screening based on the second periodic key facial image; until all periodic key facial images within the periodic key facial image sequence have been screened, a key facial image sequence is obtained.

[0008] Optionally, the production management sentiment analysis system based on multi-dimensional indicators is further configured to: collect multiple sets of sample facial images based on facial image recognition data over a historical period, and identify the similarity between two sample facial images within each set of sample facial images to obtain a set of sample facial image similarities; construct a facial image similarity recognizer using a Siamese neural network; conduct supervised training, verification, and testing of the facial image similarity recognizer using the multiple sets of sample facial images and the set of sample facial image similarities until the loss is less than required; input the first period key facial image and the second period key facial image into the facial image similarity recognizer, and obtain the facial image similarity by recognition output.

[0009] Optionally, the production management emotion analysis system based on multi-dimensional indicators is further configured to: collect a set of sample key facial image sequences and a set of sample speech sequences based on user emotion test data, and collect the user's emotion levels under different sample key facial image sequences and sample speech sequences, labeling them as a first sample user emotion level set and a second sample user emotion level set; use the sample key facial image sequence set and the first sample user emotion level set as supervised training data, and train a facial emotion recognizer using a convolutional neural network; input the key facial image sequences into the facial emotion recognizer, and output to obtain the first user emotion level; use the sample speech sequence set and the second sample user emotion level set as supervised training data, and train a speech emotion recognizer; input the speech sequences into the speech emotion recognizer, and output to obtain the second user emotion level.

[0010] Optionally, the production management sentiment analysis system based on multi-dimensional indicators is further configured to: perform environmental noise filtering on multiple key speech signals within the key speech sequence to obtain multiple signal-to-noise ratios, and calculate an environmental noise intensity sequence; calculate the mean of the environmental noise intensity sequence to obtain an average environmental noise intensity; calculate a speech quality impact coefficient based on the average environmental noise intensity; configure a preset speech weight to obtain a speech weight; and calculate a facial weight.

[0011] Optionally, the production management emotion analysis system based on multi-dimensional indicators is further used to: calculate the ratio of the preset environmental noise intensity to the average environmental noise intensity to obtain the voice quality influence coefficient; multiply the voice quality influence coefficient by the preset voice weight to perform weight calculation configuration to obtain the voice weight; and subtract the voice weight from 1 to obtain the facial weight.

[0012] Optionally, the production management emotion analysis system based on multi-dimensional indicators is further used to: calculate the weighted emotion level of the first user and the emotion level of the second user based on the facial weight and the voice weight to obtain a weighted emotion level; determine whether the voice quality influence coefficient is less than 1. If so, multiply the weighted emotion level by the voice quality influence coefficient to complete the user emotion level correction, obtain the user emotion level, and perform production management. If not, no user emotion level correction is performed.

[0013] The beneficial effects of this invention are as follows: By collecting facial image sequences and voice sequences of users within a production space, and extracting key facial image sequences from the facial image sequences according to the production cycle, and indexing key voice sequences; then, based on the key facial image sequences and voice sequences, user emotion recognition is performed to obtain a first user emotion level and a second user emotion level; on the other hand, based on the key voice sequences, environmental noise intensity is recognized to obtain an environmental noise intensity sequence, and facial weights and voice weights are configured according to the magnitude of each environmental noise intensity; then, based on the facial weights and voice weights, the first user emotion level and the second user emotion level are weighted and calculated to obtain a weighted user emotion level; finally, the voice quality influence coefficient is used to correct the weighted user emotion level to obtain the user emotion level for production management. In other words, through multimodal data fusion, key feature extraction, and noise interference correction, data accuracy and analysis efficiency can be effectively improved, and changes in employee emotions in complex production environments can be perceived in real time and accurately, thereby significantly improving the efficiency, accuracy, and reliability of employee emotion assessment and providing scientific decision support for intelligent production management in enterprises. Attached Figure Description

[0014] Figure 1 A schematic diagram of the structure of the production management sentiment analysis system based on multi-dimensional indicators provided by the present invention;

[0015] Figure 2 This is a flowchart illustrating the process of obtaining key facial image sequences in the production management emotion analysis system based on multi-dimensional indicators provided by the present invention.

[0016] The components represented by each number in the attached diagram are explained below:

[0017] The module includes a key information acquisition module 11, a user emotion recognition module 12, a weight calculation and configuration module 13, and a user emotion level acquisition module 14. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] In the description of this invention, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0021] Examples, such as Figure 1 As shown, this embodiment of the invention provides a production management sentiment analysis system based on multi-dimensional indicators, including:

[0022] The key information acquisition module 11 is used to collect facial image sequences and voice sequences of users in the production space, extract key facial image sequences from the facial image sequences according to the production cycle, and index key voice sequences.

[0023] Furthermore, the key information acquisition module 11 is also used for:

[0024] Within a continuous timeframe within the production space, facial images and voice signals of users are acquired and arranged chronologically to obtain facial image sequences and voice sequences. Production cycles within the production space are obtained, and time frames between two adjacent production cycles are extracted as keyframes to obtain a cycle keyframe sequence. Facial images under multiple cycle keyframes within the cycle keyframe sequence are extracted to obtain a cycle key facial image sequence.

[0025] Specifically, in this invention, the collection of all data (such as facial image data and voice data) is done with the explicit consent of the employees and strictly complies with relevant laws and regulations. The collection process complies with the provisions of the Personal Data Protection Law, the Privacy Protection Law, and relevant industry standards to ensure that the privacy of employees is fully protected. Before participating in data collection, employees are informed in detail of the purpose, scope, usage, and storage period of the data collection, and data collection is only carried out with the voluntary consent of the employees. In addition, all collected data will only be used for emotion recognition and production management optimization in this project and will not be used for other unauthorized purposes. Employees can withdraw their consent at any time, stop data collection, and have the right to request the deletion of the collected personal data.

[0026] First, within a continuous timeframe (such as normal working hours) in the production space (such as a production workshop), high-resolution cameras are used to capture real-time images of employees, recording changes in their facial expressions. By marking and sorting the images according to the acquisition time, a sequence of facial images is obtained. On the other hand, microphones are used to collect employees' voice data, recording their voice information during the production process (such as work feedback), and the voice signals are marked and sorted according to the acquisition time to obtain a voice sequence.

[0027] Next, the production cycle within the production space is obtained. The production cycle refers to the time period between the start of producing one part and the production of the next part. The time range of each production cycle will vary depending on the production task settings and production process. Then, time frames between two adjacent production cycles are extracted as keyframes. In particular, there are usually obvious emotional fluctuations between adjacent production cycles (such as between the production of two parts), especially after an employee finishes a work task, their facial expressions and emotional state may change significantly. Therefore, time frames with greater changes are selected from the time points of adjacent production cycles as keyframes through time series analysis. Within all production cycles, the key time frames of each cycle are obtained through time frame extraction methods, and a cycle keyframe sequence is constructed.

[0028] Then, for each periodic keyframe, the corresponding facial image data is extracted. Since employees will have different emotional fluctuations during the production process, the facial images at the keyframes can accurately capture these changes, especially at the time of task transition. Based on the facial images under multiple periodic keyframes, a periodic key facial image sequence is generated, which will become an important data source for subsequent emotion analysis.

[0029] Similarity analysis and filtering are performed on adjacent periodic key facial images within the periodic key facial image sequence to obtain the key facial image sequence.

[0030] Furthermore, such as Figure 2 As shown, the present invention further includes the following steps:

[0031] Select the first periodic key facial image within the periodic key facial image sequence and retain it as the key facial image.

[0032] Specifically, the first key facial image of the cycle is selected at the first time point (earliest timestamp) within the key facial image sequence of the cycle. This is the facial image captured at the earliest moment within the key facial image sequence of the cycle. This is usually at the beginning of the production cycle or the initial moment of employee emotional change, and can reflect the employee's emotional state at the start of the task. Then, the first key facial image of the cycle (i.e., the facial image with the earliest timestamp) is retained as the key facial image and used as the benchmark image for the emotion analysis of that cycle. This image will be used for subsequent emotion analysis, comparison, and tracking of changes in employee emotional state.

[0033] Within the periodic key facial image sequence, a second periodic key facial image is selected, and its similarity to the first periodic key facial image is identified to obtain facial image similarity.

[0034] Furthermore, the present invention also includes the following steps:

[0035] Based on facial image recognition data from historical periods, multiple sets of sample facial images are collected, and the similarity between two sample facial images within each set is identified to obtain a sample facial image similarity set. A Siamese neural network is used to construct a facial image similarity recognizer. The facial image similarity recognizer is subjected to supervised training, verification, and testing using the multiple sets of sample facial images and the sample facial image similarity set until the loss is less than required. The first period key facial image and the second period key facial image are input into the facial image similarity recognizer, and the facial image similarity is obtained by recognition output.

[0036] Specifically, the process involves querying enterprise production monitoring logs and collecting multiple sets of sample facial images based on facial image recognition data from historical periods (e.g., the last three months). Each set of sample facial images includes two sample facial images, which may have been collected under different work or emotional states. Next, the similarity between the two sample facial images within each set is determined using conventional image similarity measurement methods such as cosine similarity and Euclidean distance. By comparing the features (such as facial contours and expression features) of the two sample facial images, their similarity is judged. If the similarity between the two images is high, it indicates that the emotions or states represented by the two images are relatively similar; if the similarity is low, it indicates that there is a significant difference in the emotions or states between the two images. This process yields a set of sample facial image similarities.

[0037] The Siamese neural network consists of two branches sharing weights. Each branch receives a different input (in this case, facial images) and processes these inputs using the same network architecture. The two branches calculate the similarity of the inputs using shared weights. A facial image similarity recognizer is then constructed using the Siamese neural network. The input data for the facial image similarity recognizer is two facial images, and the output data is the image similarity. The multiple sets of sample facial images and sample facial image similarity sets are then used as training data, which is divided into training, validation, and test sets according to a predetermined ratio (e.g., 80% training set, 15% validation set, and 5% test set). The facial image similarity recognizer is further supervised through training, validation, and testing using the training, validation, and test sets. First, in each training iteration, a pair of facial images is input into the two branches of the Siamese network for processing, and their similarity is calculated. Then, a ternary loss function is used to calculate the loss between the currently predicted similarity and the actual label. Finally, the network is updated using a backpropagation algorithm. Weights are assigned to minimize the loss function and improve model accuracy. During training, the training process is monitored using a validation set to ensure that the network does not overfit or underfit. After training, the model is evaluated using a test set to check whether its performance meets the expected requirements, such as the prediction accuracy of facial image similarity detection, ensuring the reliability and effectiveness of the model in practical applications. Training continues until the loss function is less than the set expected index, such as the loss value being lower than a certain threshold (e.g., 0.05). At this point, the model considers that it has achieved sufficient training effect and can stop training, resulting in a trained facial image similarity recognizer.

[0038] Next, a second periodic key facial image is selected within the periodic key facial image sequence. The second periodic key facial image is a periodic key facial image at an adjacent time point to the first time point (earliest timestamp). Then, the first periodic key facial image and the second periodic key facial image are input into the facial image similarity recognizer for similarity analysis, and the facial image similarity is output.

[0039] If the facial image similarity is greater than or equal to the image similarity threshold, the second period key facial image is deleted, and the third period key facial image is selected for similarity analysis. If not, the second period key facial image is retained as the key facial image, and similarity analysis and screening are continued based on the second period key facial image. This process continues until all period key facial images in the period key facial image sequence have been screened, resulting in a key facial image sequence.

[0040] Specifically, firstly, an image similarity threshold is set to determine whether facial images are similar enough to decide whether to filter out the current periodic key facial images. If the similarity of a facial image is greater than or equal to the threshold, it is considered that the image is similar to the previous image, which may be redundant and needs to be replaced. If the similarity is less than the threshold, it is considered that the current facial image has sufficient independence and can be retained as a key facial image.

[0041] Next, it is determined whether the facial image similarity is greater than or equal to the image similarity threshold. If so, the second-period key facial image is considered similar to the first-period key facial image and does not need to be used as a new key image. In this case, the second-period key facial image is deleted, and the third-period key facial image is selected for similarity analysis. If not, the second-period key facial image is considered valid and retained as a key facial image. The second-period key facial image is then retained as a key facial image, and similarity analysis and screening are continued based on the second-period key facial image.

[0042] The above steps are continued until all periodic key facial images have been filtered. Each time the image is filtered, the similarity between the current facial image and the previous key image is determined, and a decision is made on whether to replace the image based on the similarity value. This process continues until all periodic key facial images in the periodic key facial image sequence have been filtered, resulting in a key facial image sequence that contains the key moments of emotional changes and the most representative facial images throughout the entire production cycle.

[0043] This similarity analysis and screening process allows for the precise selection of key facial images in each production cycle, ensuring that these images effectively reflect changes in employee emotional fluctuations. This screening method avoids redundant image data, ensuring that the final key facial image sequence is representative and efficient, providing solid data support for subsequent emotion analysis and decision-making.

[0044] The speech signals of multiple key facial images in the key facial image sequence corresponding to the time frames are extracted to obtain the key speech sequence.

[0045] Specifically, the speech signals of multiple key facial images in the key facial image sequence are then extracted from the corresponding time frames. That is, based on the timestamp of each key facial image, speech signals near that time point are collected from the production environment to construct a key speech sequence.

[0046] The user emotion recognition module 12 is used to perform user emotion recognition based on the key facial image sequence and voice sequence to obtain a first user emotion level and a second user emotion level.

[0047] Furthermore, the user emotion recognition module 12 is also used for:

[0048] Based on user emotion test data, a set of sample key facial image sequences and a set of sample speech sequences are collected. The user's emotion level under different sample key facial image sequences and sample speech sequences is collected and labeled as the first sample user emotion level set and the second sample user emotion level set. Using the sample key facial image sequence set and the first sample user emotion level set as supervised training data, a facial emotion recognizer is trained using a convolutional neural network. The key facial image sequences are input into the facial emotion recognizer, and the first user emotion level is output. Similarly, using the sample speech sequence set and the second sample user emotion level set as supervised training data, a speech emotion recognizer is trained. The speech sequences are input into the speech emotion recognizer, and the second user emotion level is output.

[0049] Specifically, firstly, based on user emotion test data, a set of key facial image sequences and a set of sample speech sequences are collected. The emotional levels of users under different key facial image sequences are then collected and labeled as the first user's emotional level, thus obtaining the first user's emotional level set. In other words, the user's emotional state is assessed based on the collected facial images. Emotional levels are calibrated through expert annotation, automatic emotion recognition tools, or experimental design. For example, emotional levels can be set from 1 to 5, where level 1 represents extremely poor emotions (such as extreme anger or frustration), and level 5 represents extremely good emotions (such as extreme joy or relaxation). Secondly, the emotional levels of users under different sample speech sequences are collected and labeled as the second user's emotional level, thus obtaining the second user's emotional level set.

[0050] Then, a facial emotion recognizer is constructed based on a convolutional neural network. The facial emotion recognizer includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input data of the input layer is a sequence of key facial images. The convolutional layer is used to extract features from the facial images (such as facial structure, texture, and emotional representation). The pooling layer is used to reduce the dimensionality of the image features, reduce computational complexity, and extract more representative features. The fully connected layer is used to combine the features extracted by the convolutional and pooling layers. The output layer is used to output the emotion level (e.g., level 1 to 5). Then, the sample key facial image sequence set and the sample first user emotion level set are used as supervised training data to supervise the training of the facial emotion recognizer. First, the sample key facial image sequence set and the sample first user emotion level set are used as input and labels to construct a training dataset, with each input image sequence corresponding to an emotion level label. Next, an appropriate loss function, such as the cross-entropy loss function, is selected to calculate the error between the predicted emotion level and the actual emotion level. Optimization algorithms such as Adam optimizer and SGD are used to minimize the loss function and update the CNN weights to gradually improve the accuracy of facial emotion recognition. During training, the CNN calculates the emotion prediction result of the image through forward propagation and then adjusts the network weights through backpropagation until the loss function converges to the expected value, obtaining the trained facial emotion recognizer. Then, the key facial image sequence is input into the facial emotion recognizer, and the first user emotion level is output.

[0051] On the other hand, a speech emotion recognizer is constructed based on a convolutional neural network. The speech emotion recognizer includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. Its input data is a speech sequence, and its output data is the user's emotion level. Then, the sample speech sequence set and the sample second user emotion level set are used as supervised training data to train the speech emotion recognizer. The training method of the speech emotion recognizer is the same as the training method of the facial emotion recognizer mentioned above, and will not be elaborated here. The training continues until the loss function converges, and the trained speech emotion recognizer is obtained. Finally, the speech sequence is input into the speech emotion recognizer, and the second user emotion level is output.

[0052] By building facial and voice emotion recognizers based on convolutional neural networks, the intelligence of emotion analysis can be improved. It can automatically extract emotional features from facial images or voice data and accurately predict the emotional state of employees, providing reliable emotional data support for production management.

[0053] The weight calculation configuration module 13 is used to identify the environmental noise intensity based on the key speech sequence, obtain the environmental noise intensity sequence, and configure and obtain facial weights and speech weights according to the magnitude of each environmental noise intensity. The speech quality influence coefficient is obtained through speech quality influence analysis and the weight calculation configuration is performed.

[0054] Furthermore, the weight calculation configuration module 13 is also used for:

[0055] Environmental noise filtering is performed on multiple key speech signals within the key speech sequence to obtain multiple signal-to-noise ratios, and an environmental noise intensity sequence is calculated; the mean of the environmental noise intensity sequence is calculated to obtain the average environmental noise intensity.

[0056] Specifically, environmental noise filtering is performed on multiple key speech signals within the key speech sequence. Noise filtering techniques (such as spectral subtraction and Wiener filtering) are used to analyze the noise components in the speech signals, obtaining multiple signal-to-noise ratios (SNRs). SNR measures the ratio between useful information and noise in a signal. The SNR of the speech signal is calculated by comparing the power of the speech and noise components. Based on the noise proportion of each signal, the environmental noise intensity is obtained. This intensity value reflects the intensity of environmental noise at each moment; a higher SNR indicates greater noise intensity and a stronger impact from environmental noise, thus obtaining an environmental noise intensity sequence. Further, the average environmental noise intensity is calculated from the multiple environmental noise intensities in the environmental noise intensity sequence. The average environmental noise intensity represents the overall level of environmental noise throughout the entire production cycle. A higher average noise intensity indicates a persistent and strong noise interference in the production environment, which can significantly impact speech recognition and emotion analysis.

[0057] Based on the average ambient noise intensity, the speech quality impact coefficient is calculated, the preset speech weights are configured, the speech weights are obtained, and the facial weights are calculated.

[0058] Furthermore, the present invention also includes the following steps:

[0059] Calculate the ratio of the preset ambient noise intensity to the average ambient noise intensity to obtain the voice quality impact coefficient; multiply the voice quality impact coefficient by the preset voice weight to perform weight calculation configuration to obtain the voice weight; subtract the voice weight from 1 to obtain the face weight.

[0060] Specifically, a preset ambient noise intensity is obtained. This preset ambient noise intensity is a standard value representing an ideal or compliant noise level (e.g., meeting relevant noise pollution standards). This value is typically set according to regulations or work environment requirements. Next, the ratio of the preset ambient noise intensity to the average ambient noise intensity is calculated, and this ratio is set as the speech quality impact coefficient. This speech quality impact coefficient reflects the influence of noise intensity on speech quality. In high-noise environments, the speech quality impact coefficient will be lower, indicating that the accuracy of speech recognition may be compromised.

[0061] Further, a preset speech weight is obtained. This speech weight is typically an initial preset value used to express the importance of speech in emotion recognition, usually between 0 and 1, and can be set according to actual conditions, such as 0.3. Next, the speech quality influence coefficient is multiplied by the preset speech weight to perform weight calculation configuration, resulting in an updated speech weight. For example, when the noise intensity is low, the speech quality influence coefficient is close to 1, and the speech weight is close to the preset value; when the noise intensity is high, the speech quality influence coefficient decreases, and the speech weight will also decrease accordingly, thereby reducing the impact of noise on emotion analysis. Finally, 1 is subtracted from the speech weight, and the difference is set as the facial weight.

[0062] By adjusting the voice weight and facial weight, it is possible to dynamically adapt to different working environments, especially in noisy production environments. This increases the reliance on facial image data and reduces the impact of noise in the voice signal on emotion recognition, thereby ensuring the accuracy, stability, and precision of emotion recognition in complex environments.

[0063] The user emotion level acquisition module 14 is used to calculate the weighted user emotion level by weighting the first user emotion level and the second user emotion level according to the facial weight and voice weight, and then correct it using the voice quality influence coefficient to obtain the user emotion level for production management.

[0064] Furthermore, the user emotion level acquisition module 14 is also used for:

[0065] Based on the facial weight and voice weight, the first user's emotion level and the second user's emotion level are weighted and calculated to obtain a weighted user emotion level; it is determined whether the voice quality influence coefficient is less than 1. If so, the weighted user emotion level is multiplied by the voice quality influence coefficient to complete the user emotion level correction and obtain the user emotion level for production management; otherwise, no user emotion level correction is performed.

[0066] Specifically, firstly, based on the facial weight and voice weight, the first user's emotion level and the second user's emotion level are weighted and rounded to obtain a weighted user emotion level. Next, it is determined whether the voice quality impact coefficient is less than 1. If so, it indicates that the actual environmental noise is higher than a preset standard, the noise significantly interferes with the voice signal, and the voice quality is poor, requiring emotion level correction. In this case, the weighted user emotion level is multiplied by the voice quality impact coefficient and rounded to complete the user emotion level correction, obtaining the user emotion level. Production management is then based on the user emotion level. The emotion level reflects the employee's emotional state during production. Managers can adjust work arrangements and provide care based on the emotion level, thereby improving work efficiency and employee satisfaction.

[0067] If the voice quality impact coefficient is greater than or equal to 1, it indicates that the environmental noise intensity is small, the employees are basically not affected by the noise, and there is no need to perform noise correction. In this case, the weighted user emotion level is output as the user emotion level.

[0068] By combining the voice quality impact coefficient for emotion level correction, the emotion analysis results can be dynamically adjusted according to the impact of environmental noise on emotion recognition, thereby further improving the accuracy and reliability of user emotion level analysis and providing a more accurate decision-making basis for production management.

[0069] The production management sentiment analysis system based on multi-dimensional indicators provided in this invention has at least the following technical effects:

[0070] 1. By using multimodal data fusion, key feature extraction, and noise interference correction, data accuracy and analysis efficiency can be effectively improved, and changes in employee emotions in complex production environments can be perceived in real time and accurately. This significantly improves the efficiency, accuracy, and reliability of employee emotion assessment, providing scientific decision support for enterprise intelligent production management.

[0071] 2. By screening key facial images, redundant image data can be avoided, ensuring that the final key facial image sequence is representative and efficient, providing solid data support for subsequent emotion analysis and decision-making, and effectively improving data analysis efficiency, thereby improving the efficiency of employee emotion recognition.

[0072] 3. By constructing facial emotion recognizers and voice emotion recognizers based on convolutional neural networks, the intelligence of emotion analysis can be improved. It can automatically extract emotional features from facial images or voice data and accurately predict the emotional state of employees, providing reliable emotional data support for production management.

[0073] 4. By adjusting the voice weight and facial weight, it can dynamically adapt to different working environments, especially in noisy production environments. It increases the reliance on facial image data and reduces the impact of noise in the voice signal on emotion recognition, thereby ensuring the accuracy, stability and precision of emotion recognition in complex environments.

[0074] 5. By combining the voice quality impact coefficient for emotion level correction, the emotion analysis results can be dynamically adjusted according to the impact of environmental noise on emotion recognition, thereby further improving the accuracy and reliability of user emotion level analysis and providing a more accurate decision-making basis for production management.

[0075] Although preferred embodiments of the invention have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0076] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A production management sentiment analysis system based on multi-dimensional indicators, characterized in that, The system includes: The key information acquisition module is used to collect facial image sequences and voice sequences of users in the production space, extract key facial image sequences from the facial image sequences according to the production cycle, and index key voice sequences. The user emotion recognition module is used to perform user emotion recognition based on the key facial image sequence and voice sequence to obtain a first user emotion level and a second user emotion level. The weight calculation configuration module is used to identify the environmental noise intensity based on the key speech sequence, obtain the environmental noise intensity sequence, and configure and obtain facial weights and speech weights according to the magnitude of each environmental noise intensity. The speech quality influence coefficient is obtained through speech quality influence analysis and the weight calculation configuration is performed. The user emotion level acquisition module is used to calculate the weighted user emotion level by weighting the first user emotion level and the second user emotion level according to the facial weight and voice weight, and then correct it using the voice quality influence coefficient to obtain the user emotion level for production management.

2. The production management sentiment analysis system based on multi-dimensional indicators according to claim 1, characterized in that, Within the production space, facial image sequences and voice sequences of users are collected. Based on the production cycle, key facial image sequences are extracted from the facial image sequences, including: Within a continuous timeframe within the production space, facial images and voice signals of users are collected and arranged in chronological order to obtain facial image sequences and voice sequences. The production cycle within the production space is obtained, and the time frame between two adjacent production cycles is extracted as a key frame to obtain the cycle key frame sequence. Extract facial images from multiple periodic keyframes within the periodic keyframe sequence to obtain a periodic key facial image sequence; Similarity analysis and filtering are performed on adjacent periodic key facial images within the periodic key facial image sequence to obtain a key facial image sequence. The speech signals of multiple key facial images in the key facial image sequence corresponding to the time frames are extracted to obtain the key speech sequence.

3. The production management sentiment analysis system based on multi-dimensional indicators according to claim 2, characterized in that, Similarity analysis and filtering are performed on adjacent periodic key facial images within the periodic key facial image sequence to obtain a key facial image sequence, including: Select the first periodic key facial image within the periodic key facial image sequence and retain it as the key facial image; Within the periodic key facial image sequence, a second periodic key facial image is selected, and its similarity to the first periodic key facial image is identified to obtain facial image similarity. If the facial image similarity is greater than or equal to the image similarity threshold, the second period key facial image is deleted and the third period key facial image is selected for similarity analysis. If not, the second period key facial image is retained as the key facial image, and similarity analysis and screening are continued based on the second period key facial image. The process continues until all periodic key facial images within the periodic key facial image sequence have been filtered out, thus obtaining the key facial image sequence.

4. The production management sentiment analysis system based on multi-dimensional indicators according to claim 3, characterized in that, Within the periodic key facial image sequence, a second periodic key facial image is selected, and its similarity to the first periodic key facial image is identified to obtain facial image similarity, including: Based on facial image recognition data from historical time periods, multiple sets of sample facial images are collected, and the similarity between two sample facial images within each set of sample facial images is identified to obtain a set of sample facial image similarities. A facial image similarity recognizer is constructed using a Siamese neural network; The facial image similarity recognizer is trained, validated, and tested under supervised supervision using the multiple sets of sample facial images and the sample facial image similarity set until the loss is less than the requirement. The first period key facial image and the second period key facial image are input into the facial image similarity recognizer, and the facial image similarity is obtained by recognition output.

5. The production management sentiment analysis system based on multi-dimensional indicators according to claim 1, characterized in that, Based on the key facial image sequence and voice sequence, user emotion recognition is performed to obtain a first user emotion level and a second user emotion level, including: Based on user emotion test data, a set of key facial image sequences and a set of voice sequences were collected. The user's emotion level under different key facial image sequences and voice sequences were collected and labeled as the first set of user emotion levels and the second set of user emotion levels. The set of key facial image sequences and the set of first user emotion levels were used as supervised training data, and a facial emotion recognizer was trained using a convolutional neural network. The key facial image sequence is input into the facial emotion recognizer, and the first user emotion level is obtained from the output. The set of sample speech sequences and the set of sample second user emotion levels are used as supervised training data to train the speech emotion recognizer. The speech sequence is input into the speech emotion recognizer, and the output yields the second user emotion level.

6. The production management sentiment analysis system based on multi-dimensional indicators according to claim 1, characterized in that, Based on the key speech sequence, environmental noise intensity is identified to obtain an environmental noise intensity sequence. Then, based on the magnitude of each environmental noise intensity, facial weights and speech weights are configured and obtained, including: Environmental noise filtering is performed on multiple key speech signals within the key speech sequence to obtain multiple signal-to-noise ratios, and an environmental noise intensity sequence is calculated. Calculate the mean of the environmental noise intensity sequence to obtain the average environmental noise intensity; Based on the average ambient noise intensity, the speech quality impact coefficient is calculated, the preset speech weights are configured, the speech weights are obtained, and the facial weights are calculated.

7. The production management sentiment analysis system based on multi-dimensional indicators according to claim 6, characterized in that, Based on the average ambient noise intensity, a speech quality impact coefficient is calculated, and preset speech weights are configured to obtain speech weights. Facial weights are also calculated, including: Calculate the ratio of the preset ambient noise intensity to the average ambient noise intensity to obtain the voice quality impact coefficient; The speech quality impact coefficient is multiplied by a preset speech weight to calculate the speech weight. The facial weight is obtained by subtracting the voice weight from 1.

8. The production management sentiment analysis system based on multi-dimensional indicators according to claim 7, characterized in that, Based on the facial weights and voice weights, the first user's emotion level and the second user's emotion level are weighted and calculated to obtain a weighted user emotion level. The voice quality impact coefficient is then used for correction to obtain the final user emotion level, including: Based on the facial weight and voice weight, the first user's emotion level and the second user's emotion level are weighted and calculated to obtain the weighted user emotion level. If the voice quality impact coefficient is less than 1, the user emotion level is corrected by multiplying the weighted user emotion level by the voice quality impact coefficient to obtain the user emotion level for production management. If not, no user emotion level correction is performed.

Images