An artificial intelligence voice interaction system and method based on multi-modal fusion

Abstract

The application discloses a kind of artificial intelligence voice interaction system and method based on multi-modal fusion, it is related to artificial intelligence technical field, the key point information of abnormal construction action is extracted in the history interaction record, and the similarity score of history interaction record is calculated, based on the similarity score, the abnormal score of each construction procedure is calculated, the static abnormal grade corresponding to each construction procedure is determined, the combination of associated feature environment parameters is determined, the dynamic abnormal score is calculated, and the dynamic abnormal grade is determined;Behavior score is calculated based on behavior characteristic data, and behavior score prediction model is generated in combination with voice prompt feature data;Real-time behavior score is predicted based on the behavior score prediction model corresponding to real-time dynamic abnormal grade, to determine whether to carry out parameter adjustment, improve the response efficiency of construction personnel to prompt information.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an artificial intelligence voice interaction system and method based on multimodal fusion. Background Technology

[0002] With the development of artificial intelligence technology, voice interaction systems are gradually being applied to scenarios such as power equipment operation and maintenance, repair command and on-site assistance. They are used to realize functions such as job information query, operation guidance issuance and status feedback. During the on-site operation of power equipment, operators usually need to complete multiple operations continuously in complex environments and interact with the system through voice to obtain job guidance or issue operation instructions. Existing voice interaction systems are usually based on single recognition or short-term context processing and are mainly designed for static or short-term stable use environments. In actual operation, the noise level, operation stage and the operation behavior of the operator will continuously change with the operation process, which will cause existing systems to be prone to problems such as fluctuations in recognition accuracy and deviations in command understanding. In addition, voice interaction systems often use fixed parameter configurations or preset update strategies, lacking the ability to manage the overall operating status of the interaction process. It is difficult to adjust the system operation according to the changes in the status during the operation. In continuous operation scenarios, the above problems often accumulate over time, affecting the stability, continuity and security of the interaction process. Therefore, how to fully integrate multimodal information such as voice, behavior, environment and equipment status in complex work scenarios, comprehensively perceive and intelligently analyze the interaction process, and on this basis achieve stable operation and adaptive control of the voice interaction system to improve the accuracy, real-time performance and system reliability of the interaction has become a technical problem that current artificial intelligence voice interaction systems need to solve. Summary of the Invention

[0003] The purpose of this invention is to provide an artificial intelligence voice interaction system and method based on multimodal fusion to solve the problems raised in the prior art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: an artificial intelligence voice interaction method based on multimodal fusion, the method comprising: Acquire video information from power construction sites, identify the construction actions of construction workers, and generate interactive records when the construction actions are abnormal; Based on historical interaction records, a similarity score is calculated for abnormal construction actions, and the static abnormality level corresponding to the construction procedure is determined according to the similarity score. Obtain the environmental parameters corresponding to the interaction record, and calculate a comprehensive environmental anomaly score based on the abnormal conditions of the environmental parameters; Based on the similarity score, comprehensive environmental anomaly score, and static anomaly level, the dynamic anomaly level corresponding to the interaction record is determined; Collect voice prompt feature data and construction worker behavior feature data from interaction records, and construct a behavior scoring prediction model; Real-time construction information is obtained, the real-time dynamic anomaly level is determined, and a real-time behavior score is predicted based on the behavior score prediction model corresponding to the real-time dynamic anomaly level, so as to determine whether the voice prompt parameters should be adjusted.

[0005] Furthermore, generating an interaction log when the construction action is abnormal includes the following steps: By acquiring video information from the construction site through monitoring equipment, video segments of construction workers completing construction tasks are extracted; In the video segment, identify continuous image frames of construction actions, obtain a set of abnormal image frames for each construction action that poses a construction risk, and compare the similarity between each abnormal image frame and the continuous image frames. If the similarity is higher than a preset threshold, it is determined to be an abnormal construction action. Collect environmental parameters at the construction site and synchronize them with video segments in time to generate time-series perception data. When abnormal construction actions are detected, issue voice prompts and generate interactive records. By deploying various sensors and monitoring equipment at the power construction site and collecting and processing construction site videos according to the preset construction action analysis time interval, it is possible to continuously monitor the entire process of construction personnel completing various construction tasks and accurately extract the video segments corresponding to each construction task, thereby providing a reliable data foundation for subsequent construction action analysis and improving the completeness and accuracy of construction behavior recognition. By introducing visual human motion capture technology, continuous image frames are extracted for each construction action of the construction workers, and the similarity is calculated with standard construction action image frames. When the similarity of the actions is lower than a preset threshold, it can be judged as an abnormal construction action. This avoids the limitations of relying solely on human experience or a single rule for judgment, and can more objectively and accurately identify non-standard or dangerous construction behaviors, effectively reducing the misjudgment rate. By synchronizing video clips, environmental parameters, and construction actions collected at the construction site, time-series perception data is generated, enabling a one-to-one correspondence between construction behavior and on-site environmental conditions. This facilitates a comprehensive analysis of the causes of abnormal construction actions and helps improve the comprehensiveness and scientific nature of construction safety risk assessment.

[0006] Furthermore, determining the static anomaly level corresponding to the construction procedure includes the following steps: Collect continuous image frames of abnormal construction actions in a certain historical interaction record, extract the human joint coordinates of each frame of the continuous image frames of abnormal construction actions, and normalize the human joint coordinates. Construct action features through the normalized human joint coordinates. The action features include the relative displacement vector between joints, joint angles, and the rate of change of joint angles. The process involves obtaining action features corresponding to abnormal construction actions with historical construction risks, calculating average action features, and performing similarity calculations between the action features of the abnormal construction actions and the average action features to obtain the similarity of the abnormal construction actions. The similarity calculation uses cosine similarity to obtain the similarity result. Based on the importance of each abnormal construction action, corresponding weight values ​​are set, and the similarity results of each abnormal construction action are weighted and summed with the corresponding weight values. The weighted results are then averaged based on the total number of abnormal construction actions to obtain the similarity score of the historical interaction records. The similarity score of the historical interaction records of each construction process is calculated and normalized. The average similarity score is calculated by averaging the normalized similarity scores. At the same time, the total number of historical perception data in the construction process is calculated. The total number of historical interaction records is divided by the total number of historical perception data to calculate the proportion of the number. The average similarity score and the proportion of the number are weighted and summed to calculate the anomaly score of each construction process. Based on the mean and standard deviation of the abnormality scores of all construction procedures, and based on the preset standard deviation multiple, the construction procedures are divided into several static abnormality levels. By collecting and summarizing historical interaction records of the same construction process, continuous image frames are extracted from abnormal construction actions, and key point information of each frame is extracted. This allows for a comprehensive analysis of the specific manifestations of each abnormal action during construction. Based on the analysis of historical data and key points of abnormal actions, detailed data support can be provided for the standardized management of the construction process, thereby optimizing the safety management and quality control of the construction process. By calculating the similarity between key information of abnormal construction actions in historical interaction records and standard image frames, and using a similarity scoring formula, the similarity of abnormal construction actions can be quantified. Furthermore, a weighted algorithm is used to comprehensively evaluate the degree of abnormality in historical interaction records, ensuring quantitative analysis of abnormal behavior. This helps to accurately identify potential dangerous construction behaviors and provides a scientific basis for subsequent intervention measures. By normalizing the similarity score of the historical interaction records of each construction process, an anomaly score is obtained for each construction process. Furthermore, the static anomaly level is calculated by the standard deviation of the anomaly score. This provides construction managers with a clear risk level classification, making the safety management of the construction site more systematic and scientific, and providing customized response measures for different levels of anomalies.

[0007] Furthermore, the comprehensive environmental anomaly score is calculated, including the following steps: A training period of several consecutive days is set, and the interaction records within the training period are obtained. The interaction records are classified according to the static anomaly level corresponding to the construction process. In the set of interaction records of the static anomaly level, environmental parameters that exceed the preset threshold range are identified as abnormal environmental factors. The proportion of each abnormal environmental factor in the static anomaly level is counted, and factors that are higher than the proportion threshold are marked as characteristic environmental parameters of the static anomaly level. Interaction records with two or more abnormal environmental parameters are marked as composite interaction records. The characteristic environmental parameters that are abnormal environmental factors are combined to generate a combination of characteristic environmental parameters for composite interaction records. In the set of composite interaction records with each combination of characteristic environmental parameters, the parameter values ​​of the combination of characteristic environmental parameters in the composite interaction records are collected to construct the numerical sequence of the combination of characteristic environmental parameters. Based on the numerical sequence of the combination of the characteristic environmental parameters, the Pearson correlation coefficient of the two characteristic environmental parameters is calculated. When the absolute value of the Pearson correlation coefficient is greater than the preset correlation threshold, it is determined that there is a correlation between the two characteristic environmental parameters. The combination of characteristic environmental parameters with correlation is marked as the associated characteristic environmental parameter combination, and the dynamic anomaly level corresponding to the interaction record is determined. By using a continuous number of days as the training period, the interaction records within the training period are summarized and analyzed. The interaction records are then grouped according to the static anomaly level corresponding to the construction process. This results in a clear hierarchical structure for environmental data under different risk levels, which is conducive to mining the environmental characteristics corresponding to each anomaly level from historical data and improving the pertinence and effectiveness of environmental factor analysis. By setting threshold ranges for environmental parameters such as temperature, humidity, light intensity, noise decibels, and air particulate matter concentration, and marking environmental parameters that do not meet the threshold range as abnormal, the frequency and proportion of abnormal environmental factors are further statistically analyzed. This allows for the selection of representative characteristic environmental parameters at specific static anomaly levels, avoiding indiscriminate analysis of all environmental parameters and reducing the interference of environmental noise on the judgment results. By marking interactive records with two or more abnormal characteristic environmental parameters simultaneously as composite interactive records and performing combined analysis on abnormal environmental factors, it is possible to identify abnormal environmental patterns formed when multiple environmental factors work together. This helps to discover composite risk characteristics that are difficult to reflect by a single environmental parameter and improves the ability to understand complex construction environments.

[0008] Furthermore, determining the dynamic anomaly level corresponding to the interaction record includes the following steps: The values ​​of characteristic environmental parameters in the collected interaction records are compared with the threshold values ​​of the characteristic environmental parameters to obtain the absolute difference. The absolute difference is used as the environmental deviation index. The environmental deviation index of each characteristic environmental parameter is weighted and summed to obtain a comprehensive environmental anomaly score. Extract the similarity score corresponding to the construction action in the interaction record and the static anomaly level corresponding to the construction procedure. Assign values ​​to the static anomaly level, normalize the similarity score and the comprehensive environmental anomaly score, and then weight and fuse the assigned static anomaly level, the normalized similarity score and the comprehensive environmental anomaly score according to the corresponding weight coefficients to calculate the dynamic anomaly score of the construction procedure. Based on the mean and standard deviation of the dynamic anomaly scores of all interaction records, and based on a preset standard deviation multiple, the interaction records are divided into several dynamic anomaly levels. By comparing the actual values ​​of characteristic environmental parameters with the corresponding threshold ranges, and calculating the absolute difference as an environmental deviation index based on the direction of exceeding the threshold, environmental anomalies are transformed from qualitative judgments of whether limits are exceeded into measurable numerical indicators. At the same time, the environmental deviation indices of multiple characteristic environmental parameters are weighted and summed to obtain a comprehensive environmental anomaly score, which can comprehensively reflect the overall degree of anomaly in the environmental conditions of the interactive record and improve the level of refinement of environmental anomaly assessment. By simultaneously introducing construction action similarity scores, comprehensive environmental anomaly scores, and static anomaly levels corresponding to construction procedures, and by normalizing and weighting each indicator, a dynamic anomaly score is calculated. This achieves a unified quantitative assessment of behavioral anomalies, environmental anomalies, and procedure risk levels, avoiding the one-sidedness of anomaly judgment caused by a single indicator, and improving the accuracy and reliability of dynamic anomaly identification. By normalizing the similarity score and the comprehensive environmental anomaly score, anomaly indicators with different dimensions and value ranges can be integrated and calculated under the same evaluation system. This effectively eliminates the impact of differences in indicator scales on the dynamic anomaly score results and improves the stability and universality of the dynamic anomaly score in different construction scenarios and time periods.

[0009] Furthermore, a behavioral rating prediction model is constructed, including the following steps: The response time is determined based on the urgency level label of the voice prompt. Within the response time, voice volume level, speech rate feature data, as well as the construction worker's reaction time and the duration of abnormal actions are extracted. Based on the ratio between the construction worker's reaction time and a preset reaction time threshold, a first score reflecting reaction speed is calculated. Based on the ratio between the duration of abnormal actions and the response time, a second score reflecting the duration of abnormal behavior is calculated. The first score and the second score are weighted and fused according to their respective weighting coefficients to obtain the behavior score. The time point at which the abnormal construction action stopped is obtained. The time period between the issuance time of the first round of prompt information and the time point at which the abnormal construction action stopped is used as the evaluation period. The voice prompt feature data within the evaluation period is assigned values. The input sequence is constructed in chronological order based on the differences in voice prompt feature data within multiple adjacent response durations, and the output sequence is constructed based on the behavioral score differences between corresponding adjacent response durations. Based on the input sequence and the output sequence, a long short-term memory network model is trained. The training process involves retaining and updating the historical voice prompt feature data in the input sequence through the network's memory units, and combining it with the voice prompt feature data of the current response duration to predict the trend of behavioral score changes. By minimizing the error between the predicted output and the actual behavioral score difference, the parameters of the long short-term memory network model are iteratively optimized to generate a behavioral score prediction model. By determining the corresponding urgency level label based on the prompt information and using the time difference between the time the prompt information is issued and the time of the next prompt information as the response duration, the response time window of construction personnel under a specific level of abnormality can be accurately defined. Furthermore, by combining behavioral characteristic data such as reaction time and duration of abnormal construction actions, a behavioral scoring model can be constructed, which transforms the qualitative description of construction personnel's response to abnormal prompts into a quantifiable evaluation result, thereby improving the objectivity and comparability of behavioral assessment. In the process of calculating behavioral scores, joint analysis of voice prompt feature data and construction worker behavioral feature data can reflect the impact of different prompt intensities and rhythms on construction workers' behavioral responses, avoid the information gaps caused by evaluating solely based on behavioral results, and enable behavioral scores to more realistically reflect the response status of construction workers in abnormal situations. By simultaneously incorporating reaction time and duration of abnormal construction actions into the behavioral scoring formula, and normalizing the result by combining it with response time, the scoring results can reflect both the speed of construction personnel's response to prompts and the efficiency of eliminating abnormal behavior. This effectively suppresses the safety risks caused by delayed response or excessively long duration of abnormal behavior, and improves the sensitivity of behavioral scoring to the effectiveness of abnormal handling. By constructing an input sequence using the differences in voice prompt feature data within multiple adjacent response durations and an output sequence using the corresponding behavioral score differences, a long short-term memory network model is introduced to model the temporal correlation between voice prompts and behavioral responses. This approach can fully utilize the influence of historical voice prompt features on changes in construction worker behavior, enabling the prediction of behavioral score change trends and overcoming the shortcomings of traditional static evaluation methods that cannot reflect the behavioral evolution process.

[0010] Further, determine whether to adjust the voice prompt parameters, including the following steps: The static anomaly level is matched with the real-time construction procedure, and the real-time comprehensive environmental anomaly score is calculated based on the real-time values ​​of characteristic environmental parameters. The real-time dynamic anomaly score is calculated by combining the real-time similarity score and the static anomaly level. The real-time dynamic anomaly level is determined, and the real-time speech feature data is input into the corresponding model to predict the behavior score. When the score is lower than the threshold, the speech volume or speech rate parameter is automatically adjusted. Obtain the behavior score prediction model corresponding to the real-time dynamic anomaly level, input the voice prompt feature data into the behavior score prediction model, and predict the real-time behavior score. If the real-time behavior score is lower than the behavior score threshold, the remote monitoring platform automatically adjusts at least one voice prompt feature parameter. If the real-time behavior score is higher than or equal to the behavior score threshold, the current voice prompt feature data is output. By acquiring real-time construction videos and determining the corresponding real-time construction procedures, and combining the static anomaly level corresponding to the construction procedure, the construction process is continuously monitored; at the same time, real-time values ​​of characteristic environmental parameters are collected and analyzed, enabling the system to perceive environmental changes in real time during construction, avoiding the lag problem of traditional post-event analysis methods, and improving the real-time performance of construction anomaly identification. When calculating the real-time environmental deviation index, the Pearson correlation coefficient between characteristic environmental parameters is introduced to jointly correct the degree of anomaly of the associated characteristic environmental parameters. This enables the environmental anomaly assessment results to reflect the coupling relationship between multiple parameters, avoid misjudgment caused by fluctuations of a single environmental parameter, and thus improve the accuracy and reliability of the real-time comprehensive environmental anomaly score. By normalizing and fusing the real-time similarity score of abnormal construction actions with the real-time comprehensive environmental anomaly score, a real-time dynamic anomaly score is calculated and the real-time dynamic anomaly level is determined, thereby achieving a comprehensive assessment of the degree of anomaly in construction behavior. This can simultaneously reflect the superimposed impact of construction action deviations and environmental risk factors, enhancing the comprehensiveness of anomaly judgment results. By calling the corresponding behavior scoring prediction model according to the real-time dynamic anomaly level and inputting voice prompt feature data to predict real-time behavior scores, the system can evaluate the response effect of construction personnel before their actual response behavior occurs, thereby transforming passive alarms into proactive intervention based on prediction results and improving the foresight of anomaly handling. When the predicted real-time behavior score is lower than the behavior score threshold, the remote monitoring platform automatically adjusts the voice prompt feature parameters; when the prediction result meets the threshold requirements, the original voice prompt strategy is maintained, thereby realizing dynamic adaptive adjustment of voice prompt intensity, speech rate or frequency, so that the prompting method matches the actual response ability of construction personnel, and avoids the problems of insufficient prompting or excessive intervention. By linking real-time anomaly identification, dynamic level determination, behavior prediction, and adaptive adjustment of voice prompts in a closed loop, a complete "monitoring-evaluation-prediction-intervention" control mechanism is formed, which helps to reduce the duration of abnormal construction and the probability of safety accidents, and significantly improves the intelligent, refined, and automated safety management capabilities of construction sites.

[0011] To better implement the above method, an artificial intelligence voice interaction system based on multimodal fusion is proposed. The system includes an interaction recording module, a static anomaly level module, a correlation parameter module, a dynamic anomaly level module, a behavior scoring prediction model module, and a judgment adjustment module. Interactive recording module: acquires video information from the power construction site, identifies the construction actions of the construction workers, and generates interactive records when the construction actions are abnormal; Static anomaly level module: Based on historical interaction records, calculate the similarity score of abnormal construction actions, and determine the static anomaly level corresponding to the construction procedure based on the similarity score; Association parameter module: Obtains environmental parameters corresponding to the interaction record, and calculates a comprehensive environmental anomaly score based on the abnormal conditions of the environmental parameters; Dynamic anomaly level module: Based on the similarity score, comprehensive environmental anomaly score, and static anomaly level, determine the dynamic anomaly level corresponding to the interaction record; Behavior scoring prediction model module: Collect voice prompt feature data and construction personnel behavior feature data from interaction records to build a behavior scoring prediction model; Judgment and Adjustment Module: Obtains real-time construction information, determines the real-time dynamic anomaly level, and predicts the real-time behavior score based on the behavior score prediction model corresponding to the real-time dynamic anomaly level, so as to determine whether to adjust the voice prompt parameters.

[0012] Furthermore, the interaction recording module includes a unit for determining abnormal construction actions and a unit for generating interaction records: Abnormal construction action determination unit: The video information of the construction site is obtained through monitoring equipment, and the video segments of the construction work completed by the construction personnel are extracted. The continuous image frames of the construction action are identified in the video segments, and the abnormal image frame set of each construction action with construction risk is obtained. The similarity of each abnormal image frame is compared with the continuous image frames. If the similarity is higher than the preset threshold, it is determined to be an abnormal construction action. Interactive recording unit: Collects environmental parameters of the construction site and synchronizes them with video segments in time to generate time-series perception data. When abnormal construction actions are detected, it issues voice prompts and generates interactive records.

[0013] Furthermore, the dynamic anomaly level module includes a unit for calculating dynamic anomaly scores and a unit for determining dynamic anomaly levels: The dynamic anomaly scoring unit is calculated as follows: Values ​​of characteristic environmental parameters in the interaction records are collected, and their absolute differences are calculated with the threshold values ​​of the characteristic environmental parameters. These absolute differences are used as environmental deviation indicators. The environmental deviation indicators of each characteristic environmental parameter are weighted and summed to obtain a comprehensive environmental anomaly score. The similarity scores corresponding to construction actions and the static anomaly levels corresponding to construction procedures in the interaction records are extracted. The static anomaly levels are assigned values, and the similarity scores and comprehensive environmental anomaly scores are normalized. The assigned static anomaly levels, the normalized similarity scores, and the comprehensive environmental anomaly scores are weighted and fused according to their respective weighting coefficients to calculate the dynamic anomaly score of the construction procedure. Determine the dynamic anomaly level unit: Based on the mean and standard deviation of the dynamic anomaly scores of all interaction records, and using a preset standard deviation multiple, divide the interaction records into several dynamic anomaly levels.

[0014] Compared with the prior art, the beneficial effects of the present invention are: it integrates construction video, key points of human body movements, environmental parameters and voice interaction data at the same time, avoids the misjudgment problem caused by relying solely on a single visual or environmental threshold judgment, realizes multi-dimensional and collaborative recognition of abnormal construction behavior, and significantly improves the accuracy and stability of anomaly detection. By constructing static anomaly levels for construction procedures based on historical data, and further combining real-time construction action similarity and environmental deviation to calculate dynamic anomaly scores, continuous assessment from "long-term risk characteristics" to "real-time risk status" is achieved, overcoming the shortcomings of fixed anomaly levels and delayed response in existing technologies. By conducting correlation analysis on characteristic environmental parameters, and only jointly evaluating combinations of environmental parameters that have significant correlations, the interference caused by isolated environmental fluctuations is effectively reduced, and the ability of the comprehensive environmental anomaly score to represent real risks is improved. A behavior scoring model is built based on the actual response behavior of construction workers, and the voice prompt parameters are dynamically adjusted using the prediction results, so that the voice interaction is transformed from "passive reminder" to "adaptive guidance", which significantly improves the response efficiency of construction workers to prompt information. By modeling the characteristics of voice prompts and the changing trends of behavioral scores using a long short-term memory network, the system can predict the behavioral responses of construction workers and adjust parameters before abnormal risks escalate, thereby enhancing the system's proactive early warning and risk prevention capabilities. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an artificial intelligence voice interaction method based on multimodal fusion according to the present invention. Figure 2 This is a schematic diagram of the structure of an artificial intelligence voice interaction system based on multimodal fusion according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 and Figure 2 This invention provides a technical solution: an artificial intelligence voice interaction method based on multimodal fusion, the method comprising: Acquire video information from power construction sites, identify the construction actions of construction workers, and generate interactive records when the construction actions are abnormal; The process of generating an interaction log when the construction action is abnormal includes the following steps: By acquiring video information from the construction site through monitoring equipment, video segments of construction workers completing construction tasks are extracted; In the video segment, identify continuous image frames of construction actions, obtain a set of abnormal image frames for each construction action that poses a construction risk, and compare the similarity between each abnormal image frame and the continuous image frames. If the similarity is higher than a preset threshold, it is determined to be an abnormal construction action. Collect environmental parameters at the construction site and synchronize them with video segments in time to generate time-series perception data. When abnormal construction actions are detected, issue voice prompts and generate interactive records. For example, in step S101: several types of sensors and monitoring equipment are deployed at the power construction site, the analysis time interval of the construction action is preset, and video information of the construction site is obtained through the monitoring equipment according to the analysis time interval. The video information is identified by visual recognition technology, and the video segment corresponding to each construction operation completed by the construction personnel is extracted. Step S102: In the video segments corresponding to each construction operation, the continuous image frames corresponding to each construction action of the construction worker are identified and extracted by visual human motion capture technology. The continuous image frames are compared with the standard image frames of the construction actions to calculate the action similarity. When the action similarity is lower than the preset action similarity threshold, the construction action is judged as an abnormal construction action. Step S103: Collect environmental parameters of the construction site corresponding to the video segment through sensors, perform time synchronization processing on the video segment, environmental parameters, and construction actions to generate time-series perception data, and when abnormal construction actions are identified, issue prompt information to the construction personnel through a voice interaction device, set the perception data with voice interaction as an interaction record, and upload the interaction record to the remote monitoring platform.

[0018] Based on historical interaction records, a similarity score is calculated for abnormal construction actions, and the static abnormality level corresponding to the construction procedure is determined according to the similarity score. Determining the static anomaly level corresponding to a construction procedure includes the following steps: Collect continuous image frames of abnormal construction actions in a certain historical interaction record, extract the human joint coordinates of each frame of the continuous image frames of abnormal construction actions, and normalize the human joint coordinates. Construct action features through the normalized human joint coordinates. The action features include the relative displacement vector between joints, joint angles, and the rate of change of joint angles. The process involves obtaining action features corresponding to abnormal construction actions with historical construction risks, calculating average action features, and performing similarity calculations between the action features of the abnormal construction actions and the average action features to obtain the similarity of the abnormal construction actions. The similarity calculation uses cosine similarity to obtain the similarity result. Based on the importance of each abnormal construction action, corresponding weight values ​​are set, and the similarity results of each abnormal construction action are weighted and summed with the corresponding weight values. The weighted results are then averaged based on the total number of abnormal construction actions to obtain the similarity score of the historical interaction records. The similarity score of the historical interaction records of each construction process is calculated and normalized. The average similarity score is calculated by averaging the normalized similarity scores. At the same time, the total number of historical perception data in the construction process is calculated. The total number of historical interaction records is divided by the total number of historical perception data to calculate the proportion of the number. The average similarity score and the proportion of the number are weighted and summed to calculate the anomaly score of each construction process. Based on the mean and standard deviation of the abnormality scores of all construction procedures, and based on the preset standard deviation multiple, the construction procedures are divided into several static abnormality levels. For example, step S201: Obtain the construction procedures of power construction, collect the construction procedures corresponding to the historical interaction records, summarize the historical interaction records belonging to the same construction procedure, collect the continuous image frames of abnormal construction actions in a certain historical interaction record in the historical interaction record set of a certain construction procedure, and extract the key point information of each frame in the continuous image frames of abnormal construction actions, wherein the key point information is human joints. Step S202: Calculate the similarity between the key point information and the key point information corresponding to the standard image frame to obtain the similarity of the abnormal construction action. Calculate the similarity score of the historical interaction records according to the following formula: ; Where D represents the similarity score, En represents the similarity of the nth abnormal construction action, Fn represents the preset weight of the nth abnormal construction action, and m represents the total number of abnormal construction actions. Step S203: Calculate the similarity score of historical interaction records for each construction process, normalize the similarity score, and calculate the anomaly score for each construction process according to the following formula: ; Where A represents the anomaly score of the construction process, Ba represents the similarity score of the a-th historical interaction record after normalization, b represents the total number of historical interaction records in the construction process, b' represents the total number of historical perception data in the construction process, and C1 and C2 represent the weights of the similarity score and the total number, respectively. Step S204: Summarize the anomaly scores of all construction procedures, calculate the average score as e1 and the standard deviation as e2, preset the number of static anomaly levels as k1, and set the corresponding level coefficients for each static anomaly level as {d1, d2, ..., dk1}. According to the formula Gi=e1+di×e2, where Gi represents the anomaly score threshold corresponding to the i-th static anomaly level, and di represents the level coefficient corresponding to the i-th static anomaly level, the anomaly score range of each static anomaly level is determined based on the anomaly score threshold, and the static anomaly level corresponding to each construction procedure is determined. For example, if the construction procedure is to install insulators on the pole, the system retrieves historical interaction records of the same construction procedure from the remote monitoring platform within the past 30 days. There are a total of 12 historical interaction records, of which 5 were identified as containing abnormal construction actions, as detailed below: Record 01 shows the abnormal movement type as excessive forward leaning; Record 03 shows the abnormal action type as single-handed support operation; Record 05 shows an abnormal movement type: inconsistent hand height; Record 07 shows the abnormal movement type as excessive lumbar twisting; Record 09 shows the abnormal action type as center of gravity shift; Taking excessive forward leaning as an example of abnormal movement, the following is an example of extracting human joint coordinates: In record A, the neck is (350, 205), the hip is (305, 265), and the knee is (300, 355). Normalization is performed with the hip joint as the origin to obtain the normalized joint coordinates: The neck is (350−305, 205−265) = (45, -60), the hip is (0, 0), and the knee is (300−305, 355−265) = (-5, 90). The following motion characteristics are calculated using joint coordinates: The trunk vector is the relative displacement vector from the neck to the hip: (45, -60); The forward lean angle is the angle between the trunk vector and the vertical direction: arctan(45 / 60) = 36.87; Assuming the joint angle change rate is 11° in the previous frame and the inter-frame time interval is 0.1 seconds: (36.87-11) / 0.1=258.7; The action feature vector of A is recorded as: [45, -60, 36.87, 258.7]; Assume that the average action feature vector corresponding to the historical abnormal construction actions is [60, -60, 45, 350]; The feature vector magnitudes of record A and the history are respectively: The magnitude of the feature vector of A is recorded as: ; The feature vector length for recording history is: ; The dot product of the eigenvectors is calculated as follows: 60×45+(-60)×(-60)+45×36.87+350×258.7=17621.85; The calculated cosine similarity is: 17621.85 / 353.35*267.56=0.94, which gives the similarity of the abnormal actions in record A as 0.94. By comparing the above key points with the standard action template, the similarity of the abnormal actions in record 01 was calculated to be 0.72, with a weight of 1.2. The similarity of the abnormal actions in record 03 is 0.68, with a weight of 1.3; The similarity of the abnormal actions in record 05 is 0.75, with a weight of 1.1; The similarity of the abnormal actions in record 07 is 0.70, with a weight of 1.2; The similarity of the abnormal actions in record 09 is 0.73, with a weight of 1.2; The calculated similarity score is (0.72×1.2+0.68×1.3+0.75×1.1+0.70×1.2+0.73×1.2) / 5=0.86; Assuming the normalized similarity scores of the 12 historical interaction records are [0.21, 0.35, 0.18, 0.42, 0.30, 0.25, 0.38, 0.19, 0.41, 0.27, 0.22, 0.33], the total number of historical perception data is 38, the total number of historical interaction records is 12, and the weights for the similarity scores and the total number are 0.7 and 0.3 respectively, the anomaly score is calculated as follows: (0.21+0.35+0.18+0.42+0.30+0.25+0.38+0.19+0.41+0.27+0.22+0.33) / 12×0.7+12 / 38×0.3=0.293×0.7+0.316×0.3=0.21+0.09=0.3.

[0019] Obtain the environmental parameters corresponding to the interaction record, and calculate a comprehensive environmental anomaly score based on the abnormal conditions of the environmental parameters; The calculation of the comprehensive environmental anomaly score includes the following steps: A training period of several consecutive days is set, and the interaction records within the training period are obtained. The interaction records are classified according to the static anomaly level corresponding to the construction process. In the set of interaction records of the static anomaly level, environmental parameters that exceed the preset threshold range are identified as abnormal environmental factors. The proportion of each abnormal environmental factor in the static anomaly level is counted, and factors that are higher than the proportion threshold are marked as characteristic environmental parameters of the static anomaly level. Interaction records with two or more abnormal environmental parameters are marked as composite interaction records. The characteristic environmental parameters that are abnormal environmental factors are combined to generate a combination of characteristic environmental parameters for composite interaction records. In the set of composite interaction records with each combination of characteristic environmental parameters, the parameter values ​​of the combination of characteristic environmental parameters in the composite interaction records are collected to construct the numerical sequence of the combination of characteristic environmental parameters. Based on the numerical sequence of the combination of the characteristic environmental parameters, the Pearson correlation coefficient of the two characteristic environmental parameters is calculated. When the absolute value of the Pearson correlation coefficient is greater than the preset correlation threshold, it is determined that there is a correlation between the two characteristic environmental parameters. The combination of characteristic environmental parameters with correlation is marked as the associated characteristic environmental parameter combination, and the dynamic anomaly level corresponding to the interaction record is determined. For example, in step S301: set a continuous number of days as the training period, obtain the interaction records within the training period, collect the construction procedures corresponding to each interaction record, determine the static anomaly level corresponding to the interaction record based on the static anomaly level corresponding to the construction procedure, and summarize the interaction records belonging to the same static anomaly level. Step S302: In the set of interaction records for a certain static anomaly level, extract the environmental parameters collected by environmental sensors from each interaction record. The environmental parameters include temperature, humidity, light intensity, noise level (decibels), and air particulate matter concentration. A threshold range is preset for each environmental parameter. When an environmental parameter does not meet the corresponding threshold range, the environmental parameter is marked as an abnormal environmental factor. The frequency of occurrence of each abnormal environmental factor in the static anomaly level is counted, and the proportion of occurrence of each abnormal environmental factor in the static anomaly level is calculated. Abnormal environmental factors with an occurrence proportion higher than a preset occurrence proportion threshold are marked as characteristic environmental parameters of the static anomaly level. Step S303: Mark the interaction record with two or more feature environment parameters as abnormal environment factors as a composite interaction record, combine the feature environment parameters as abnormal environment factors to generate feature environment parameter combinations of composite interaction records, collect the parameter values ​​of feature environment parameter combinations in the composite interaction record in each set of composite interaction records with feature environment parameter combinations, and construct the numerical sequence of the feature environment parameter combinations. Step S304: Based on the numerical sequence of the combination of the characteristic environmental parameters, calculate the Pearson correlation coefficient of the two characteristic environmental parameters according to the following formula: ; Where H represents the Pearson correlation coefficient of the two feature environment parameters, Xp and Yp are the parameter values ​​of the two feature environment parameters in the p-th composite interaction record, q is the total number of composite interaction records, and X and Y are the average values ​​of the two feature environment parameters in all composite interaction records. When the absolute value of the Pearson correlation coefficient is greater than the preset correlation threshold, it is determined that there is a correlation between the two feature environment parameters, and the feature environment parameter combination with correlation is marked as the associated feature environment parameter combination. For example, suppose the training period is set to 10 days, and 30 interaction records of the insulator installation process on the pole are collected during this period, with the following threshold range set for each environmental parameter: Temperature: 10 to 35 degrees Celsius; Humidity: 30 to 80; Light intensity: 200 to 1200; Noise level (dB): 45 to 75; Air particulate matter concentration: 0 to 50; The environmental parameter data of some interaction records are shown below: Record 01: Temperature 36°C, Humidity 78%, Illumination 1100%, Noise 72%, Airborne Particulate Matter 45%, Abnormal environmental factor is excessive temperature; Record 02: Temperature 28°C, Humidity 85%, Illumination 95%, Noise 55%, Air Particulate Matter 60%. Abnormal environmental factors are excessive humidity and excessive air particulate matter. Record 03: Temperature 33°C, Humidity 62%, Illumination 1200°C, Noise 80°C, Airborne Particulate Matter 50°C. The abnormal environmental factor is excessive noise. Based on the statistical analysis of the abnormal environmental factors recorded, the following frequencies and proportions were obtained: The frequency of temperature exceeding the standard was 4 times, with an occurrence rate of 4 / 30 = 13.3%. The frequency of humidity exceeding the standard was 6, with an occurrence rate of 6 / 30 = 20.0%. The frequency of excessive light exposure was 2, with an occurrence rate of 2 / 30 = 6.7%. The frequency of noise exceeding the standard was 5 times, and the occurrence rate was 5 / 30 = 16.7%. The frequency of exceeding the standard for air particulate matter was 3, with an occurrence rate of 3 / 30 = 10.0%. By setting a threshold of 15% for the occurrence rate, excessive humidity and excessive noise are marked as characteristic environmental factors. The following are some records marked as composite interaction records: Record 02 shows that the combination of environmental factors is excessive humidity and excessive air particulate matter. Record 04 shows that the combination of characteristic environmental factors is excessive humidity and excessive noise. Record 05 shows that the combination of environmental factors is excessive humidity and excessive air particulate matter.

[0020] Based on the similarity score, comprehensive environmental anomaly score, and static anomaly level, the dynamic anomaly level corresponding to the interaction record is determined; Determining the dynamic anomaly level corresponding to the interaction record includes the following steps: The values ​​of characteristic environmental parameters in the collected interaction records are compared with the threshold values ​​of the characteristic environmental parameters to obtain the absolute difference. The absolute difference is used as the environmental deviation index. The environmental deviation index of each characteristic environmental parameter is weighted and summed to obtain a comprehensive environmental anomaly score. Extract the similarity score corresponding to the construction action in the interaction record and the static anomaly level corresponding to the construction procedure. Assign values ​​to the static anomaly level, normalize the similarity score and the comprehensive environmental anomaly score, and then weight and fuse the assigned static anomaly level, the normalized similarity score and the comprehensive environmental anomaly score according to the corresponding weight coefficients to calculate the dynamic anomaly score of the construction procedure. Based on the mean and standard deviation of the dynamic anomaly scores of all interaction records, and based on a preset standard deviation multiple, the interaction records are divided into several dynamic anomaly levels. For example, in step S401: collect the values ​​of characteristic environmental parameters in the interaction record. If the values ​​are lower than the threshold of the characteristic environmental parameter, calculate the absolute difference between the value and the minimum value of the threshold of the characteristic environmental parameter. If the values ​​are higher than the threshold of the characteristic environmental parameter, calculate the absolute difference between the value and the maximum value of the threshold of the characteristic environmental parameter. Use the absolute difference as the environmental deviation index. Sum the environmental deviation indexes of each characteristic environmental parameter by weight to obtain a comprehensive environmental anomaly score. Step S402: Extract the similarity scores corresponding to construction actions and the static anomaly levels corresponding to construction procedures from the interaction records. Assign values ​​to the static anomaly levels, normalize the similarity scores and the overall environmental anomaly scores, and calculate the dynamic anomaly scores according to the following formula: ; Where M represents the dynamic anomaly score, S represents the normalized similarity score, W represents the normalized comprehensive environmental anomaly score, V represents the static anomaly level assignment corresponding to the construction procedure, and r1, r2, and r3 are the preset weight coefficients of the similarity score, comprehensive environmental anomaly score, and static anomaly level, respectively, and satisfy r1+r2+r3=1. Step S403: Summarize the dynamic anomaly scores of all interaction records, calculate the average value of the dynamic anomaly scores as e3 and the standard deviation as e4, preset the number of dynamic anomaly levels as k2, and set the corresponding level coefficients for each dynamic anomaly level as {u1, u2, ..., uk2}. According to the formula Rz=e3+uz×e4, where Rz represents the dynamic anomaly score threshold corresponding to the z-th dynamic anomaly level, uz represents the level coefficient corresponding to the z-th dynamic anomaly level, and determine the dynamic anomaly score range of each dynamic anomaly level based on the dynamic anomaly score threshold, and determine the dynamic anomaly level corresponding to each interaction record. For example, record the humidity as 88 and the noise as 82 for R1; Recorded humidity level for R2 as 85% and noise level as 78%. The recorded humidity level for R3 was 76%, and the noise level was 81%. The preset humidity weight is 0.6, and the noise weight is 0.4; The recorded humidity deviation of R1 is |88-80|=8, and the noise deviation is |82-75|=7. The calculated comprehensive environmental anomaly score is 8×0.6+7×0.4=4.8+2.8=7.6. The recorded humidity deviation for R2 is |85-80|=5, and the noise deviation is |78-75|=3. The calculated comprehensive environmental anomaly score is 0.6×5+0.4×3=3+1.2=4.2. The recorded humidity deviation of R3 is 0, which is within the range. The noise deviation is |81-75|=6. The calculated comprehensive environmental anomaly score is 0.6×0+0.4×6=2.4.

[0021] Collect voice prompt feature data and construction worker behavior feature data from interaction records, and construct a behavior scoring prediction model; The construction of the behavior rating prediction model includes the following steps: The response time is determined based on the urgency level label of the voice prompt. Within the response time, voice volume level, speech rate feature data, as well as the construction worker's reaction time and the duration of abnormal actions are extracted. Based on the ratio between the construction worker's reaction time and a preset reaction time threshold, a first score reflecting reaction speed is calculated. Based on the ratio between the duration of abnormal actions and the response time, a second score reflecting the duration of abnormal behavior is calculated. The first score and the second score are weighted and fused according to their respective weighting coefficients to obtain the behavior score. The time point at which the abnormal construction action stopped is obtained. The time period between the issuance time of the first round of prompt information and the time point at which the abnormal construction action stopped is used as the evaluation period. The voice prompt feature data within the evaluation period is assigned values. The input sequence is constructed in chronological order based on the differences in voice prompt feature data within multiple adjacent response durations, and the output sequence is constructed based on the behavioral score differences between corresponding adjacent response durations. Based on the input sequence and the output sequence, a long short-term memory network model is trained. The training process involves retaining and updating the historical voice prompt feature data in the input sequence through the network's memory units, and combining it with the voice prompt feature data of the current response duration to predict the trend of behavioral score changes. By minimizing the error between the predicted output and the actual behavioral score difference, the parameters of the long short-term memory network model are iteratively optimized to generate a behavioral score prediction model. For example, in step S501: In an interaction record of a certain dynamic abnormality level, collect the prompt information in a certain interaction record, determine the urgency label corresponding to the prompt information according to the instruction identifier corresponding to the prompt information in the preset voice library, take the time of the prompt information as the start time, take the time of the next round of prompt information as the end time, and take the time difference between the end time and the start time as the response duration. Step S502: Extract voice prompt feature data and construction worker behavior feature data within the response time. The voice prompt feature data includes voice volume level and voice speed. The behavior feature data includes the reaction time from the issuance of the prompt information to the cessation of the abnormal construction action and the duration of the abnormal construction action. Calculate the behavior score according to the following formula: ; Where Q represents the behavior score, t1 represents the reaction time, t2 represents the preset reaction time threshold, t3 represents the duration, t represents the response time, and α1 and α2 are the weighting coefficients for reaction time and duration, respectively. Step S503: Obtain the time point when the abnormal construction action stops. Take the time period between the issuance time of the first round of prompt information and the time point when the abnormal construction action stops as the evaluation period. Assign values ​​to the voice prompt feature data within the evaluation period. Construct an input sequence in chronological order using the differences in voice prompt feature data within multiple adjacent response durations. Construct an output sequence using the behavioral score differences between corresponding adjacent response durations. Based on the input sequence and output sequence, train a long short-term memory network model. The training process involves retaining and updating historical voice prompt feature data in the input sequence through the network's memory units. Combined with the voice prompt feature data of the current response duration, predict the trend of behavioral score changes. By minimizing the error between the predicted output and the actual behavioral score difference, iteratively optimize the parameters of the long short-term memory network model to generate a behavioral score prediction model. For example, the system will issue a voice prompt when an abnormal situation occurs, such as: "Attention, humidity exceeds the standard, please adjust the equipment settings as soon as possible!" The urgency level label is Level I (Low Urgency) = "Warning: Abnormal temperature, please check"; Level II (Medium Emergency) = "Emergency: Humidity too high, check immediately"; Level III (High Emergency) = "Emergency: Equipment failure, stop work immediately"; The record shows that the R1 alert message indicates excessive humidity. The alert message was issued at 10:05:00, and the next alert message will be issued at 10:10:00. The urgency level is labeled as Level II. The record shows that the R2 prompt message is "too much noise" and the prompt message was issued at 10:15:00. The next prompt message will be issued at 10:18:00. The urgency level label is Level I. The record shows that the R3 alert message is for a high temperature anomaly, issued at 10:25:00, with the next alert scheduled for 10:30:00. The urgency level is labeled as Level III. Record the response time of R1 as 10:10:00−10:05:00=5 minutes; The response time of R2 was recorded as 10:18:00−10:15:00=3 minutes; The response time of R3 was recorded as 10:30:00−10:25:00=5 minutes; Voice volume levels, volume settings: Low=1, Medium=2, High=3; Speech speed settings: Slow=1, Medium=2, Fast=3; The voice prompt feature data is shown below: Record R1 volume level as 3 and speech rate level as 2; Record R2 volume level as 2 and speech rate level as 1; Record R3 volume level as 3 and speech rate level as 3; The preset reaction time threshold is 3 minutes; The behavioral characteristic data is shown below: The R1 response time was recorded as 2 minutes, the duration of the abnormal construction action was 4 minutes, and the response time was 5 minutes. The R2 response time was recorded as 3 minutes, the duration of the abnormal construction action was 6 minutes, and the response time was 3 minutes. The R3 response time was recorded as 4 minutes, the duration of the abnormal construction action was 8 minutes, and the response time was 5 minutes. The recorded R1 behavioral score is 0.5×2 / 3+0.5×4 / 5=0.333+0.8=1.133; The recorded R2 behavioral score is 0.5 × 3 / 3 + 0.5 × 6 / 3 = 0.5 + 1 = 1.5; The recorded R3 behavioral score is 0.5×4 / 3+0.5×8 / 5=0.667+0.8=1.467; The input sequence is: record R1 volume level 3, speech rate level 2, and behavior score 1.133; Record R2 volume level 2, speech rate level 1, and behavior score 1.5; Record R3 volume level 3, speech rate level 3, behavior score 1.467; The output sequence records the difference in R2 behavioral scores as 1.5 - 1.133 = 0.367; The recorded difference in R3 behavioral scores was 1.467 - 1.5 = -0.033; The input and output sequences are fed into the LSTM model for training. The LSTM model uses historical speech features to predict the changing trend of behavior scores. By continuously optimizing the parameters, the difference between the predicted behavior scores and the actual behavior scores is minimized, and the final prediction model is obtained.

[0022] Obtain real-time construction information, determine the real-time dynamic anomaly level, and predict the real-time behavior score based on the behavior score prediction model corresponding to the real-time dynamic anomaly level, so as to determine whether to adjust the voice prompt parameters. Determining whether to adjust the voice prompt parameters includes the following steps: The static anomaly level is matched with the real-time construction procedure, and the real-time comprehensive environmental anomaly score is calculated based on the real-time values ​​of characteristic environmental parameters. The real-time dynamic anomaly score is calculated by combining the real-time similarity score and the static anomaly level. The real-time dynamic anomaly level is determined, and the real-time speech feature data is input into the corresponding model to predict the behavior score. When the score is lower than the threshold, the speech volume or speech rate parameter is automatically adjusted. For example, step S601: acquire real-time construction video, determine the real-time construction procedure corresponding to the real-time construction video, and acquire the static anomaly level corresponding to the real-time construction procedure; Step S602: Based on the static anomaly level, obtain the set of characteristic environmental parameters of the static anomaly level, collect the real-time value of each characteristic environmental parameter, when the real-time value of any characteristic environmental parameter does not meet the preset threshold range, calculate the real-time environmental deviation index, obtain the combination of associated characteristic environmental parameters corresponding to the characteristic environmental parameter, multiply the real-time environmental deviation index by the Pearson correlation coefficient of the two characteristic environmental parameters, calculate the real-time environmental deviation index of the associated characteristic environmental parameter, and perform a weighted summation of the real-time environmental deviation index of each characteristic environmental parameter to obtain the real-time comprehensive environmental anomaly score; Step S603: Obtain the real-time similarity score of abnormal construction actions, normalize the real-time similarity score and the real-time comprehensive environmental anomaly score to calculate the real-time dynamic anomaly score, and determine the real-time dynamic anomaly level. Step S604: Obtain the behavior score prediction model corresponding to the real-time dynamic anomaly level, input the voice prompt feature data into the behavior score prediction model, and predict the real-time behavior score. If the real-time behavior score is lower than the behavior score threshold, the remote monitoring platform automatically adjusts at least one voice prompt feature parameter. If the real-time behavior score is higher than or equal to the behavior score threshold, the current voice prompt feature data is output.

[0023] To better implement the above method, an artificial intelligence voice interaction system based on multimodal fusion is proposed. The system includes an interaction recording module, a static anomaly level module, a correlation parameter module, a dynamic anomaly level module, a behavior scoring prediction model module, and a judgment adjustment module. Interactive recording module: acquires video information from the power construction site, identifies the construction actions of the construction workers, and generates interactive records when the construction actions are abnormal; The interactive recording module includes a unit for determining abnormal construction actions and a unit for generating interactive records. Abnormal construction action determination unit: The video information of the construction site is obtained through monitoring equipment, and the video segments of the construction work completed by the construction personnel are extracted. The continuous image frames of the construction action are identified in the video segments, and the abnormal image frame set of each construction action with construction risk is obtained. The similarity of each abnormal image frame is compared with the continuous image frames. If the similarity is higher than the preset threshold, it is determined to be an abnormal construction action. Interactive recording unit: Collects environmental parameters of the construction site and synchronizes them with video segments in time to generate time-series perception data. When abnormal construction actions are detected, it issues voice prompts and generates interactive records.

[0024] Static anomaly level module: Based on historical interaction records, calculate the similarity score of abnormal construction actions, and determine the static anomaly level corresponding to the construction procedure based on the similarity score; Association parameter module: Obtains environmental parameters corresponding to the interaction record, and calculates a comprehensive environmental anomaly score based on the abnormal conditions of the environmental parameters; Dynamic anomaly level module: Based on the similarity score, comprehensive environmental anomaly score, and static anomaly level, determine the dynamic anomaly level corresponding to the interaction record; The dynamic anomaly level module includes a unit for calculating dynamic anomaly scores and a unit for determining dynamic anomaly levels. The dynamic anomaly scoring unit is calculated as follows: Values ​​of characteristic environmental parameters in the interaction records are collected, and their absolute differences are calculated with the threshold values ​​of the characteristic environmental parameters. These absolute differences are used as environmental deviation indicators. The environmental deviation indicators for each characteristic environmental parameter are weighted and summed to obtain a comprehensive environmental anomaly score. The similarity scores corresponding to construction actions and the static anomaly levels corresponding to construction procedures in the interaction records are extracted. The static anomaly levels are assigned values, and the similarity scores and comprehensive environmental anomaly scores are normalized. The assigned static anomaly levels, the normalized similarity scores, and the comprehensive environmental anomaly scores are weighted and fused according to their respective weighting coefficients to calculate the dynamic anomaly score for the construction procedure. Determine the dynamic anomaly level unit: Based on the mean and standard deviation of the dynamic anomaly scores of all interaction records, and using a preset standard deviation multiple, divide the interaction records into several dynamic anomaly levels.

[0025] Behavior scoring prediction model module: Collect voice prompt feature data and construction personnel behavior feature data from interaction records to build a behavior scoring prediction model; Judgment and Adjustment Module: Obtains real-time construction information, determines the real-time dynamic anomaly level, and predicts the real-time behavior score based on the behavior score prediction model corresponding to the real-time dynamic anomaly level, so as to determine whether to adjust the voice prompt parameters.

[0026] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. An artificial intelligence voice interaction method based on multimodal fusion, characterized in that, The method includes: Acquire video information from power construction sites, identify the construction actions of construction workers, and generate interactive records when the construction actions are abnormal; Based on historical interaction records, a similarity score is calculated for abnormal construction actions, and the static abnormality level corresponding to the construction procedure is determined according to the similarity score. Obtain the environmental parameters corresponding to the interaction record, and calculate a comprehensive environmental anomaly score based on the abnormal conditions of the environmental parameters; Based on the similarity score, comprehensive environmental anomaly score, and static anomaly level, the dynamic anomaly level corresponding to the interaction record is determined; Collect voice prompt feature data and construction worker behavior feature data from interaction records, and construct a behavior scoring prediction model; The construction of the behavior rating prediction model includes the following steps: The response time is determined based on the urgency level label of the voice prompt. Within the response time, voice volume level, speech rate feature data, as well as the construction worker's reaction time and the duration of abnormal actions are extracted. Based on the ratio between the construction worker's reaction time and a preset reaction time threshold, a first score reflecting reaction speed is calculated. Based on the ratio between the duration of abnormal actions and the response time, a second score reflecting the duration of abnormal behavior is calculated. The first score and the second score are weighted and fused according to their respective weighting coefficients to obtain the behavior score. The time point at which the abnormal construction action stopped is obtained. The time period between the issuance time of the first round of prompt information and the time point at which the abnormal construction action stopped is used as the evaluation period. The voice prompt feature data within the evaluation period is assigned values. The input sequence is constructed in chronological order based on the differences in voice prompt feature data within multiple adjacent response durations, and the output sequence is constructed based on the behavioral score differences between corresponding adjacent response durations. Based on the input sequence and the output sequence, a long short-term memory network model is trained. The training process involves retaining and updating the historical voice prompt feature data in the input sequence through the network's memory units, and combining it with the voice prompt feature data of the current response duration to predict the trend of behavioral score changes. By minimizing the error between the predicted output and the actual behavioral score difference, the parameters of the long short-term memory network model are iteratively optimized to generate a behavioral score prediction model. Obtain real-time construction information, determine the real-time dynamic anomaly level, and predict the real-time behavior score based on the behavior score prediction model corresponding to the real-time dynamic anomaly level, so as to determine whether to adjust the voice prompt parameters. Determining whether to adjust the voice prompt parameters includes the following steps: Based on the real-time construction process matching static anomaly level, a real-time comprehensive environmental anomaly score is calculated based on the real-time value of feature environmental parameters; a real-time dynamic anomaly score is calculated by combining the real-time similarity score and the static anomaly level, and the real-time dynamic anomaly level is determined. Real-time speech feature data is input into the corresponding model to predict the behavior score, and the speech volume or speech rate parameter is automatically adjusted when it is below the threshold.

2. The artificial intelligence voice interaction method based on multimodal fusion according to claim 1, characterized in that, The process of generating an interaction log when the construction action is abnormal includes the following steps: By acquiring video information from the construction site through monitoring equipment, video segments of construction workers completing construction tasks are extracted; In the video segment, identify continuous image frames of construction actions, obtain a set of abnormal image frames for each construction action that poses a construction risk, and compare the similarity between each abnormal image frame and the continuous image frames. If the similarity is higher than a preset threshold, it is determined to be an abnormal construction action. The system collects environmental parameters from the construction site and synchronizes them with video segments to generate time-series perception data. When abnormal construction actions are detected, it issues voice prompts and generates interactive records.

3. The artificial intelligence voice interaction method based on multimodal fusion according to claim 2, characterized in that, Determining the static anomaly level corresponding to the construction procedure includes the following steps: Collect continuous image frames of abnormal construction actions in a certain historical interaction record, extract the human joint coordinates of each frame of the continuous image frames of abnormal construction actions, and normalize the human joint coordinates. Construct action features through the normalized human joint coordinates. The action features include the relative displacement vector between joints, joint angles, and the rate of change of joint angles. The process involves obtaining action features corresponding to abnormal construction actions with historical construction risks, calculating average action features, and performing similarity calculations between the action features of the abnormal construction actions and the average action features to obtain the similarity of the abnormal construction actions. The similarity calculation uses cosine similarity to obtain the similarity result. Based on the importance of each abnormal construction action, corresponding weight values ​​are set, and the similarity results of each abnormal construction action are weighted and summed with the corresponding weight values. The weighted results are then averaged based on the total number of abnormal construction actions to obtain the similarity score of the historical interaction records. The similarity score of historical interaction records for each construction process is calculated and normalized. The average similarity score is then calculated. Simultaneously, the total number of historical perception data points for each construction process is calculated. The percentage of historical interaction records is calculated by dividing the total number of historical perception data points by the total number of historical interaction data points. The average similarity score and the percentage of historical interaction data are then weighted and summed to calculate the anomaly score for each construction process. Based on the mean and standard deviation of the abnormality scores of all construction procedures, and using a preset standard deviation multiple, the construction procedures are divided into several static abnormality levels.

4. The artificial intelligence voice interaction method based on multimodal fusion according to claim 1, characterized in that, The calculation of the comprehensive environmental anomaly score includes the following steps: A training period of several consecutive days is set, and the interaction records within the training period are obtained. The interaction records are classified according to the static anomaly level corresponding to the construction process. In the set of interaction records of the static anomaly level, environmental parameters that exceed the preset threshold range are identified as abnormal environmental factors. The proportion of each abnormal environmental factor in the static anomaly level is counted, and factors that are higher than the proportion threshold are marked as characteristic environmental parameters of the static anomaly level. Interaction records with two or more abnormal environmental parameters are marked as composite interaction records. The characteristic environmental parameters that are abnormal environmental factors are combined to generate a combination of characteristic environmental parameters for composite interaction records. In the set of composite interaction records with each combination of characteristic environmental parameters, the parameter values ​​of the combination of characteristic environmental parameters in the composite interaction records are collected to construct the numerical sequence of the combination of characteristic environmental parameters. Based on the numerical sequence of the combination of the characteristic environmental parameters, the Pearson correlation coefficient of the two characteristic environmental parameters is calculated. When the absolute value of the Pearson correlation coefficient is greater than the preset correlation threshold, it is determined that there is a correlation between the two characteristic environmental parameters, and the combination of characteristic environmental parameters with correlation is marked as the associated characteristic environmental parameter combination.

5. The artificial intelligence voice interaction method based on multimodal fusion according to claim 4, characterized in that, Determining the dynamic anomaly level corresponding to the interaction record includes the following steps: The numerical values ​​of the characteristic environmental parameters associated with the combination of characteristic environmental parameters in the collection and interaction records are used to calculate the absolute difference with the threshold of the characteristic environmental parameters. The absolute difference is used as the environmental deviation index. The environmental deviation index of each characteristic environmental parameter is weighted and summed to obtain a comprehensive environmental anomaly score. Extract the similarity scores corresponding to construction actions in the interaction records, determine the static anomaly level corresponding to the construction procedure, assign values ​​to the static anomaly levels, normalize the similarity scores and comprehensive environmental anomaly scores, and weight and fuse the static anomaly level values, normalized similarity scores and comprehensive environmental anomaly scores according to the corresponding weight coefficients to calculate the dynamic anomaly scores of the construction procedures. Based on the mean and standard deviation of the dynamic anomaly scores of all interaction records, and using a preset standard deviation multiple, the interaction records are divided into several dynamic anomaly levels.

6. An artificial intelligence voice interaction system based on multimodal fusion, used to implement the artificial intelligence voice interaction method based on multimodal fusion as described in any one of claims 1-5, characterized in that, The system includes an interaction recording module, a static anomaly level module, a correlation parameter module, a dynamic anomaly level module, a behavior scoring prediction model module, and a judgment adjustment module. The interactive recording module acquires video information from the power construction site, identifies the construction actions of the construction workers, and generates an interactive record when the construction actions are abnormal. The static anomaly level module calculates the similarity score of abnormal construction actions based on historical interaction records, and determines the static anomaly level corresponding to the construction procedure based on the similarity score. The associated parameter module: obtains the environmental parameters corresponding to the interaction record, and calculates a comprehensive environmental anomaly score based on the abnormal conditions of the environmental parameters; The dynamic anomaly level module determines the dynamic anomaly level corresponding to the interaction record based on the similarity score, the comprehensive environmental anomaly score, and the static anomaly level. The behavior scoring prediction model module collects voice prompt feature data and construction personnel behavior feature data from the interaction records to construct a behavior scoring prediction model. The judgment and adjustment module acquires real-time construction information, determines the real-time dynamic anomaly level, and predicts a real-time behavior score based on the behavior score prediction model corresponding to the real-time dynamic anomaly level, so as to determine whether to adjust the voice prompt parameters.

7. The artificial intelligence voice interaction system based on multimodal fusion according to claim 6, characterized in that, The interactive recording module includes a unit for determining abnormal construction actions and a unit for generating interactive records. The abnormal construction action determination unit: acquires video information of the construction site through monitoring equipment, extracts video segments of construction workers completing construction tasks, identifies continuous image frames of construction actions in the video segments, obtains a set of abnormal image frames for each construction action that poses a construction risk, and compares the similarity of each abnormal image frame with the continuous image frames. If the similarity is higher than a preset threshold, it is determined to be an abnormal construction action. The interactive recording generation unit collects environmental parameters from the construction site and synchronizes them with video segments in time to generate time-series perception data. When abnormal construction actions are detected, it issues voice prompts and generates interactive records.

8. The artificial intelligence voice interaction system based on multimodal fusion according to claim 6, characterized in that, The dynamic anomaly level module includes a dynamic anomaly score calculation unit and a dynamic anomaly level determination unit: The dynamic anomaly scoring unit collects the values ​​of characteristic environmental parameters in the interaction records, calculates the absolute difference between the values ​​and the threshold values ​​of the characteristic environmental parameters, uses the absolute difference as an environmental deviation index, and performs a weighted summation of the environmental deviation indices of each characteristic environmental parameter to obtain a comprehensive environmental anomaly score. It also extracts the similarity score corresponding to the construction actions in the interaction records and the static anomaly level corresponding to the construction procedures, assigns values ​​to the static anomaly levels, normalizes the similarity scores and the comprehensive environmental anomaly scores, and performs a weighted fusion of the static anomaly level values, the normalized similarity scores, and the comprehensive environmental anomaly scores according to their corresponding weight coefficients to calculate the dynamic anomaly score of the construction procedures. The dynamic anomaly level determination unit: Based on the mean and standard deviation of the dynamic anomaly scores of all interaction records, and based on a preset standard deviation multiple, divides the interaction records into several dynamic anomaly levels.

Images