Comprehensive operation and maintenance management platform based on Internet of Things

Abstract

The invention relates to the technical field of management of the Internet of Things, in particular to a comprehensive operation and maintenance management platform based on the Internet of Things, which is provided with an Internet of Things acquisition module, a data monitoring module, an analysis module, an instruction control module and an instruction response module, the instruction characteristic determining module is used for determining instruction characteristics of the voice signals, the analysis module is used for calculating instruction characterization parameters for the voice signals in a time domain section based on the instruction characteristics so as to determine instruction clear categories of the voice signals, and the instruction control module is used for controlling the instruction characterization parameters. And the instruction response module is used for analyzing the voice signal based on the instruction clear category of the voice signal in the time domain section and determining a control instruction, and the instruction response module is used for controlling preset equipment in the classroom based on the determined control instruction. According to the invention, by determining the control instruction, the precision of equipment control is improved. And the accuracy and the management efficiency of voice analysis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things management, and in particular to a comprehensive operation and maintenance management platform based on the Internet of Things. Background Art

[0002] The integrated operation and maintenance management platform is a platform that integrates a variety of operation and maintenance tools and technologies. Its application scenarios are very wide, including but not limited to manufacturing, e-commerce, finance, telecommunications, the Internet, etc. They all rely on a stable IT system to support their business continuity and customer experience, improve the efficiency and quality of operation and maintenance work, reduce the occurrence of failures and accidents, and provide comprehensive security management functions. Especially with the continuous development of technologies such as cloud computing, big data, and artificial intelligence, the integrated operation and maintenance management platform is also constantly innovating and improving, and it also has great application prospects in the campus environment. It can uniformly manage and maintain various information resources on campus, including hardware, software, networks and other infrastructure, and improve management efficiency and management quality.

[0003] Chinese patent publication number: CN118574148A, discloses an integrated operation and maintenance management system based on the Internet of Things and its method, the system includes: a number of monitoring systems, each monitoring system is deployed in a communication base station, and is used to monitor the power parameters and environmental parameters of the communication base station in real time; a monitoring station, the monitoring station includes: at least one main station, each main station is connected to a number of auxiliary stations, and each auxiliary station is deployed within a pre-divided area; each auxiliary station is used to collect the power parameters and environmental parameters monitored by all monitoring systems within the area where the auxiliary station is located, determine whether the power parameters and / or environmental parameters are abnormal, analyze the abnormal power parameters and / or environmental parameters, generate abnormal reports and warning information, and upload the abnormal reports and warning information to the main station. The present invention enhances the monitoring function of the power environment of the communication base station, and can effectively improve the automation of the operation and maintenance management of the communication base station.

[0004] Chinese patent publication number: CN118863868A, discloses a system operation and maintenance integrated management platform, including: energy-consuming equipment monitoring system, used to obtain the operating data and operating status of each energy-consuming equipment in the power dispatching data network; full-process operation and maintenance control module, through intelligent dispatching operation work; energy management system, relying on the Internet of Things and big data to divide energy utilization into regions, categories, time divisions, and items for real-time detection, automatic statistics and analysis. In the present invention, the energy management system has obvious effects in load optimization, fault prediction accuracy, power outage frequency reduction and energy efficiency improvement. Based on the integrated application of artificial intelligence, the Internet of Things and big data analysis technology, scientific data extraction and intelligent management are realized at the data processing layer, and intelligent inspection, active operation and maintenance and intelligent diagnosis are realized at the application layer, and the business aggregation, process integration and data fusion of the energy operation and maintenance management functions of the industrial park are realized.

[0005] However, there are still the following problems in the prior art:

[0006] The existing integrated operation and maintenance management platform is mostly used in industrial parks, but it has a wide range of uses and is also applicable in school scenarios. When the integrated operation and maintenance management platform is used to manage campus scenarios, when voice analysis is required, more than one person's voice signals may be included, especially at break time, when the number of people speaking increases suddenly. In this case, the platform still analyzes all voice signals, which will reduce the analysis speed, resulting in lower recognition accuracy and lower accuracy of voice analysis, affecting the accuracy of the analyzed control instructions. Summary of the invention

[0007] To this end, the present invention provides a comprehensive operation and maintenance management platform based on the Internet of Things, which is used to solve the problem that when voice analysis is required, more than one person's voice signals may be included, especially at get out of class time, when the number of people speaking increases suddenly. In this case, the platform still analyzes all voice signals, which will reduce the analysis speed, resulting in lower recognition accuracy, lower accuracy of voice analysis, and affecting the accuracy of the analyzed control instructions.

[0008] To achieve the above objectives, the present invention provides a comprehensive operation and maintenance management platform based on the Internet of Things, which includes:

[0009] The Internet of Things acquisition module includes a voice acquisition unit arranged in a target area of ​​the classroom for acquiring voice signals and a monitoring unit for acquiring monitoring data;

[0010] A data monitoring module, which is connected to the Internet of Things acquisition module, is used to analyze the voice signal in each time domain segment, and determine the instruction characteristics of the voice signal based on the analysis result, including the semantic relevance corresponding to the voice signal and the fluctuation value of the voice signal in the time domain dimension;

[0011] An analysis module connected to the data monitoring module, for calculating a command characterization parameter for the voice signal in a time domain segment based on the command feature, so as to determine a command clarity category of the voice signal;

[0012] An instruction control module is connected to the analysis module and is used to analyze the voice signal based on the instruction clarity category of the voice signal in the time domain segment to determine the control instruction, including:

[0013] Determine the sight direction of each monitoring target based on the monitoring data, determine the key monitoring target based on the angle between the sight direction and the predetermined receiving direction, synchronously determine the lip shape data of the key monitoring target, match the lip shape data with the voice signal to obtain a text signal to determine the control instruction;

[0014] Or, directly convert the voice signal into a text signal to determine the control instruction;

[0015] The instruction response module is connected to the Internet of Things acquisition module and the instruction control module, and is used to control the predetermined equipment in the classroom based on the determined control instruction.

[0016] Furthermore, the data monitoring module determines the semantic relevance corresponding to the speech signal, including:

[0017] Used to convert the voice signals in each time domain segment into text information;

[0018] Used to extract keywords from the text information;

[0019] Used to determine the semantic relevance of the text information according to the keywords.

[0020] Furthermore, the data monitoring module determines the fluctuation value of the speech signal in the time domain dimension, including:

[0021] Used to determine the peak value of the speech signal in each time domain segment;

[0022] The variance used to determine each peak value in the time domain segment is the fluctuation value.

[0023] Furthermore, the analysis module calculates instruction representation parameters for the speech signal in the time domain segment, including:

[0024] The ratio of the semantic relevance to the reference semantic relevance is determined as a semantic relevance influencing parameter;

[0025] To determine the ratio of the reference fluctuation value to the fluctuation value as the fluctuation value influencing parameter;

[0026] The weighted sum of the semantic relevance influencing parameter and the fluctuation value influencing parameter is used to determine the instruction representation parameter.

[0027] Further, the analysis module determines the command clarity category of the voice signal, wherein:

[0028] If the instruction characterization parameter is less than the instruction characterization parameter threshold, determining that the instruction clarity category is a weak clarity instruction;

[0029] If the instruction characterization parameter is greater than or equal to the instruction characterization parameter threshold, it is determined that the instruction clarity category is a strong clarity instruction.

[0030] Furthermore, the instruction control module analyzes the voice signal based on the instruction clarity category of the voice signal in the time domain segment to determine the control instruction, wherein:

[0031] If the command clarity category is a weakly clear command, the sight direction of each monitoring target is determined based on the monitoring data, the key monitoring target is determined based on the angle between the sight direction and the predetermined receiving direction, the lip shape data of the key monitoring target is determined synchronously, and the lip shape data is matched with the voice signal to obtain a text signal to determine the control command;

[0032] If the command clarity category is a strong clear command, the voice signal is directly converted into a text signal to determine the control command.

[0033] Furthermore, the command control module determines the sight direction of each monitoring target, including:

[0034] Determine a facial reference point group of a monitoring target and construct a facial plane according to the facial reference point group;

[0035] Used to determine the direction of the vector perpendicular to the facial plane as the sight direction of the monitoring target;

[0036] The facial reference point group is three points within the facial contour area.

[0037] Furthermore, the command control module determines the key monitoring target based on the angle between the line of sight direction and the predetermined receiving direction, including:

[0038] Used to determine the angle between the line of sight direction corresponding to the monitoring target and the predetermined receiving direction;

[0039] If the degree of the angle is less than the degree of the reference angle, it is determined that the monitoring target corresponding to the line of sight direction is a key monitoring target.

[0040] Furthermore, the instruction control module matches the lip shape data with the voice signal, including:

[0041] Determine a number of associated lip shape keywords based on the lip shape data;

[0042] To match each keyword in the voice signal with the lip shape keyword;

[0043] Converting a speech signal into a text signal and removing unmatched keywords from the text signal;

[0044] If the keyword in the voice signal does not match any of the lip shape keywords, the keyword is determined to be an unmatched keyword.

[0045] Furthermore, the instruction response module controls the predetermined device in the classroom based on the determined control instruction, including:

[0046] Used to determine control instructions according to text signals;

[0047] to determine the predetermined device in the classroom corresponding to the control instruction;

[0048] Used to send the control instruction to the predetermined device.

[0049] Compared with the prior art, the present invention is provided with an Internet of Things acquisition module, a data monitoring module, an analysis module, a command control module and a command response module. The data monitoring module is used to analyze the voice signal in each time domain segment and determine the command characteristics of the voice signal, including the semantic relevance corresponding to the voice signal and the fluctuation value of the voice signal in the time domain dimension. The analysis module is used to calculate the command characterization parameters for the voice signal in the time domain segment based on the command characteristics to determine the command clarity category of the voice signal. The command control module is used to analyze the voice signal based on the command clarity category of the voice signal in the time domain segment and determine the control command. The command response module is used to control the predetermined equipment in the classroom based on the determined control command. The present invention improves the accuracy of equipment control by determining the control command. The accuracy of voice analysis and management efficiency are improved.

[0050] In particular, the present invention analyzes the voice signals in each time domain segment, and converts the voice signals into text information and audio information, and then determines the command characteristics of the voice signals. In actual use, when receiving the voice signals of the monitored target, the target device is controlled according to the voice signals to complete operations such as switching. For a single monitored target, the fluctuation value of the voice signal emitted by it in the time domain dimension is small, but in a campus environment, especially during the break time, the background is noisy, and the voice signal of the monitored target may be interfered with, which is reflected in the increased difficulty of subsequently distinguishing the voice signals of the monitored targets, and then causing errors in the identified control commands, and the target device cannot be effectively controlled. Based on this, the present invention considers conducting an overall analysis of the collected voice signals, determining the semantic relevance corresponding to the voice signals and the fluctuation value of the voice signals in the time domain dimension, providing a data basis for calculating the instruction characterization parameters, making the recognition of the monitored target voice signals more accurate, and improving the accuracy of determining the control commands.

[0051] In particular, the present invention determines the instruction clarity category of the voice signal by calculating the instruction characterization parameters of the voice signal in the time domain segment. When analyzing the voice signal, there will be a clearer voice signal. At this time, if the received voice signal is still subjected to complex analysis, this non-adaptive adjustment and analysis method makes the analysis efficiency of the voice signal low. Based on this, the present invention calculates the instruction characterization parameters, classifies the voice signal, directly determines the control instruction for the voice signal with strong and clear instructions, and performs signal matching on the voice signal with weak and clear instructions, so that the recognition of the monitored target voice signal is more accurate, and the accuracy of determining the control instruction is improved.

[0052] In particular, the present invention determines control instructions based on key monitoring targets and lip shape data. For the situation where multiple voice signals are received in a noisy environment, it is necessary to analyze the voice signals to obtain accurate control instructions. Based on this, the present invention considers real-time monitoring of target areas in the classroom, and determines key monitoring targets based on monitoring data, matches the lip shape data of the key monitoring targets with the voice signals, and then determines the control instructions, so that the recognition of the monitoring target voice signals is more accurate, thereby improving the accuracy of determining the control instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of the structure of an integrated operation and maintenance management platform based on the Internet of Things according to an embodiment of the present invention;

[0054] Figure 2 A logic block diagram of determining the instruction clarity category of the voice signal according to an embodiment of the present invention;

[0055] Figure 3 A logic block diagram for determining a control instruction according to an embodiment of the present invention;

[0056] Figure 4 This is a logic block diagram for determining key monitoring targets according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0059] It should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] See also Figure 1-Figure 4 As shown, Figure 1 is a schematic diagram of the structure of an integrated operation and maintenance management platform based on the Internet of Things according to an embodiment of the present invention, Figure 2 is a logic block diagram of determining the instruction clarity category of the voice signal according to an embodiment of the present invention, Figure 3is a logic block diagram of determining a control instruction according to an embodiment of the present invention, Figure 4 The present invention is a logic block diagram for determining key monitoring targets in an embodiment of the present invention. The present invention is a comprehensive operation and maintenance management platform based on the Internet of Things, which includes:

[0061] The Internet of Things acquisition module includes a voice acquisition unit arranged in a target area of ​​the classroom for acquiring voice signals and a monitoring unit for acquiring monitoring data;

[0062] Specifically, the target area of ​​the classroom is the visible area of ​​the integrated operation and maintenance management platform. It can be understood that only when voice control of the integrated operation and maintenance management platform is performed in the target area can the integrated operation and maintenance management platform be activated for operation. This will not be elaborated here.

[0063] Specifically, there is no limitation on the collection method of the voice collection unit. For example, it can be a voice collector. There are mainly the following types of voice collectors:

[0064] Digital pickup: The digital pickup adds clock circuit, storage, and digital processor, which can minimize noise and restore human voice, and is suitable for environments with high performance requirements.

[0065] Microphone array: The microphone array uses the spatial filtering characteristics to form a directional sound pickup beam by locating the angle of the awakened person, and suppresses the noise outside the beam to ensure high recording quality.

[0066] Audio acquisition and playback system: This type of system includes the conditioning, acquisition, storage and playback of voice signals. Based on the principle of digital storage and recovery of voice signals, it uses A / D, D / A conversion technology and interpolation compression algorithm of voice signals to realize the digital storage and playback functions of voice signals.

[0067] The above-mentioned voice collectors can all collect voice signals, which will not be described in detail here.

[0068] Specifically, there is no limitation on the collection method of the monitoring unit. For example, it can be a camera. It only needs to ensure that the lip shape of any target in the target area can be monitored. This is the existing technology and will not be elaborated.

[0069] Specifically, there is no limitation on the specific content of the voice signal. In this embodiment, the voice signal can be a sentence or a command phrase, for example, "Please help me turn on the air conditioner and adjust the temperature to 26 degrees", "Turn on the air conditioner", "Set the temperature to 26 degrees", etc., which will not be repeated here.

[0070] A data monitoring module, which is connected to the Internet of Things acquisition module, is used to analyze the voice signal in each time domain segment, and determine the instruction characteristics of the voice signal based on the analysis result, including the semantic relevance corresponding to the voice signal and the fluctuation value of the voice signal in the time domain dimension;

[0071] An analysis module connected to the data monitoring module, for calculating a command characterization parameter for the voice signal in a time domain segment based on the command feature, so as to determine a command clarity category of the voice signal;

[0072] An instruction control module is connected to the analysis module and is used to analyze the voice signal based on the instruction clarity category of the voice signal in the time domain segment to determine the control instruction, including:

[0073] Determine the sight direction of each monitoring target based on the monitoring data, determine the key monitoring target based on the angle between the sight direction and the predetermined receiving direction, synchronously determine the lip shape data of the key monitoring target, match the lip shape data with the voice signal to obtain a text signal to determine the control instruction;

[0074] Or, directly convert the voice signal into a text signal to determine the control instruction;

[0075] It can be understood that by determining the control instruction based on the voice signal, the voice signal can be converted into a text signal, and the control instruction corresponding to the voice signal can be determined by pre-establishing an association relationship between the text signal and the control instruction. For example, taking the text signal corresponding to the voice signal as "turn on the air conditioner", an association relationship between "turn on the air conditioner" and the control instruction is pre-established, and then the corresponding control instruction is determined to control the air conditioner. Of course, other methods can also be used, and technical personnel in this field can choose according to needs, which will not be repeated here.

[0076] The instruction response module is connected to the Internet of Things acquisition module and the instruction control module, and is used to control the predetermined equipment in the classroom based on the determined control instruction.

[0077] Specifically, the data monitoring module determines the semantic relevance corresponding to the speech signal, including:

[0078] Used to convert the voice signals in each time domain segment into text information;

[0079] Used to extract keywords from the text information;

[0080] Used to determine the semantic relevance of the text information according to the keywords.

[0081] Specifically, there is no limitation on the method of converting the voice signal into text information, as long as the voice signal conversion can be achieved, for example, it can be;

[0082] Automatic Speech Recognition (ASR) system: Using computer software to recognize and understand speech commands and convert them into text, which is the prior art and can perform real-time conversion, improving the efficiency of determining semantic relevance.

[0083] Speech recognition APIs and SDKs: In the prior art, speech recognition application programming interfaces (APIs) and software development kits (SDKs) can easily integrate speech recognition capabilities into applications. For example, Google Cloud Speech-to-Text, IBM Watson Speech to Text, Amazon Transcribe, etc.

[0084] Specifically, before extracting keywords, the keywords need to be determined. In the prior art, the methods for determining keywords can be based on word segmentation tools, TF-IDF, and keyword extraction tools, etc. In this embodiment, TF-IDF is used to extract keywords, and the steps are as follows.

[0085] Step S1, clean the text information, including removing punctuation marks, stop words (such as common words like "of", "is", etc.), and may also include stemming or lemmatization.

[0086] Step S2, create a vocabulary to store historical text information, and the vocabulary contains all non-repeating words.

[0087] Step S3, calculate the term frequency of each word in the vocabulary, that is, the number of times the word appears in the historical text information.

[0088] Step S4, for each word in the vocabulary, calculate its inverse document frequency in all documents, and the inverse document frequency is calculated by formula (1).

[0089]

[0090] Among them, N represents the total number of historical text information, and |{d ∈ D: t ∈ d}| represents the number of historical text information containing the word t.

[0091] Step S5, calculate the TF-IDF value of each word in the historical text information, and the TF-IDF value is calculated by formula (2).

[0092] TF-IDF(t, d) = TF(t, d) × IDF(t) (2)

[0093] Among them, TF(t, d) represents the term frequency of the word t in the historical text information d, and IDF(t) represents the inverse document frequency of the word t.

[0094] Step S6, creating a matrix, wherein the rows represent historical text information, the columns represent words in the vocabulary, and the values ​​in the matrix are the TF-IDF values ​​of the corresponding words in the corresponding historical text information;

[0095] Step S7, for the text information, select the word with the highest TF-IDF value as the keyword. It can be understood that a keyword determination threshold can be set to select the top N highest words to be determined as the keywords of the text information.

[0096] Regarding the method of extracting keywords, those skilled in the art may select one according to actual conditions, which will not be described in detail here.

[0097] Specifically, there is no limitation on the method for calculating the semantic relevance of text information, as long as it can achieve the purpose of calculating the semantic relevance. For example, the following methods can be used:

[0098] Vector space model method: convert keywords into vectors, calculate the similarity between vectors through methods such as cosine similarity, and determine the mean similarity between the corresponding vectors of each word in the text information as the semantic association.

[0099] Semantic network method: construct a relationship network between keywords, measure the correlation between words through indicators such as path length, and determine the average correlation between words in the text information as the semantic correlation.

[0100] Of course, other forms may also be used, which will not be described in detail. In this embodiment, the semantic relevance is calculated by using cosine similarity.

[0101] Specifically, the data monitoring module determines the fluctuation value of the speech signal in the time domain dimension, including:

[0102] Used to determine the peak value of the speech signal in each time domain segment;

[0103] The variance used to determine each peak value in the time domain segment is the fluctuation value.

[0104] Specifically, there is no limitation on the method for detecting the peak value of the speech signal, for example, it may be:

[0105] Sliding window method: the continuous speech signal is divided into several segments, and the maximum amplitude of the signal strength, that is, the peak value, is found in each segment of the speech signal. It can be understood that the length of the speech signal after being divided depends on the length of the sliding window. The sliding window can be used to divide the speech signal into several segments.

[0106] The basic formula principle method of peak detection is to divide the speech signal into several frames of fixed length. For each frame, calculate the absolute value of the signal strength of all sample points in the frame to obtain a vector, perform peak detection on the vector, and find the maximum value in the vector, which is the peak value of the frame. Repeat the above steps to obtain several peak values.

[0107] It is understandable that the above methods can be used alone or in combination to improve the accuracy and robustness of peak detection.

[0108] Specifically, calculating the variance of each peak value in the time domain segment is an existing technology, and those skilled in the art can choose it according to the actual situation. It only needs to obtain the variance of each peak value in the time domain segment, which will not be elaborated here.

[0109] Specifically, by analyzing the voice signals in each time domain segment and converting the voice signals into text information and audio information, the command characteristics of the voice signals are determined. In actual use, when the voice signal of the monitored target is received, the target device is controlled according to the voice signal to complete operations such as switching. For a single monitored target, the fluctuation value of the voice signal emitted by it in the time domain dimension is small, but in a campus environment, especially during get out of class breaks, the background is noisy and the voice signal of the monitored target may be interfered with, which is reflected in the increased difficulty of subsequently distinguishing the voice signals of the monitored targets, thereby causing errors in the identified control instructions and making it impossible to effectively control the target device. Based on this, the present invention considers conducting an overall analysis of the collected voice signals, determining the semantic relevance corresponding to the voice signals and the fluctuation value of the voice signals in the time domain dimension, providing a data basis for calculating the instruction characterization parameters, making the recognition of the monitored target voice signals more accurate, and improving the accuracy of the control instruction determination.

[0110] Specifically, the analysis module calculates the instruction representation parameters for the speech signal in the time domain segment, including:

[0111] The ratio of the semantic relevance to the reference semantic relevance is determined as a semantic relevance influencing parameter;

[0112] To determine the ratio of the reference fluctuation value to the fluctuation value as the fluctuation value influencing parameter;

[0113] The weighted sum of the semantic relevance influencing parameter and the fluctuation value influencing parameter is used to determine the instruction representation parameter.

[0114] Specifically, the benchmark semantic relevance is calculated in advance. Several semantic relevances of speech signals emitted by a single monitoring target are obtained in advance, and the average value of the several semantic relevances is calculated. 0.85 times of the average value is determined as the benchmark semantic relevance.

[0115] Specifically, the reference fluctuation value is calculated in advance. The speech signals in a classroom environment within several cycles are obtained in advance, and the fluctuation values ​​are analyzed. The average value of the fluctuation values ​​is calculated as the reference fluctuation value.

[0116] Specifically, the weight coefficient of the parameter affecting the semantic relevance is 0.64, and the weight coefficient of the parameter affecting the fluctuation value is 0.36.

[0117] Specifically, the analysis module determines the command clarity category of the voice signal, wherein:

[0118] If the instruction characterization parameter is less than the instruction characterization parameter threshold, determining that the instruction clarity category is a weak clarity instruction;

[0119] If the instruction characterization parameter is greater than or equal to the instruction characterization parameter threshold, it is determined that the instruction clarity category is a strong clarity instruction.

[0120] Specifically, the instruction characterization parameter threshold is selected within the interval [1.05, 1.15].

[0121] Specifically, the instruction control module analyzes the voice signal based on the instruction clarity category of the voice signal in the time domain segment to determine the control instruction, wherein:

[0122] If the command clarity category is a weakly clear command, the sight direction of each monitoring target is determined based on the monitoring data, the key monitoring target is determined based on the angle between the sight direction and the predetermined receiving direction, the lip shape data of the key monitoring target is determined synchronously, and the lip shape data is matched with the voice signal to obtain a text signal to determine the control command;

[0123] If the command clarity category is a strong clear command, the voice signal is directly converted into a text signal to determine the control command.

[0124] Specifically, the present invention determines the instruction clarity category of the voice signal by calculating the instruction characterization parameters of the voice signal in the time domain segment. When analyzing the voice signal, there will be a clearer voice signal. At this time, if the received voice signal is still subjected to complex analysis, this non-adaptive adjustment and analysis method makes the analysis efficiency of the voice signal low. Based on this, the present invention calculates the instruction characterization parameters, classifies the voice signal, directly determines the control instruction for the voice signal with strong and clear instructions, and performs signal matching on the voice signal with weak and clear instructions, so that the recognition of the monitored target voice signal is more accurate, and the accuracy of determining the control instruction is improved.

[0125] Specifically, the command control module determines the sight direction of each monitoring target, including:

[0126] Determine a facial reference point group of a monitoring target and construct a facial plane according to the facial reference point group;

[0127] Used to determine the direction of the vector perpendicular to the facial plane as the sight direction of the monitoring target;

[0128] The facial reference point group is three points within the facial contour area.

[0129] It can be understood that three points can form a plane. In some possible implementations, the eyes, the tip of the nose, and the corners of the mouth can be determined as reference points.

[0130] Specifically, the method of identifying facial reference points is not limited. For example, facial feature extraction of Dlib may be used, including the following steps:

[0131] Step S01, using Dlib to determine the face in the monitoring data. Usually, the method will return the location information of the face, including the left boundary, upper boundary, right boundary and lower boundary of the face;

[0132] Step S02, using the shape_predictor class of Dlib to predict the key feature points of the face. This usually requires a pre-trained model file (such as shapepredictor68_face_landmarks.dat), which can recognize 68 key points on the face, including the eyes and the tip of the nose;

[0133] Step S03, marking the identified feature points, and extracting the coordinate data of each feature point and saving it for subsequent analysis and processing, such as plane construction and other operations.

[0134] Specifically, the vertical vector of the facial plane is also the normal vector of the facial plane. It can be understood that the normal vector can be obtained by calculating the cross product of two vectors of the facial plane. In practical applications, the calculation of the normal vector is usually automatically completed by a computer vision library or three-dimensional graphics software. Those skilled in the art only need to call the corresponding function or method to obtain the normal vector of the facial plane, which will not be repeated here.

[0135] Specifically, the command control module determines the key monitoring target based on the angle between the line of sight and the predetermined receiving direction, including:

[0136] Used to determine the angle between the line of sight direction corresponding to the monitoring target and the predetermined receiving direction;

[0137] If the degree of the angle is less than the degree of the reference angle, it is determined that the monitoring target corresponding to the line of sight direction is a key monitoring target.

[0138] Specifically, the predetermined receiving direction is determined according to the voice collection unit and the monitoring unit. It can be understood that the voice collection unit can receive voice signals within an area, and the monitoring unit can obtain monitoring data within the area. Usually, the monitoring unit and the voice collection unit are set within 30 cm.

[0139] Due to the positional relationship between the monitoring unit and the voice collection unit, the predetermined receiving direction is determined based on the shooting direction of the monitoring unit, and the shooting direction of the monitoring unit is determined as the predetermined receiving direction.

[0140] Specifically, the purpose of setting the angle is to identify whether the line of sight direction of the monitored target is significantly offset from the predetermined receiving direction. Therefore, the angle is selected within the interval [30°, 90°].

[0141] Specifically, the instruction control module matches the lip data with the voice signal, including:

[0142] Determine a number of associated lip shape keywords based on the lip shape data;

[0143] To match each keyword in the voice signal with the lip shape keyword;

[0144] Converting the speech signal into a text signal, and removing unmatched keywords from the text signal;

[0145] If the keyword in the voice signal does not match any of the lip shape keywords, the keyword is determined to be an unmatched keyword.

[0146] It is understandable that there may be more than one key monitoring target, so in this implementation, lip-keyword matching is performed on the key monitoring target to make the recognition of the monitoring target voice signal more accurate.

[0147] Specifically, there is no limitation on the method of determining the lip-sync keywords. For example, it can be:

[0148] Deep learning technology: Using a deep learning framework, the speaker’s face is identified from the image, and the features of the mouth shape changes during continuous speaking are extracted. These features are input into the lip reading recognition model to identify the corresponding pronunciation, and finally determine a number of mouth shape keywords associated with the pronunciation. It is understandable that for mouth shape, the same mouth shape may be associated with multiple mouth shape keywords, which will not be elaborated here.

[0149] Dataset application: You can use special lip reading recognition datasets, such as LRW, LRW-1000 and Ou l uVS2. These datasets contain a large amount of video and audio data. Using the dataset, you can identify several lip shape keywords.

[0150] Wav2Lip technology: A deep learning-based audio and video synchronization technology that achieves high-precision lip synchronization by analyzing audio signals and video frames. It converts the input audio into a spectrogram, matches the best mouth shape position in the video frame, and performs lip shape transformation at that position based on the audio signal, and identifies several lip shape keywords based on the lip shape transformation.

[0151] The above method determines the lip shape to identify the solution of speech content from different angles and technical means, combines visual information and voice information, and improves the accuracy and robustness of recognition. It is understandable that accurate lip shape keywords may not be determined through lip shape data. Technical personnel in this field can choose from the above methods according to the application environment.

[0152] Specifically, the present invention determines control instructions based on key monitoring targets and lip shape data. When receiving multiple voice signals in a noisy environment, it is necessary to analyze the voice signals to obtain accurate control instructions. Based on this, the present invention considers real-time monitoring of target areas in the classroom, and determines key monitoring targets based on monitoring data, matches the lip shape data of the key monitoring targets with the voice signals, and then determines the control instructions, so that the recognition of the monitoring target voice signals is more accurate, thereby improving the accuracy of determining the control instructions.

[0153] Specifically, the instruction response module controls the predetermined device in the classroom based on the determined control instruction, including:

[0154] Used to determine control instructions according to text signals;

[0155] to determine the predetermined device in the classroom corresponding to the control instruction;

[0156] Used to send the control instruction to the predetermined device.

[0157] Specifically, the control instruction is an electrical signal, which changes according to the change of the control target. It only needs to control the corresponding action to be performed on the device, which will not be elaborated here.

[0158] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A comprehensive operation and maintenance management platform based on the Internet of Things, characterized in that: include: The Internet of Things acquisition module includes a voice acquisition unit arranged in a target area of ​​the classroom for acquiring voice signals and a monitoring unit for acquiring monitoring data; A data monitoring module, which is connected to the Internet of Things acquisition module, is used to analyze the voice signal in each time domain segment, and determine the instruction characteristics of the voice signal based on the analysis result, including the semantic relevance corresponding to the voice signal and the fluctuation value of the voice signal in the time domain dimension; An analysis module connected to the data monitoring module, for calculating a command characterization parameter for the voice signal in a time domain segment based on the command feature, so as to determine a command clarity category of the voice signal; An instruction control module is connected to the analysis module and is used to analyze the voice signal based on the instruction clarity category of the voice signal in the time domain segment to determine the control instruction, including: Determine the sight direction of each monitoring target based on the monitoring data, determine the key monitoring target based on the angle between the sight direction and the predetermined receiving direction, synchronously determine the lip shape data of the key monitoring target, match the lip shape data with the voice signal to obtain a text signal to determine the control instruction; Or, directly convert the voice signal into a text signal to determine the control instruction; The instruction response module is connected to the Internet of Things acquisition module and the instruction control module, and is used to control the predetermined equipment in the classroom based on the determined control instruction.

2. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The data monitoring module determines the semantic relevance corresponding to the speech signal, including: Used to convert the voice signals in each time domain segment into text information; Used to extract keywords from the text information; Used to determine the semantic relevance of the text information according to the keywords.

3. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The data monitoring module determines the fluctuation value of the voice signal in the time domain dimension, including: Used to determine the peak value of the speech signal in each time domain segment; The variance used to determine each peak value in the time domain segment is the fluctuation value.

4. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The analysis module calculates instruction representation parameters for the speech signal in the time domain segment, including: The ratio of the semantic relevance to the reference semantic relevance is determined as a semantic relevance influencing parameter; To determine the ratio of the reference fluctuation value to the fluctuation value as the fluctuation value influencing parameter; The weighted sum of the semantic relevance influencing parameter and the fluctuation value influencing parameter is used to determine the instruction representation parameter.

5. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The analysis module determines the command clarity category of the voice signal, wherein: If the instruction characterization parameter is less than the instruction characterization parameter threshold, determining that the instruction clarity category is a weak clarity instruction; If the instruction characterization parameter is greater than or equal to the instruction characterization parameter threshold, it is determined that the instruction clarity category is a strong clarity instruction.

6. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The command control module analyzes the voice signal based on the command clarity category of the voice signal in the time domain segment to determine the control command, wherein: If the command clarity category is a weakly clear command, the sight direction of each monitoring target is determined based on the monitoring data, the key monitoring target is determined based on the angle between the sight direction and the predetermined receiving direction, the lip shape data of the key monitoring target is determined synchronously, and the lip shape data is matched with the voice signal to obtain a text signal to determine the control command; If the command clarity category is a strong clear command, the voice signal is directly converted into a text signal to determine the control command.

7. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The command control module determines the sight direction of each monitoring target, including: Determine a facial reference point group of a monitoring target and construct a facial plane according to the facial reference point group; Used to determine the direction of the vector perpendicular to the facial plane as the sight direction of the monitoring target; The facial reference point group is three points within the facial contour area.

8. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1 is characterized in that: The command control module determines the key monitoring target based on the angle between the line of sight and the predetermined receiving direction. include, Used to determine the angle between the line of sight direction corresponding to the monitoring target and the predetermined receiving direction; If the degree of the angle is less than the degree of the reference angle, it is determined that the monitoring target corresponding to the line of sight direction is a key monitoring target.

9. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1, characterized in that: The command control module matches the lip shape data with the voice signal, including: Determine a number of associated lip shape keywords based on the lip shape data; To match each keyword in the voice signal with the lip shape keyword; Converting a speech signal into a text signal and removing unmatched keywords from the text signal; If the keyword in the voice signal does not match any of the lip shape keywords, the keyword is determined to be an unmatched keyword.

10. The integrated operation and maintenance management platform based on the Internet of Things according to claim 1, characterized in that: The instruction response module controls the predetermined device in the classroom based on the determined control instruction, including: Used to determine control instructions according to text signals; to determine the predetermined device in the classroom corresponding to the control instruction; Used to send the control instruction to the predetermined device.

Images