Security monitoring multi-modal interactive control system based on speech image multi-feature fusion

Abstract

This invention relates to the field of security monitoring technology. Specifically, it relates to a multimodal interactive control system for security monitoring based on multi-feature fusion of voice and image. It includes a feature extraction module, a region division module, a voice update module, a risk marking module, and a risk determination module. The feature extraction module acquires building information of the security area and, combined with this building information, extracts image and voice features of the security area through security devices. This invention achieves refined perception and analysis of the security area by deeply integrating building information with image and voice features, effectively improving the accuracy of anomaly identification and solving the problems of missed and false judgments caused by environmental factors. Simultaneously, the feature extraction module enables the synchronous acquisition and spatiotemporal alignment of building information, image features, and voice features, laying a data foundation for subsequent multi-feature fusion analysis.

Description

Technical Field

[0001] This invention relates to the field of security monitoring technology, and more specifically, to a multimodal interactive control system for security monitoring based on the fusion of multiple features of voice and image. Background Technology

[0002] Security monitoring systems aim to achieve real-time perception, anomaly identification, and risk management of the monitored area, ultimately preventing security incidents, tracing the process of events, and protecting the safety of people and property within the area. Currently, mainstream security monitoring still focuses on single-modal image monitoring, with some scenarios supplemented by simple audio acquisition devices for sound recording. However, recognition methods based solely on image features are easily affected by environmental factors. Low light, rain, fog, and obstructions can directly lead to the failure of image feature extraction, resulting in missed or false anomaly detections and making accurate risk identification impossible. Furthermore, simple audio acquisition is only used for recording purposes and does not combine the building features and scene attributes of the monitored area for zonal analysis and dynamic updates of audio features. This makes it difficult for audio information to effectively assist in anomaly detection and achieve early risk warnings from the sound dimension. To reduce this situation, a multimodal interactive control system for security monitoring based on multi-feature fusion of voice and image is proposed. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal interactive control system for security monitoring based on the fusion of multiple features of voice and image, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, a multimodal interactive control system for security monitoring based on multi-feature fusion of voice and image is provided, including a feature extraction module, a region division module, a voice update module, a risk marking module, and a risk determination module; The feature extraction module is used to obtain building information of the security area, and combine the building information with security devices to extract image and voice features of the security area; The region division module is used to identify human and building features in security areas based on image data, divide multiple security sub-regions according to building information and features, and establish a voice word database for each security sub-region. The voice update module is used to extract voice features of security sub-regions, set relevant thresholds for security sub-regions based on building features, perform correlation analysis between voice features and terminology database and compare relevant thresholds to update the voice terminology database. The risk marking module is used to set risky voices based on the voice word library of the security sub-area, and to determine the risk by comparing the voice features of the security area with the risky voices. When a voice is determined to be risky, the module combines image features to match the person to whom it belongs and marks him as a risky person. The risk determination module is used to compare the characteristics of people in the security sub-area with those of high-risk individuals. When the comparisons overlap, the security device is controlled to adjust the monitoring area and record the behavioral data of the high-risk individuals. At the same time, risk thresholds are set for different behavioral types based on the danger of risk terms. The behavioral data is compared with the corresponding risk thresholds. If the data exceeds the threshold, it is determined to be high-risk and the location is sent to the security personnel.

[0005] As a further improvement to this technical solution, the feature extraction module extracts building information of the security area by connecting to the control terminal of the security area. Building information includes static building information such as the building layout, structure, functional zoning, wall distribution, and entrance / exit locations of the security area; The system connects to security devices installed in the security area, acquires image data and extracts image features through the security devices, and simultaneously acquires audio data and extracts speech features, thus completing the synchronous extraction of image and speech features. The security device consists of image acquisition equipment and audio acquisition equipment.

[0006] As a further improvement to this technical solution, the region division module identifies the human and architectural features of the security area based on image data; The security zone is divided into multiple security sub-zones based on building information and features. At the same time, a dedicated voice vocabulary database for each security sub-zone is constructed based on the building information and features corresponding to that sub-zone.

[0007] As a further improvement to this technical solution, in the voice update module, the security device to which the voice features belong is determined, the installation location of the security device is determined simultaneously, and the security sub-region is matched based on the installation location of the security device to obtain the security sub-region corresponding to each security device, thereby matching the voice features obtained by the security device to the corresponding security sub-region.

[0008] As a further improvement to this technical solution, the voice update module sets relevant thresholds for security sub-areas based on building characteristics; The greater the difference in building features between security sub-regions, the higher the relevant threshold. The smaller the differences between building features, the lower the correlation threshold; For the same security sub-region, the speech features of the security sub-region are compared with the speech vocabulary database to obtain the correlation value between the speech features and the speech vocabulary database. Then the correlation value is compared with the correlation threshold. When the correlation value is greater than the correlation threshold, the speech feature is added to the speech vocabulary database to complete the update of the speech vocabulary database; Conversely, if the correlation value is less than the correlation threshold, it will not be included.

[0009] As a further improvement to this technical solution, the risk marking module collects damage-related voice entries, uses the collected voice entries as reference risk voices, and then, for each security sub-area, selects voice entries with security risks from the corresponding voice entry library of the security sub-area as relevant risk voices. By combining reference risk voices with relevant risk voices, personalized expansion of voice entries is carried out, thereby generating multiple exclusive risk voices for security sub-areas; Among them, the risk voice used for risk assessment in the security sub-area includes reference risk voice, related risk voice, and generated exclusive risk voice.

[0010] As a further improvement to this technical solution, the risk marking module performs risk determination by comparing the voice characteristics of the security area with the risk voices of each security sub-area; When a voice feature is determined to be a risky voice, the person to whom the voice feature belongs is matched with the image feature to obtain the person who made the voice feature, and the risky person is marked in the corresponding security sub-area according to the person's corresponding characteristics. Conversely, if the voice features are not identified as risky, monitoring will continue.

[0011] As a further improvement to this technical solution, the risk determination module compares the characteristics of the people identified in the security sub-area with the characteristics of the at-risk people. When the comparison matches and the person is determined to be a high-risk person, the monitoring angle, focus and tracking range of the security device are adjusted according to the location of the high-risk person, and the behavioral data of the high-risk person is recorded. Conversely, if the comparison does not match, monitoring should continue.

[0012] As a further improvement to this technical solution, the risk determination module sets risk thresholds for various behavior types based on the risky voices emitted by the risky individuals. The higher the danger of risky speech, the lower the risk threshold of the corresponding behavior type; The lower the risk level of a risky speech, the higher the risk threshold for the corresponding behavior type. The behavioral data of high-risk individuals is compared with the corresponding risk thresholds for various behavioral types. When the behavioral data of a corresponding behavioral type exceeds the risk threshold, it is judged as high risk, and the real-time location of the high-risk individual is sent to security personnel. Conversely, if the behavioral data for the corresponding behavior type does not exceed the risk threshold, monitoring will continue. When a security device loses its image acquisition function, it is immediately identified as high-risk.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This multimodal interactive control system for security monitoring based on multi-feature fusion of voice and image achieves refined perception and analysis of security areas by deeply integrating building information with image and voice features. This effectively improves the accuracy of anomaly identification and solves the problems of missed and false judgments caused by environmental factors. At the same time, the feature extraction module realizes the synchronous acquisition and spatiotemporal alignment of building information, image features and voice features, laying a data foundation for subsequent multi-feature fusion analysis. Furthermore, based on building information and building features, the system performs refined division of security sub-areas and constructs a dedicated voice term library based on the scene attributes of each sub-area. This gives voice feature analysis strong scene adaptability, avoids the drawbacks of applying the same recognition standard to different functional areas, and greatly improves the pertinence and effectiveness of voice feature analysis.

[0014] 2. In this multimodal interactive control system for security monitoring based on the fusion of multiple features of voice and image, voice features are accurately matched to corresponding security sub-areas. Differentiated thresholds are set according to the differences in building features within the sub-areas to determine the effectiveness of voice features and supplement the vocabulary library. This allows the voice vocabulary library to be continuously improved as the monitoring scene changes, always maintaining a high degree of adaptability to the regional scene. This effectively solves the problem that traditional voice analysis systems are fixed and cannot adapt to scene changes, providing a reliable foundation for the accurate determination of subsequent risk voices. At the same time, the risk marking module constructs a three-layer risk voice system of reference risk voices, related risk voices, and exclusive risk voices. This retains the unified judgment benchmark for general destructive voices while combining the scene characteristics of each sub-area to realize the personalized expansion of risk voices. This makes risk voice judgment both universal and scene-specific, realizing cross-modal association from risk voice recognition to accurate matching of risky persons, and completing the rapid location of risk subjects. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the multimodal interactive control system for security monitoring based on multi-feature fusion of voice and image according to the present invention; Figure 2 This is a flowchart illustrating the feature extraction module of the present invention; Figure 3 This is a flowchart illustrating the region division module of the present invention; Figure 4 This is a flowchart illustrating the voice update module of the present invention; Figure 5 This is a flowchart illustrating the risk marking module of the present invention; Figure 6 This is a flowchart illustrating the risk determination module of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figures 1-6 As shown, the purpose of this embodiment is to provide a multimodal interactive control system for security monitoring based on the fusion of multiple features of voice and image, including a feature extraction module, a region division module, a voice update module, a risk marking module, and a risk determination module; The feature extraction module is used to obtain building information of the security area, and then extract image and voice features of the security area by combining the building information with security devices. In the feature extraction module, the building information of the security area is extracted by connecting to the control terminal of the security area. Building information includes static building information such as the building layout, structure, functional zoning, wall distribution, and entrance / exit locations of the security area; The system connects to security devices installed in the security area, acquires image data and extracts image features through the security devices, and simultaneously acquires audio data and extracts speech features. During the acquisition process, the timestamps of the image data and audio data are aligned to complete the synchronous extraction of image features and speech features. The security device consists of image acquisition equipment and audio acquisition equipment.

[0018] The region segmentation module is used to identify human and building features in security areas based on image data, divide multiple security sub-regions according to building information and features, and establish a voice vocabulary library for each security sub-region. In the area segmentation module, the characteristics of people and buildings in the security area are identified based on image data; Based on the collected image data, the image recognition algorithm simultaneously identifies the characteristics of people and buildings within the security area. The characteristics of people include face, body shape, posture, and trajectory features, while the characteristics of buildings include building outline, area boundary, fixed facilities, and functional attributes. The algorithm accurately extracts and classifies the two types of features. The security zone is divided into multiple security sub-zones based on building information and features. At the same time, a dedicated voice vocabulary database for each security sub-zone is constructed based on the building information and features corresponding to that sub-zone.

[0019] Based on the building information obtained in the early stage, the identified building features are compared and calibrated to ensure that the building information is consistent with the actual identified building features; Subsequently, the zoning principle was determined, using the building structure (wall distribution, building outline) as the physical boundary and the functional attribute characteristics (functional zoning) as the logical basis, taking into account the location of entrances and exits and the distribution of fixed facilities, to reasonably divide the entire security area. At the same time, during the division process, it is necessary to ensure that the boundaries of each sub-area are clear and the scope is reasonable, and that it has the conditions for independent monitoring and management. In the end, the entire security area was divided into multiple independent and controllable security sub-areas. For each security sub-zone after division, the building information corresponding to the sub-zone (such as functional zoning type, fixed facility type, entrance and exit location) and the identified building characteristics of the sub-zone (such as area boundary characteristics, functional attribute characteristics) are retrieved first. Subsequently, based on the needs of the sub-area scenarios, voice entries suitable for that area are selected. For example, in the office area, entries such as "office conversation, equipment operation sound, access control opening and closing sound" are selected; in the warehouse area, entries such as "goods handling sound, equipment start-up and shutdown sound" are selected; and in the passageway area, entries such as "footsteps sound, voice sound" are selected. Then, the selected voice entries are organized, coded, and entered into a dedicated entry library. Each sub-area corresponds to an independent voice entry library, clarifying the relationship between each entry and the building information and building features of that sub-area, thus completing the construction of a dedicated voice entry library for each security sub-area.

[0020] The voice update module is used to extract voice features of security sub-regions, set relevant thresholds for security sub-regions based on building features, perform correlation analysis between voice features and terminology database and compare relevant thresholds to update the voice terminology database; In the voice update module, the security device to which the voice features belong is determined, and the installation location of the security device is simultaneously determined. Based on the installation location of the security device, a matching process is performed with the security sub-region to obtain the security sub-region corresponding to each security device. Thus, the voice features obtained by the security device are matched to the corresponding security sub-region. The steps are as follows: The system identifies the security device from which the current voice feature originates, simultaneously acquires the coordinates of the device's installation location, performs spatial location matching between this location and all pre-defined security sub-regions, determines which sub-region the installation location falls within, and thus identifies the unique security sub-region corresponding to the device. After location matching is completed, the voice features collected by the device are assigned to and bound to the corresponding security sub-region, achieving accurate classification of voice features by security sub-region.

[0021] In the voice update module, relevant thresholds are set for security sub-areas based on building characteristics; The greater the difference in building features between security sub-regions, the higher the relevant threshold. The smaller the differences between building features, the lower the correlation threshold; For the same security sub-region, the speech features of the security sub-region are compared with the speech word library to obtain the correlation value between the speech features and the speech words in the speech word library. By calculating the similarity, the correlation value between the current speech features and all existing speech words in the speech word library is obtained to obtain a quantitative matching degree index, which provides a data basis for subsequent threshold comparison. Then the relevant values ​​are compared with the relevant thresholds; When the correlation value is greater than the correlation threshold, the speech feature is determined to be a valid word and is added to the speech word library of the corresponding security sub-region to complete the dynamic update of the word library; Conversely, if the correlation value is less than the correlation threshold, the speech feature is determined to be an invalid entry and will not be included in the speech entry database to maintain the purity of the entry database.

[0022] The risk labeling module is used to set risky voices based on the voice vocabulary library of the security sub-area, and to determine the risk by comparing the voice features of the security area with the risky voices. When a voice is determined to be risky, the module combines image features to match the person to whom it belongs and labels him as a risky person. In the risk labeling module, voice entries related to damage are collected, and the collected voice entries are used as reference risk voices. Then, for each security sub-area, voice entries with security risks are selected from the corresponding voice entry library of the security sub-area as relevant risk voices. A wide range of audio entries related to destructive behavior are collected, covering typical scenarios such as violent destruction, facility damage, and malicious intrusion, including but not limited to sounds of breaking glass, picking locks, fighting, malicious impacts, and threatening and abusive language. The collected audio entries are then screened, deduplicated, and standardized, and invalid and irrelevant entries are removed. The selected valid destructive audio entries are used as reference risk audio to form a general risk audio set, providing a unified benchmark for risk assessment in each sub-region. By combining reference risk voices with relevant risk voices, personalized expansion of voice entries is carried out, thereby generating multiple exclusive risk voices for security sub-areas; For each defined security sub-zone, the dedicated voice term library corresponding to that sub-zone is retrieved. At the same time, the building information (functional zoning, fixed facilities) and building characteristics (area attributes, facility types) of the sub-zone are combined to select voice terms with security risks that fit the scenario of that sub-zone from the voice term library as relevant risk voice terms. Among them, the risk voice used for risk assessment in the security sub-area includes reference risk voice, related risk voice, and generated exclusive risk voice.

[0023] The reference risk voice is fused with the relevant risk voice of each sub-area. A voice semantic extension algorithm is used to personalize the extension based on the characteristics of the sub-area scene, generating multiple exclusive risk voices adapted to the sub-area (e.g., the sound of breaking glass is extended to the sound of breaking glass in the office area and the sound of breaking glass in the warehouse area, and the voice recognition dimension is optimized based on the regional environment). Finally, the risk voice set used for risk determination in each security sub-area includes the reference risk voice, the relevant risk voice of the area, and the generated exclusive risk voice.

[0024] In the risk labeling module, the voice characteristics of the security area are compared with the risk voices of each security sub-area to determine the risk. When a voice feature is determined to be a risky voice, the person to whom the voice feature belongs is matched with the image feature to obtain the person who made the voice feature, and the risky person is marked in the corresponding security sub-area according to the person's corresponding characteristics. Immediately retrieve the synchronously acquired image features, combine them with the sound source location of the voice features, locate the person in the corresponding position in the image, bind the person's features (face, body shape, posture, etc.) with the voice features, mark the person as a risk person in the security sub-area to which the voice features belong, and complete the accurate location of the risk subject. Conversely, if the voice features are not identified as risky, monitoring will continue.

[0025] The risk determination module is used to compare the characteristics of people in the security sub-area with those of high-risk individuals. When the comparisons overlap, the security device is controlled to adjust the monitoring area and record the behavioral data of the high-risk individual. At the same time, risk thresholds are set for different behavioral types based on the danger of risk terms. The behavioral data is compared with the corresponding risk thresholds. If the data exceeds the threshold, it is judged as high-risk and the location is sent to the security personnel.

[0026] In the risk assessment module, the characteristics of individuals identified within the security sub-area are compared with the characteristics of individuals at risk. When the comparison matches and the person is confirmed as a high-risk individual, the security device will immediately adjust the monitoring angle, focus and tracking range based on the high-risk individual's real-time location to achieve continuous focused tracking of the high-risk individual and simultaneously record the high-risk individual's behavioral data, including movement trajectory, dwell time, action type, contact with target and other information. Conversely, if the comparison does not match, monitoring should continue.

[0027] In the risk assessment module, risk thresholds are set for various behavior types based on the risky voice messages issued by the risky individuals. The higher the danger of risky speech, the lower the risk threshold of the corresponding behavior type; The lower the risk level of a risky speech, the higher the risk threshold for the corresponding behavior type. The behavioral data of high-risk individuals is compared with the corresponding risk thresholds for various behavioral types. When the behavioral data of a corresponding behavioral type exceeds the risk threshold, it is judged as high risk, and the real-time location of the high-risk individual is sent to security personnel. Conversely, if the behavioral data for the corresponding behavior type does not exceed the risk threshold, monitoring will continue. When a security device loses its image acquisition function, it is immediately identified as high-risk.

[0028] The system monitors the working status of security devices in real time. When a security device loses its image acquisition function, or when image data is interrupted or abnormal, it directly determines that the current area is in a high-risk state in order to ensure security. It immediately sends the abnormal status and location information to security personnel without relying on behavioral data and threshold comparison.

[0029] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal interactive control system for security monitoring based on voice and image multi-feature fusion, characterized in that: It includes a feature extraction module, a region segmentation module, a voice update module, a risk labeling module, and a risk determination module; The feature extraction module is used to obtain building information of the security area, and combine the building information with security devices to extract image and voice features of the security area; The region division module is used to identify human and building features in security areas based on image data, divide multiple security sub-regions according to building information and features, and establish a voice word database for each security sub-region. The voice update module is used to extract voice features of security sub-regions, set relevant thresholds for security sub-regions based on building features, perform correlation analysis between voice features and terminology database and compare relevant thresholds to update the voice terminology database. The risk marking module is used to set risky voices based on the voice word library of the security sub-area, and to determine the risk by comparing the voice features of the security area with the risky voices. When a voice is determined to be risky, the module combines image features to match the person to whom it belongs and marks him as a risky person. The risk determination module is used to compare the characteristics of people in the security sub-area with those of high-risk individuals. When the comparisons overlap, the security device is controlled to adjust the monitoring area and record the behavioral data of the high-risk individuals. At the same time, risk thresholds are set for different behavioral types based on the danger of risk terms. The behavioral data is compared with the corresponding risk thresholds. If the data exceeds the threshold, it is determined to be high-risk and the location is sent to the security personnel.

2. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the feature extraction module, building information of the security area is extracted by connecting to the control terminal of the security area. Building information includes static building information such as the building layout, structure, functional zoning, wall distribution, and entrance / exit locations of the security area; The system connects to security devices installed in the security area, acquires image data and extracts image features through the security devices, and simultaneously acquires audio data and extracts speech features, thus completing the synchronous extraction of image and speech features. The security device consists of image acquisition equipment and audio acquisition equipment.

3. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the area division module, the characteristics of people and buildings in the security area are identified based on image data; The security zone is divided into multiple security sub-zones based on building information and features. At the same time, a dedicated voice vocabulary database for each security sub-zone is constructed based on the building information and features corresponding to that sub-zone.

4. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the voice update module, the security device to which the voice features belong is determined, the installation location of the security device is determined simultaneously, and the security sub-region is matched based on the installation location of the security device to obtain the security sub-region corresponding to each security device, thereby matching the voice features obtained by the security device to the corresponding security sub-region.

5. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the voice update module, relevant thresholds are set for security sub-areas based on building characteristics; The greater the difference in building features between security sub-regions, the higher the relevant threshold. The smaller the differences between building features, the lower the correlation threshold; For the same security sub-region, the speech features of the security sub-region are compared with the speech vocabulary database to obtain the correlation value between the speech features and the speech vocabulary database. Then the correlation value is compared with the correlation threshold. When the correlation value is greater than the correlation threshold, the speech feature is added to the speech vocabulary database to complete the update of the speech vocabulary database; Conversely, if the correlation value is less than the correlation threshold, it will not be included.

6. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the risk labeling module, damage-related voice entries are collected, and the collected voice entries are used as reference risk voices. Then, for each security sub-area, voice entries with security risks are selected from the voice entry library corresponding to the security sub-area as relevant risk voices. By combining reference risk voices with relevant risk voices, personalized expansion of voice entries is carried out, thereby generating multiple exclusive risk voices for security sub-areas; Among them, the risk voice used for risk assessment in the security sub-area includes reference risk voice, related risk voice, and generated exclusive risk voice.

7. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the risk marking module, the voice characteristics of the security area are compared with the risk voices of each security sub-area to determine the risk. When a voice feature is determined to be a risky voice, the person to whom the voice feature belongs is matched with the image feature to obtain the person who made the voice feature, and the risky person is marked in the corresponding security sub-area according to the person's corresponding characteristics. Conversely, if the voice features are not identified as risky, monitoring will continue.

8. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the risk determination module, the characteristics of the people identified in the security sub-area are compared with the characteristics of the at-risk people; When the comparison matches and the person is determined to be a high-risk person, the monitoring angle, focus and tracking range of the security device are adjusted according to the location of the high-risk person, and the behavioral data of the high-risk person is recorded. Conversely, if the comparison does not match, monitoring should continue.

9. The security monitoring multimodal interactive control system based on voice and image multi-feature fusion according to claim 1, characterized in that: In the risk determination module, risk thresholds are set for various behavior types based on the risky voices emitted by the risky individuals. The higher the danger of risky speech, the lower the risk threshold of the corresponding behavior type; The lower the risk level of a risky speech, the higher the risk threshold for the corresponding behavior type. The behavioral data of high-risk individuals is compared with the corresponding risk thresholds for various behavioral types. When the behavioral data of a corresponding behavioral type exceeds the risk threshold, it is judged as high risk, and the real-time location of the high-risk individual is sent to security personnel. Conversely, if the behavioral data for the corresponding behavior type does not exceed the risk threshold, monitoring will continue. When a security device loses its image acquisition function, it is immediately identified as high-risk.

Images