Future community management system and method based on the integration of Internet of Things and multimedia

Abstract

The present application discloses a future community management system and method based on the fusion of the Internet of Things and multimedia. After collecting and transmitting the video data of children's play status and the sound data of children's play respectively through the camera and sound detector of the shared playground, the data processing and analysis algorithm based on artificial intelligence and multimedia fusion technology is introduced in the future community background management server to analyze and fuse the semantic temporal characteristics of children's play behavior and the semantic temporal characteristics of children's play sound, so as to comprehensively characterize the children's play status, thereby monitoring the children's play status and risk warning. In this way, the children's play status can be monitored in real time in the future community through the fusion semantics of multimedia to monitor and warn children's outdoor safety, thereby reducing the safety hazards of children's outdoor play and realizing more intelligent future community management.

Description

Technical Field

[0001] The present application relates to the field of intelligent management, and more specifically, to a future community management system and method based on the integration of the Internet of Things and multimedia. Background Art

[0002] With the rapid development of science and technology, future communities have become an important part of urban modernization. Future community management refers to the intelligent and efficient management of communities based on advanced technologies such as the Internet of Things, artificial intelligence, and big data. With the continuous advancement of science and technology, future community management systems will become more intelligent and convenient, providing residents with a safer and more convenient living environment.

[0003] In community management, child safety is an important monitoring project. For example, children are prone to play risks when playing in community playgrounds. In traditional communities, cameras are installed near playgrounds and the surveillance images collected by the cameras are manually observed. If threatening or potentially dangerous behaviors are observed, the management staff will issue safety reminders. However, this management model requires manual participation, which is not only time-consuming but also has poor management effects, because it is impossible for people to always pay attention to the surveillance images.

[0004] Therefore, an optimized future community management system is expected. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a future community management system and method based on the Internet of Things and multimedia fusion, which respectively collects children's play status video data and children's play sound data through the camera and sound detector of the shared playground and transmits them, and then introduces data processing and analysis algorithms based on artificial intelligence and multimedia fusion technology in the future community background management server to analyze and fuse the semantic temporal characteristics of children's play behavior and the semantic temporal characteristics of children's play sound, so as to comprehensively characterize the children's play status, thereby monitoring the children's play status and risk warning. In this way, the children's play status can be monitored in real time in the future community through the fusion semantics of multimedia to monitor and warn children's outdoor safety, thereby reducing the safety hazards of children's outdoor play and realizing more intelligent future community management.

[0006] According to one aspect of the present application, a future community management system based on the integration of the Internet of Things and multimedia is provided, which includes:

[0007] A sensor network, wherein the sensor network includes a camera and a sound detector deployed in a shared playground, the camera is used to collect video data of children's play status, and the sound detector is used to collect sound data of children's play;

[0008] A data compression module, used for compressing the children's play state video data and the children's play sound data to obtain compressed children's play state video data and compressed children's play sound data;

[0009] An Internet of Things communication network, used for transmitting the compressed children's play status video data and the compressed children's play sound data to a future community backend management server;

[0010] The future community backend management server includes:

[0011] A children's play multimedia data decoding module, used for decompressing the compressed children's play state video data and the compressed children's play sound data to obtain the children's play state video data and the children's play sound data;

[0012] A children's play multimedia data feature analysis module, used to extract features from the children's play state video data and the children's play sound data to obtain a children's play behavior time series dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector;

[0013] A children's play state multimodal fusion representation module, used for inputting the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector into a multimedia aggregator based on the CLIP model to obtain a multimodal children's play state representation matrix;

[0014] The child risk warning module is used to determine whether to generate a risk warning prompt based on the multimodal children's play state representation matrix.

[0015] According to another aspect of the present application, a future community management method based on the integration of the Internet of Things and multimedia is provided, which includes:

[0016] Collect video data of children's play status and sound data of children's play;

[0017] Compressing the children's play state video data and the children's play sound data to obtain compressed children's play state video data and compressed children's play sound data;

[0018] Transmitting the compressed children's play state video data and the compressed children's play sound data to the future community backend management server;

[0019] On the future community backend management server, the compressed children's play state video data and the compressed children's play sound data are decompressed to obtain the children's play state video data and the children's play sound data;

[0020] On the future community backend management server, feature extraction is performed on the children's play state video data and the children's play sound data to obtain a children's play behavior time series dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector;

[0021] In the future community backend management server, the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector are input into a multimedia amalgamator based on the CLIP model to obtain a multi-modal children's play state representation matrix; and

[0022] In the future community backend management server, it is determined whether to generate a risk warning prompt based on the multimodal children's play state representation matrix.

[0023] Compared with the prior art, the present application provides a future community management system and method based on the fusion of the Internet of Things and multimedia. After collecting and transmitting the video data of children's play status and the sound data of children's play respectively through the camera and sound detector of the shared playground, the data processing and analysis algorithm based on artificial intelligence and multimedia fusion technology is introduced in the future community background management server to analyze and fuse the semantic temporal characteristics of children's play behavior and the semantic temporal characteristics of children's play sound, so as to comprehensively characterize the children's play status, thereby monitoring the children's play status and risk warning. In this way, the children's play status can be monitored in real time in the future community through the fusion semantics of multimedia to monitor and warn children's outdoor safety, thereby reducing the safety hazards of children's outdoor play and realizing more intelligent future community management. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 A block diagram of a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application;

[0026] Figure 2 This is a system architecture diagram of a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application;

[0027] Figure 3 A block diagram of the training phase of a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application;

[0028] Figure 4 A block diagram of a future community backend management server in a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application;

[0029] Figure 5 It is a block diagram of a children's play multimedia data feature analysis module in a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application;

[0030] Figure 6 It is a flow chart of a future community management method based on the integration of the Internet of Things and multimedia according to an embodiment of the present application; DETAILED DESCRIPTION

[0031] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0032] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0033] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0034] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0035] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0036] With the rapid development of science and technology, the application of Internet of Things and multimedia technology in community management is becoming more and more extensive. Internet of Things technology realizes real-time data collection and transmission by connecting various sensors and devices, while multimedia technology provides richer information expression and processing capabilities by integrating multiple information forms (such as video, audio, etc.). The combination of the two provides a strong technical support for the construction of intelligent and automated future community management systems. The future community management system based on the integration of Internet of Things and multimedia can realize the comprehensive representation of the semantics of children's play status by analyzing and integrating the semantic features in the video data of children's play status and the sound data of children's play in the shared playground, so as to more accurately monitor the children's play status and predict the children's play risks for early warning.

[0037] In the technical solution of the present application, a future community management system based on the integration of the Internet of Things and multimedia is proposed. Figure 1 It is a block diagram of a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application. Figure 2 : is a system architecture diagram of a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, a future community management system 300 based on the integration of the Internet of Things and multimedia includes: a sensor network 310, wherein the sensor network includes a camera and a sound detector deployed in a shared playground, the camera is used to collect video data of children's play status, and the sound detector is used to collect sound data of children's play; a data compression module 320, used to compress the video data of the children's play status and the sound data of the children's play to obtain compressed video data of the children's play status and compressed sound data of the children's play; an Internet of Things communication network 330, used to transmit the compressed video data of the children's play status and the compressed sound data of the children's play to the future community background management server 340.

[0038] In particular, the sensor network 310, wherein the sensor network includes a camera and a sound detector deployed in a shared playground, the camera is used to collect video data of children's play status, and the sound detector is used to collect sound data of children's play. It should be understood that the actions of children in the video can be identified and analyzed through the video data of children's play status, and their activity level and behavioral characteristics can be understood. In addition, by analyzing the voice characteristics of children through sound data, their emotional state can be inferred. In one example, the video and sound data are fused and analyzed to more comprehensively understand the children's play status, for example, by combining the actions and expressions in the video with the emotional characteristics of the sound, a more accurate judgment can be obtained.

[0039] In particular, the data compression module 320 is used to compress the children's play state video data and the children's play sound data to obtain compressed children's play state video data and compressed children's play sound data. Wherein, compressing the children's play state video data and the children's play sound data helps to optimize the data storage and transmission process.

[0040] In particular, the IoT communication network 330 is used to transmit the compressed children's play state video data and the compressed children's play sound data to the future community backend management server. In particular, in a specific example of the present application, Figure 4 As shown, the future community backend management server 340 includes: a children's play multimedia data decoding module 341, which is used to decompress the compressed children's play state video data and the compressed children's play sound data to obtain the children's play state video data and the children's play sound data; a children's play multimedia data feature analysis module 342, which is used to extract features from the children's play state video data and the children's play sound data respectively to obtain a children's play behavior temporal dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector; a children's play state multimodal fusion representation module 343, which is used to input the children's play behavior temporal dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector into a multimedia aggregator based on the CLIP model to obtain a multimodal children's play state representation matrix; a child risk warning module 344, which is used to determine whether to generate a risk warning prompt based on the multimodal children's play state representation matrix.

[0041] Specifically, the children's play multimedia data decoding module 341 is used to decompress the compressed children's play state video data and the compressed children's play sound data to obtain the children's play state video data and the children's play sound data. By decompressing the compressed children's play state video data and the children's play sound data, the original quality and information of the data can be restored, making the data more suitable for subsequent analysis, processing and display.

[0042] Specifically, the children's play multimedia data feature analysis module 342 is used to extract features from the children's play state video data and the children's play sound data to obtain a children's play behavior time series dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector. In particular, in a specific example of the present application, Figure 5As shown, the children's play multimedia data feature analysis module 342 includes: a children's play state key frame sampling unit 3421, which is used to perform key frame sampling on the children's play state video data to obtain a time series of children's play state key frames; a children's play behavior semantic feature extraction unit 3422, which is used to extract children's play behavior state features from each children's play state key frame in the time series of the children's play state key frames to obtain a sequence of children's play behavior semantic feature vectors; a children's play behavior semantic time series dynamic aggregation representation unit 3423, which is used to pass the sequence of children's play behavior semantic feature vectors through a children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector; a children's play sound time-frequency feature expression unit 3424, which is used to perform time-frequency graph conversion on the children's play sound data and then perform semantic feature expression on the obtained children's play sound time-frequency graph to obtain the children's play sound time-frequency representation vector.

[0043] More specifically, the children's play state key frame sampling unit 3421 is used to perform key frame sampling on the children's play state video data to obtain a time series of key frames of the children's play state. Considering that the children's play state video data usually contains a large amount of redundant information about children's play, if the entire video is analyzed and processed, a large amount of computing resources will be consumed, a large amount of storage space will be wasted, and most of the video content is repeated or insignificant. Based on this, in order to be able to extract the most representative and critical video frames in the video, thereby realizing effective analysis and monitoring of the children's play state and reducing storage requirements, in the technical solution of the present application, the children's play state video data is further key frame sampled to obtain a time series of key frames of the children's play state.

[0044] More specifically, the children's play behavior semantic feature extraction unit 3422 is used to extract the children's play behavior state features of each children's play state key frame in the time series of the children's play state key frame to obtain a sequence of children's play behavior semantic feature vectors. It should be understood that since each children's play state key frame in the time series of the children's play state key frame has behavioral semantic features about the children's play state at a specific moment, the correlation relationship between these behavioral semantic features in time series can be used to analyze the children's play state well and to warn of hidden behavioral hazards. Based on this, in order to capture the behavioral semantic features of the children's play state, so as to better represent and analyze the semantics of the children's play state, in the technical solution of the present application, each children's play state key frame in the time series of the children's play state key frame is further extracted by a children's play behavior state feature extractor based on MobileNetV3 to obtain a sequence of children's play behavior semantic feature vectors. Through the feature extractor processing based on MobileNetV3, semantic features related to the children's play behavior status under each children's play status key frame can be extracted. These features can better represent the children's behavior patterns and action characteristics, and provide more powerful support for subsequent children's behavior timing analysis, action recognition and safety hazard monitoring and early warning.

[0045] More specifically, the children's play behavior semantic time series dynamic aggregation representation unit 3423 is used to pass the sequence of the children's play behavior semantic feature vectors through the children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector. It should be understood that each children's play behavior semantic feature vector in the sequence of the children's play behavior semantic feature vectors respectively represents the children's play behavior semantic feature information at each moment, and there is a time series correlation relationship and dynamic change characteristics between these children's play behavior semantics. In other words, children's behavior often has time series, and different behavior patterns and changes will appear over time. Therefore, in order to better analyze the full-time domain behavior characteristics and semantics of the children's play state, so as to more accurately carry out early warning of children's play risks, in the technical solution of the present application, the sequence of the children's play behavior semantic feature vectors is further passed through the children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector. Through the processing of the children's play behavior time series message transmission aggregation analysis network, it is possible to conduct a more in-depth time series analysis and dynamic aggregation of the children's play behavior characteristics at each moment, so as to capture the dynamic process of the children's play behavior semantic characteristics changing over time, so as to better understand and characterize the children's behavior patterns and time series changes, improve the accuracy and effect of the analysis and identification of children's play behavior, and more accurately perform risk detection and early warning of children's play. In a specific example of the present application, the sequence of the children's play behavior semantic feature vectors is passed through the children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector, including: calculating the first The semantic feature vectors of children's play behaviors are summed up by position to obtain the children's play behavior neighbor semantic fusion feature vector; the sum of the trainable preset hyperparameter and one is used as the weighting coefficient for the first The position feature values ​​in the children's play behavior semantic feature vectors are weighted to obtain a weighted optimized children's play behavior semantic feature vector; the children's play behavior neighbor semantic fusion feature vector and the weighted optimized children's play behavior semantic feature vector are summed by position and processed by a multilayer perceptron to obtain a first The invention relates to a children's play behavior semantic time series message transmission feature vector corresponding to each of the children's play behavior semantic feature vectors; and the positional sum of the children's play behavior semantic time series message transmission feature vectors corresponding to each of the children's play behavior semantic feature vectors in the sequence of the children's play behavior semantic feature vectors is calculated to obtain the children's play behavior time series dynamic aggregation semantic representation vector.

[0046] In summary, in the above embodiment, the sequence of the children's play behavior semantic feature vectors is passed through the children's play behavior time series message passing aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector, including: the sequence of the children's play behavior semantic feature vectors is passed through the children's play behavior time series message passing aggregation analysis network to process with the following message passing aggregation analysis formula to obtain the children's play behavior time series dynamic aggregation semantic representation vector; wherein, the message passing aggregation analysis formula is:

[0047]

[0048] in, is the first in the sequence of the children's play behavior semantic feature vectors semantic feature vectors of children's play behaviors, is the first in the sequence of the children's play behavior semantic feature vectors The semantic feature vectors of children's play behaviors, is a trainable preset hyperparameter, represents a multi-layer perceptron, is the number of feature vectors in the sequence of the children's play behavior semantic feature vectors, A semantic representation vector is dynamically aggregated for the children's play behavior time series.

[0049] More specifically, the children's play sound time-frequency feature expression unit 3424 is used to perform semantic feature expression on the obtained children's play sound time-frequency graph after converting the children's play sound data into a time-frequency graph to obtain the children's play sound time-frequency representation vector. Considering that there is semantic information about the sound state of children's play in the children's play sound data, the semantic information of this sound state can also provide a basis for the children's play state and risk. However, since the semantic information in the children's play sound data is relatively complex and there is more noise in the shared playground, it is difficult to accurately and effectively capture the characteristics of the children's play sound data. Therefore, in the technical solution of the present application, by converting the sound signal into a form represented in the time-frequency domain, the frequency characteristics and time characteristics of the sound data are better understood and analyzed, so as to capture the sound semantic characteristics of the children's play state in the sound data. Specifically, in the technical solution of the present application, first, the children's play sound data is subjected to wavelet transformation to obtain the children's play sound time-frequency graph. Wavelet transform can convert sound signals from time domain to frequency domain, provide detailed information of sound signals in time and frequency, help analyze the frequency components and time series characteristics of sound signals, such as frequency distribution, energy distribution, etc., which helps to reflect the changes of different frequency components over time, so as to identify different frequency features and spectrum lines in sound signals, and provide richer feature information for subsequent sound analysis and semantic recognition. Furthermore, in order to extract key sound features from children's play sound data, so as to characterize the semantics of the sound state when children are playing, in the technical solution of the present application, the children's play sound time-frequency diagram is further passed through the children's play sound state time-frequency feature extractor based on the DenseNet model to obtain the children's play sound time-frequency representation vector. It should be understood that the DenseNet model, as a deep learning model, has a strong ability in extracting features and can effectively extract meaningful features from complex sound data. By processing the sound frequency graph through the DenseNet model, the time-frequency graph data of the sound can be converted into a higher-level and more abstract representation, and the complex time-frequency features in the sound data can be captured, such as the time-frequency semantic information of children's play sounds, providing better feature representation for subsequent sound analysis, children's play status identification, and safety hazard detection and warning.

[0050] It is worth mentioning that in other specific examples of the present application, the children's play state video data and the children's play sound data can also be feature extracted respectively in other ways to obtain the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector, for example: input the children's play state video data and the children's play sound data; use a pre-trained convolutional neural network (CNN) model (such as ResNet, VGG, etc.) to extract features from video frames; perform time series dynamic aggregation on the extracted video frame features, and use a recurrent neural network (RNN), a long short-term memory network (LSTM) or an attention mechanism to capture the temporal relationship between video frames to obtain the children's play behavior time series dynamic aggregation semantic representation vector; use sound processing technology (such as short-time Fourier transform) to convert the sound data into a time-frequency representation; extract the features of the sound data in time and frequency; integrate the time-frequency features of the sound data into a vector representation, and average pooling or maximum pooling can be performed on the features of different time periods, or time series modeling can be performed using a recurrent neural network (RNN) or other models to obtain the time-frequency representation vector of the children's play sound.

[0051] Specifically, the children's play state multimodal fusion representation module 343 is used to input the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector into a multimedia fusion device based on the CLIP model to obtain a multimodal children's play state representation matrix. Considering that the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector respectively represent the behavior semantic time series aggregation characteristics and sound time-frequency semantic characteristics of the children's play state, the two belong to different modal data types. Therefore, in order to be able to combine the children's play behavior time series dynamic features and sound time-frequency features to provide more comprehensive and comprehensive children's play state information, it is necessary to fuse these two types of children's play state feature information from different data sources together to provide a more comprehensive and richer multimodal representation, so as to more accurately represent and understand the children's play state. Specifically, in the technical solution of the present application, the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector are further input into a multimedia fusion device based on the CLIP model to obtain a multimodal children's play state representation matrix. Through the processing of the CLIP model-based multimedia aggregator, it is possible to help learn the correlation and semantic information between different data sources and different modal data in the children's play status, so as to better understand the multimodal connection and representation semantics between children's behavior semantics and sound semantics, so as to improve the accuracy and effectiveness of the analysis tasks and early warning tasks for children's play status.

[0052] Specifically, the child risk warning module 344 is used to determine whether to generate a risk warning prompt based on the multimodal children's play state representation matrix. In a specific example of the present application, the multimodal children's play state representation matrix is ​​passed through a classifier-based children's play behavior safety monitoring result generator to obtain a safety monitoring result, and the safety monitoring result is used to indicate whether a risk warning prompt is generated. In other words, the multimodal semantic fusion representation information of the children's play state is used for classification processing to more accurately monitor the children's play state and warn of risks. In this way, the children's play state can be monitored in real time in the future community through the fusion semantics of multimedia to monitor and warn children's outdoor safety, thereby reducing the hidden safety hazards of children's outdoor play and realizing more intelligent future community management.

[0053] It should be understood that before using the above-mentioned neural network model for inference, it is necessary to train the children's play behavior state feature extractor based on MobileNetV3, the children's play behavior time series message transmission aggregation analysis network, the children's play sound state time-frequency feature extractor based on the DenseNet model, the CLIP model-based multimedia aggregator, and the classifier-based children's play behavior safety monitoring result generator. That is to say, according to the future community management system 300 based on the Internet of Things and multimedia fusion of the present application, it also includes a training phase for training the children's play behavior state feature extractor based on MobileNetV3, the children's play behavior time series message transmission aggregation analysis network, the children's play sound state time-frequency feature extractor based on the DenseNet model, the CLIP model-based multimedia aggregator, and the classifier-based children's play behavior safety monitoring result generator.

[0054] Figure 3 FIG. 1 is a block diagram of the training phase of a future community management system based on the integration of the Internet of Things and multimedia according to an embodiment of the present application. Figure 3As shown, according to the embodiment of the present application, the future community management system based on the integration of the Internet of Things and multimedia includes: a training module 400, including: a training data acquisition unit 410, used to acquire training data, wherein the training data includes the training children's play status video data collected by the camera deployed in the shared playground; and the training children's play sound data collected by the sound detector; a training state key frame sampling unit 420, used to perform key frame sampling on the training children's play status video data to obtain a time series of training children's play status key frames; a training behavior semantic feature extraction unit 430, used to extract the training children's play status key frames; Each training child play state key frame in the time series of frames is respectively passed through a child play behavior state feature extractor based on MobileNetV3 to obtain a sequence of training child play behavior semantic feature vectors; a training semantic temporal dynamic aggregation representation unit 440 is used to pass the sequence of training child play behavior semantic feature vectors through a child play behavior temporal message transmission aggregation analysis network to obtain a training child play behavior temporal dynamic aggregation semantic representation vector; a training sound wavelet transform unit 450 is used to perform wavelet transform on the training child play sound data to obtain a training child play sound time-frequency diagram; a training sound state time-frequency semantic feature extraction unit 460, used for passing the training children's play sound time-frequency graph through the children's play sound state time-frequency feature extractor based on the DenseNet model to obtain the training children's play sound time-frequency representation vector; the multimodal fusion representation unit 470, used for inputting the training children's play behavior time series dynamic aggregation semantic representation vector and the training children's play sound time-frequency representation vector into the multimedia fusion based on the CLIP model to obtain the training multimodal children's play state representation matrix; the classification loss unit 480, used for passing the training multimodal children's play state representation matrix through the children's play behavior safety monitoring result generator based on the classifier to obtain the class probability value and Classification loss function value; a multi-distribution noise loss function calculation value unit 490, used to calculate the multi-distribution noise loss function value based on the class probability value; a training unit 500, used to train the MobileNetV3-based children's play behavior state feature extractor, the children's play behavior time series message passing aggregation analysis network, the DenseNet model-based children's play sound state time-frequency feature extractor, the CLIP model-based multimedia aggregator and the classifier-based children's play behavior safety monitoring result generator with the weighted sum of the classification loss function value and the multi-distribution noise loss function value as the loss function value.

[0055] That is, in a preferred example, when updating the model parameters through gradient back propagation based on the classification loss function, a multi-distribution noise loss function is further introduced, and the calculation of the multi-distribution noise loss function includes the following steps: obtaining the class probability value obtained by passing the training multimodal children's play state representation matrix through a classifier-based children's play behavior safety monitoring result generator, and calculating the absolute value of the difference between the class probability value and one to obtain a probability difference; determining the characteristic mean and characteristic variance of each eigenvalue of the training multimodal children's play state representation matrix as a feature set; determining the first product of each eigenvalue in the training multimodal children's play state representation matrix and the class probability value and the second product of the probability difference and the characteristic mean; summing the squares of the difference between the first product and the second product corresponding to each eigenvalue in the training multimodal children's play state representation matrix and dividing it by the scale of the multimodal children's play state representation matrix; subtracting the product of the sum of the squares of the class probability value and the probability difference and the characteristic variance from the division result to obtain the multi-distribution noise loss function.

[0056] Here, with respect to the time-frequency correlation features of the children's play sound data and the semantic temporal correlation features of the children's play behavior of the children's play status video data, within the cross-modal feature association domain, the correlation differences in the distribution of different modal features along their respective dimensions lead to the out-of-distribution noise of the cross-modal correlation feature representation set, which generates the class distribution noise of the multimodal children's play state representation matrix as a whole, thereby affecting the accuracy of the classification result. The above-mentioned multi-distribution noise loss function sets the approximate variance expectation characteristics of the out-of-distribution noise of the feature set of the training multimodal children's play state representation matrix relative to the Bernoulli sampling instances of the feature set based on the class probability distribution characteristics expressed by the class probability values ​​and probability differences of the training multimodal children's play state representation matrix through the classifier. Therefore, by using the multi-distribution noise loss function to measure the distribution feature noise based on the class probability distribution characteristics expressed by the class probability value and the probability difference, the statistical noise distribution can be used to degrade the possible sampling noise characteristics based on the probability crossover random selection characteristics of the expected variance in the class probability distribution of each eigenvalue of the training multimodal children's play state representation matrix during the classification process, thereby realizing the class distribution noise reduction of self-supervised learning in the process of model parameter updating to improve the accuracy of safety monitoring results. In this way, the monitoring and risk warning of children's play status can be carried out more accurately, thereby reducing the safety hazards of children's play and realizing more intelligent future community management.

[0057] As described above, the future community management system 300 based on the integration of the Internet of Things and multimedia according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a future community management algorithm based on the integration of the Internet of Things and multimedia. In one possible implementation, the future community management system 300 based on the integration of the Internet of Things and multimedia according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the future community management system 300 based on the integration of the Internet of Things and multimedia can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the future community management system 300 based on the integration of the Internet of Things and multimedia can also be one of the many hardware modules of the wireless terminal.

[0058] Alternatively, in another example, the future community management system 300 based on the integration of the Internet of Things and multimedia and the wireless terminal may also be separate devices, and the future community management system 300 based on the integration of the Internet of Things and multimedia may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0059] Furthermore, a future community management method based on the integration of the Internet of Things and multimedia is also provided.

[0060] Figure 6 FIG. 1 is a flow chart of a future community management method based on the integration of the Internet of Things and multimedia according to an embodiment of the present application. Figure 6 As shown, according to the future community management method based on the Internet of Things and multimedia fusion according to the embodiment of the present application, the method includes the following steps: S1, decompressing the compressed children's play status video data and the compressed children's play sound data to obtain the children's play status video data and the children's play sound data; S2, respectively extracting features from the children's play status video data and the children's play sound data to obtain a children's play behavior temporal dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector; S3, inputting the children's play behavior temporal dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector into a multimedia aggregator based on the CLIP model to obtain a multimodal children's play status representation matrix; S4, determining whether to generate a risk warning prompt based on the multimodal children's play status representation matrix.

[0061] In summary, the future community management method based on the Internet of Things and multimedia fusion according to the embodiment of the present application is explained. After collecting and transmitting the video data of children's play status and the sound data of children's play respectively through the camera and sound detector of the shared playground, the data processing and analysis algorithm based on artificial intelligence and multimedia fusion technology is introduced in the future community background management server to analyze and fuse the semantic temporal characteristics of children's play behavior and the semantic temporal characteristics of children's play sound, so as to comprehensively characterize the children's play status, thereby monitoring the children's play status and risk warning. In this way, the children's play status can be monitored in real time in the future community through the fusion semantics of multimedia to monitor and warn children's outdoor safety, thereby reducing the safety hazards of children's outdoor play and realizing more intelligent future community management.

[0062] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A future community management system based on the integration of the Internet of Things and multimedia, characterized in that: include: A sensor network, wherein the sensor network includes a camera and a sound detector deployed in a shared playground, the camera is used to collect video data of children's play status, and the sound detector is used to collect sound data of children's play; A data compression module, used for compressing the children's play state video data and the children's play sound data to obtain compressed children's play state video data and compressed children's play sound data; An Internet of Things communication network, used for transmitting the compressed children's play status video data and the compressed children's play sound data to a future community backend management server; The future community backend management server includes: A children's play multimedia data decoding module, used for decompressing the compressed children's play state video data and the compressed children's play sound data to obtain the children's play state video data and the children's play sound data; A children's play multimedia data feature analysis module, used for extracting features from the children's play state video data and the children's play sound data to obtain a children's play behavior temporal dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector; A children's play state multimodal fusion representation module, used for inputting the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector into a multimedia fusion device based on the CLIP model to obtain a multimodal children's play state representation matrix; A child risk warning module, used to determine whether to generate a risk warning prompt based on the multimodal child play state representation matrix; The children's entertainment multimedia data feature analysis module includes: A children's play state key frame sampling unit, used for performing key frame sampling on the children's play state video data to obtain a time sequence of children's play state key frames; A children's play behavior semantic feature extraction unit is used to extract children's play behavior state features from each children's play state key frame in the time series of the children's play state key frames to obtain a sequence of children's play behavior semantic feature vectors; A children's play behavior semantic time series dynamic aggregation representation unit, used for passing the sequence of the children's play behavior semantic feature vectors through a children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector; A children's play sound time-frequency feature expression unit, used for performing time-frequency conversion on the children's play sound data and then performing semantic feature expression on the obtained children's play sound time-frequency graph to obtain the children's play sound time-frequency representation vector; The children's play behavior semantic temporal dynamic aggregation representation unit is used to: Calculate the first The semantic feature vectors of children's play behaviors are summed up by position to obtain the children's play behavior neighbor semantic fusion feature vector; The sum of the trainable preset hyperparameter and one is used as the weighting coefficient for the weighting the feature values ​​of each position in the children's play behavior semantic feature vector to obtain a weighted optimized children's play behavior semantic feature vector; The children's play behavior neighbor semantic fusion feature vector and the weighted optimized children's play behavior semantic feature vector are added by position and then processed by a multi-layer perceptron to obtain the first a children's play behavior semantic time-series message transmission feature vector corresponding to the children's play behavior semantic feature vector; The positional sum of the children's play behavior semantic time series message transmission feature vectors corresponding to each children's play behavior semantic feature vector in the sequence of the children's play behavior semantic feature vectors is calculated to obtain the children's play behavior time series dynamic aggregation semantic representation vector.

2. The future community management system based on the integration of the Internet of Things and multimedia according to claim 1 is characterized in that: The children's play behavior semantic feature extraction unit is used to: respectively pass each children's play state key frame in the time series of the children's play state key frame through a children's play behavior state feature extractor based on MobileNetV3 to obtain a sequence of the children's play behavior semantic feature vectors.

3. The future community management system based on the integration of Internet of Things and multimedia according to claim 2 is characterized in that: The children's play sound time-frequency feature expression unit comprises: A children's play sound wavelet transform subunit, used for performing wavelet transform on the children's play sound data to obtain a children's play sound time-frequency diagram; The children's play sound state time-frequency semantic feature extraction unit is used to obtain a children's play sound time-frequency representation vector by passing the children's play sound time-frequency graph through a children's play sound state time-frequency feature extractor based on a DenseNet model.

4. The future community management system based on the integration of Internet of Things and multimedia according to claim 3 is characterized in that: The child risk warning module is used to: pass the multimodal child play state representation matrix through a classifier-based child play behavior safety monitoring result generator to obtain a safety monitoring result, and the safety monitoring result is used to indicate whether a risk warning prompt is generated.

5. The future community management system based on the integration of Internet of Things and multimedia according to claim 4 is characterized in that: It also includes a training module for training the children's play behavior state feature extractor based on MobileNetV3, the children's play behavior time series message transmission aggregation analysis network, the children's play sound state time-frequency feature extractor based on the DenseNet model, the CLIP model-based multimedia aggregator and the classifier-based children's play behavior safety monitoring result generator.

6. The future community management system based on the integration of Internet of Things and multimedia according to claim 5 is characterized in that: The training module comprises: A training data acquisition unit, used to acquire training data, wherein the training data includes video data of the children's play status of the training collected by the camera deployed in the shared playground; and sound data of the children's play of the training collected by the sound detector; A training state key frame sampling unit, used for performing key frame sampling on the training children's play state video data to obtain a time series of training children's play state key frames; A training behavior semantic feature extraction unit, used for respectively passing each training child play state key frame in the time series of the training child play state key frame through a child play behavior state feature extractor based on MobileNetV3 to obtain a sequence of training child play behavior semantic feature vectors; A training semantic temporal dynamic aggregation representation unit is used to pass the sequence of the training children's play behavior semantic feature vectors through a children's play behavior temporal message transmission aggregation analysis network to obtain a training children's play behavior temporal dynamic aggregation semantic representation vector; A training sound wavelet transform unit, used for performing wavelet transform on the training children's play sound data to obtain a training children's play sound time-frequency diagram; A training sound state time-frequency semantic feature extraction unit, used for passing the training children's play sound time-frequency graph through a children's play sound state time-frequency feature extractor based on a DenseNet model to obtain a training children's play sound time-frequency representation vector; A multimodal fusion representation unit, used for inputting the training children's play behavior time-series dynamic aggregation semantic representation vector and the training children's play sound time-frequency representation vector into a multimedia aggregator based on a CLIP model to obtain a training multimodal children's play state representation matrix; A classification loss unit, used for passing the training multimodal children's play state representation matrix through the classifier-based children's play behavior safety monitoring result generator to obtain a class probability value and a classification loss function value; A multi-distribution noise loss function calculation value unit, used to calculate a multi-distribution noise loss function value based on the class probability value; A training unit is used to train the children's play behavior state feature extractor based on MobileNetV3, the children's play behavior time series message passing aggregation analysis network, the children's play sound state time-frequency feature extractor based on the DenseNet model, the multimedia aggregator based on the CLIP model and the classifier-based children's play behavior safety monitoring result generator by using the weighted sum of the classification loss function value and the multi-distribution noise loss function value as the loss function value.

7. A future community management method based on the integration of the Internet of Things and multimedia, characterized in that: include: Collect video data of children's play status and sound data of children's play; Compressing the children's play state video data and the children's play sound data to obtain compressed children's play state video data and compressed children's play sound data; Transmitting the compressed children's play state video data and the compressed children's play sound data to the future community backend management server; On the future community backend management server, the compressed children's play state video data and the compressed children's play sound data are decompressed to obtain the children's play state video data and the children's play sound data; On the future community backend management server, feature extraction is performed on the children's play state video data and the children's play sound data to obtain a children's play behavior time series dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector; In the future community backend management server, the children's play behavior time series dynamic aggregation semantic representation vector and the children's play sound time-frequency representation vector are input into a multimedia fusion device based on the CLIP model to obtain a multi-modal children's play state representation matrix; as well as In the future community backend management server, determining whether to generate a risk warning prompt based on the multimodal children's play state representation matrix; The steps of extracting features from the children's play state video data and the children's play sound data to obtain a children's play behavior temporal dynamic aggregation semantic representation vector and a children's play sound time-frequency representation vector include: Performing key frame sampling on the children's play state video data to obtain a time sequence of key frames of the children's play state; Extracting the child play behavior state features from each child play state key frame in the time series of the child play state key frames to obtain a sequence of child play behavior semantic feature vectors; Passing the sequence of the children's play behavior semantic feature vectors through a children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector; After performing time-frequency conversion on the children's amusement sound data, semantic feature expression is performed on the obtained children's amusement sound time-frequency graph to obtain the children's amusement sound time-frequency representation vector; The process of passing the sequence of the children's play behavior semantic feature vectors through a children's play behavior time series message transmission aggregation analysis network to obtain the children's play behavior time series dynamic aggregation semantic representation vector includes: Calculate the first The semantic feature vectors of children's play behaviors are summed up by position to obtain the children's play behavior neighbor semantic fusion feature vector; The sum of the trainable preset hyperparameter and one is used as the weighting coefficient for the weighting the feature values ​​of each position in the children's play behavior semantic feature vector to obtain a weighted optimized children's play behavior semantic feature vector; The children's play behavior neighbor semantic fusion feature vector and the weighted optimized children's play behavior semantic feature vector are added by position and then processed by a multi-layer perceptron to obtain the first a children's play behavior semantic time-series message transmission feature vector corresponding to the children's play behavior semantic feature vector; The positional sum of the children's play behavior semantic time series message transmission feature vectors corresponding to each children's play behavior semantic feature vector in the sequence of the children's play behavior semantic feature vectors is calculated to obtain the children's play behavior time series dynamic aggregation semantic representation vector.

Images