A community resident behavior prediction speculation method based on an Internet of Things gateway preprocessing

Abstract

The application provides a kind of community resident behavior prediction speculation method based on Internet of Things gateway pre-processing, the application dynamically supervises the behavior characteristics of community residents through industrial system, and pre-conducts behavior monomorphism supervision on the distributed industrial gateway, in combination with the identification speculation service of behavior in the background, based on the historical travel sequence big data of community residents, the behavior prediction result matched with the space-time characteristics of the community residents is speculated and recorded and saved.Can distribute the computing power of behavior monitoring in the background on the front gateway server, reduce the background pressure, and the background does not need to handle community resident behavior data generally, only need to supervise monomorphism behavior characteristics, find alarm behavior characteristics and notify gateway to carry out feature joint sampling, finally, according to the space-time characteristics of the community residents after fusion, behavior prediction analysis is carried out, so as to realize dynamic, multi-modal safety management, track and predict resident behavior through diversified AI identification and management, realize community intelligent management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart community, and in particular, it refers to a community resident behavior prediction speculation method and device based on an Internet of Things gateway preprocessing, an electronic device and a computer readable storage medium. BACKGROUND

[0002] The significance of community resident behavior recognition and prediction lies in the following points:

[0003] Improving community management efficiency: By recognizing and predicting resident behavior, community managers can plan resource allocation in advance. For example, if it is predicted that the use frequency of fitness facilities in a certain area will increase, maintenance and upgrading can be arranged in advance to avoid facility damage affecting resident use, thereby improving overall management level and optimizing resident life experience.

[0004] Enhancing community security guarantee: Identifying abnormal behavior and predicting potential security threats helps to take preventive measures in time. For example, if it is identified that strangers frequently loiter in the community during a certain period of time, by further analyzing and predicting possible security risks, community security personnel can strengthen patrol to reduce the probability of occurrence of bad events such as theft and damage, and guarantee the safety of residents' life and property.

[0005] Optimizing community service supply: Understanding resident daily behavior patterns and demand changes can enable the community to provide more practical services. For example, if it is predicted that the demand for health lectures among elderly residents will increase, the community can organize relevant medical experts to carry out targeted lectures to improve the accuracy and satisfaction of services and improve the life quality of community residents.

[0006] Promoting harmonious development of the community: By analyzing resident behavior, the interaction rules and potential conflict points among residents are found out, and coordination and communication are carried out in advance. For example, if it is predicted that parking space shortage may cause neighbor disputes, the community can plan new parking areas or optimize parking rules in advance to create a harmonious community atmosphere and enhance the sense of belonging and cohesion of residents.

[0007] In the prior art, the common method for identifying and predicting community resident behavior is to identify and predict behavior through behavior monitoring or based on sensor monitoring.

[0008] Based on sensor-based methods, the access control and monitoring system such as the access control device and monitoring camera installed at the community entrance and key area can record the resident's entry and exit time, location, etc. Through long-term accumulation of data, the resident's daily travel habits are analyzed, abnormal entry and exit behaviors such as frequent outings during non-working hours, long time without returning, etc. are identified, and future travel trends are predicted to assist community safety management and service arrangement; or through environmental sensors, combined with resident activity data in different environments, the correlation between resident behavior and environmental factors is analyzed, for example, residents are more inclined to go to places with air conditioning in high temperature weather, and the possible activity location of residents is predicted through environmental data to provide basis for community activity organization and facility deployment.

[0009] And based on behavior monitoring, machine learning algorithms are also popular, such as using supervised learning algorithms such as decision trees, support vector machines, etc. to train labeled resident behavior data and establish a behavior prediction model. Residents can be divided into different groups according to behavior characteristics, and individualized services and management strategies can be developed for different groups. For example, the invention patent with publication number CN117197755A discloses a community personnel identity monitoring and recognition method and device, which integrates the collected community personnel image, sound and infrared scanning data through multiple sensor units, uses a deep learning method to establish a face recognition and special makeup detection model, extracts and trains image features based on the established model to more efficiently recognize disguised behaviors including makeup and face covering, constructs a model through a convolutional neural network (CNN) to improve accuracy, and then analyzes the behavior of community personnel using a deep learning model, including walking posture, hand gestures and standing posture analysis, and analyzes sensor data to extract behavior patterns.

[0010] Although the above-mentioned common methods for identifying and predicting community resident behavior can play a certain role, there are still some technical pain points:

[0011] First, the monitoring of community residents is insufficient, usually only some behavior data of residents are monitored and identified, such as only collecting behavior sensor data (environmental sensor data) for prediction. This single behavior identification (mainly referring to directly identifying behavior based on monitoring data) method makes it difficult for the community to monitor and control the multiple behaviors of residents, cannot monitor more behavior patterns, and cannot predict the next behavior of residents;

[0012] Second, the existing monitoring scheme is mainly static and fixed monitoring, that is, it can only identify the static behavior of residents based on the current point or the behavior characteristics learned based on the model (such as face features or gesture features), and cannot predict behavior according to user behavior rules, and lacks dynamic rule prediction ability. Moreover, when the existing monitoring scheme monitors user behavior, it uploads user behavior data to the background for processing, performs behavior identification and supervision in the background, and has no feature diversification identification function, thereby greatly increasing the background computing power cost and operation pressure. SUMMARY

[0013] In order to solve the technical problems existing in the prior art, the present application provides the following technical solutions:

[0014] In one aspect, a method for predicting community resident behavior based on Internet of Things gateway preprocessing is provided, which is implemented by an electronic device and includes:

[0015] S1, collecting a multi-source heterogeneous data set of community residents according to a preset sampling frequency and uploading an industrial gateway unit, wherein the multi-source heterogeneous data set includes spatial flow behavior images and / or time series flow sensing data;

[0016] S2, receiving the multi-source heterogeneous data set by the industrial gateway unit and preprocessing the multi-source heterogeneous data set, supervising the preprocessed multi-source heterogeneous data set, identifying each modal data feature in the multi-source heterogeneous data set and uploading it to the background server;

[0017] S3, supervising by the background server whether the single-state data feature is an alarm behavior feature:

[0018] If yes, a feature joint sampling instruction is issued to the industrial gateway unit, and step S4 is entered;

[0019] Otherwise, the data feature of the next mode is continuously supervised for alarm;

[0020] S4, in response to the feature joint sampling instruction, the industrial gateway unit fuses each modal data feature in the multi-source heterogeneous data set, generates corresponding community resident space-time features and uploads them to the background server;

[0021] S5, the background server predicts the behavior prediction result matched with the community resident space-time feature based on the historical travel sequence big data of the community residents and records and saves it.

[0022] Preferably, the industrial gateway unit comprises:

[0023] (1) a processor for providing data processing and logical communication;

[0024] (2) a data classifier, configured to classify the multi-source heterogeneous data set into corresponding spatial flow behavior images or time-series flow sensing data, and input the spatial-time dual-channel feature processing model according to types;

[0025] (3) a spatial-time dual-channel feature processing model, comprising:

[0026] a spatial feature processing channel based on MobileNetV3, configured to identify and extract spatial flow behavior image features in the spatial flow behavior images;

[0027] a time-series feature processing channel based on a BiLSTM network and a 1D-CNN network, configured to identify and extract time-series flow sensing features in the time-series flow sensing data;

[0028] (4) a feature fusioner, configured to adopt a gated feature fusion mechanism to perform feature fusion on the spatial flow behavior image features and the time-series flow sensing features, and generate community resident space-time features;

[0029] (5) an industrial communication gateway module, configured to upload the community resident space-time features to the background server in real time.

[0030] Preferably, in step S2, the identification of the modal data features in the multi-source heterogeneous data set and the single-mode upload to the background server are specifically:

[0031] The industrial gateway unit randomly selects one single-mode data feature from the spatial flow behavior image features or the time-series flow sensing features generated by this sampling, and uploads the data feature to the background server through the industrial communication gateway module.

[0032] Preferably, in S5, the background server infers a behavior prediction result matched with the community resident space-time features based on historical travel sequence big data of community residents and records and saves the behavior prediction result, which comprises:

[0033] Based on the historical travel sequence big data of a plurality of community residents collected and stored on the background server, a resident behavior prediction model is constructed;

[0034] The community resident space-time features are input into the resident behavior prediction model, and the resident behavior prediction model infers a behavior prediction result of the community resident at a next space-time point matched with the community resident space-time features;

[0035] The behavior prediction result of the community resident at the next space-time point is bound with the community resident ID, and is recorded and saved to a background database.

[0036] Preferably, the construction method of the resident behavior prediction model comprises:

[0037] collecting historical travel sequence big data of a plurality of community residents;

[0038] extracting community resident spatiotemporal features of each community resident at different spatiotemporal points in the historical travel sequence big data, and labeling behavior prediction features of next spatiotemporal points;

[0039] counting each labeled behavior prediction feature to obtain a feature set, and dividing the feature set into a training set and a validation set according to a preset proportion;

[0040] inputting the training set into a preset LSTM model to generate an initial resident behavior prediction model;

[0041] verifying behavior prediction accuracy of the initial resident behavior prediction model using the validation set;

[0042] if the verification is qualified, deploying and applying the resident behavior prediction model to the background server;

[0043] otherwise, retraining the resident behavior prediction model.

[0044] In another aspect, a device for predicting community resident behavior based on Internet of Things gateway preprocessing is provided. The device is applied to a method for predicting community resident behavior based on Internet of Things gateway preprocessing. The device comprises:

[0045] (1) an Internet of Things collection system, configured to collect a multi-source heterogeneous data set of community residents according to a preset sampling frequency and upload the data set to an industrial gateway unit, wherein the multi-source heterogeneous data set comprises spatial flow behavior images and / or time series flow sensing data;

[0046] (2) an industrial gateway unit, configured to receive the multi-source heterogeneous data set and preprocess the data set; and perform feature supervision on the preprocessed multi-source heterogeneous data set, identify each modality data feature in the multi-source heterogeneous data set, and upload the data feature to a background server in a single modality; and / or, in response to a feature joint sampling instruction issued by the background server, fuse each modality data feature in the multi-source heterogeneous data set, generate corresponding community resident spatiotemporal features, and upload the features to the background server;

[0047] (3) a background server, configured to supervise whether the single modality data feature is an alarm behavior feature:

[0048] if yes, issuing a feature joint sampling instruction to the industrial gateway unit; and / or, based on historical travel sequence big data of community residents, predicting behavior prediction results matching the community resident spatiotemporal features and recording and saving the results;

[0049] Conversely, the data features of the next modality are continuously supervised for alarm;

[0050] The Internet of Things collection system is in communication connection with the industrial gateway unit.

[0051] The industrial gateway unit is in communication connection with the background server.

[0052] In another aspect, an electronic device is provided, comprising: a processor; a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, implement any one of the above-described methods for predicting community resident behavior based on Internet of Things gateway pre-processing.

[0053] In another aspect, a computer readable storage medium is provided, the storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement any one of the above-described methods for predicting community resident behavior based on Internet of Things gateway pre-processing.

[0054] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0055] The present application dynamically supervises the behavior characteristics of community residents through industrial systems, and pre-processes the behavior characteristics on the distributed industrial gateway, combines the background behavior recognition and prediction service, based on the historical travel sequence big data of community residents, predicts and records the behavior prediction results matching the spatio-temporal characteristics of the community residents. The computing power of the background behavior monitoring can be distributed on the front-end gateway server, reducing the background pressure, and the background only needs to supervise the single behavior characteristics, and the alarm behavior characteristics are notified to the gateway for feature joint sampling, and finally the behavior prediction analysis is performed according to the fused spatio-temporal characteristics of the community residents, so as to realize dynamic and multi-modal safety management, track and predict the behavior of residents through diversified AI recognition, and manage the community intelligently. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1 It is a kind of based on the community resident behavior prediction of the Internet of Things gateway pre-processing method flow chart provided by the embodiments of the present application;

[0058] Figure 2It is a community resident behavior prediction speculation device block diagram based on the front processing of the Internet gateway provided by the embodiment of the application.

[0059] Figure 3 And Figure 4 It is a background configuration and running management page schematic diagram of the industrial gateway unit provided by the embodiment of the application.

[0060] Figure 5 It is a hardware system composition structure schematic diagram of the industrial gateway unit provided by the embodiment of the application.

[0061] Figure 6 It is a prediction page schematic diagram of the current behavior prediction provided by the embodiment of the application.

[0062] Figure 7 It is a structure schematic diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0063] The technical solutions in the application will be described below with reference to the drawings.

[0064] In the embodiments of the application, the words such as "example", "for example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0065] In the embodiments of the application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0066] In the embodiments of the application, sometimes the subscript such as W1 can be mistakenly used in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0067] In order to make the technical problems, technical solutions and advantages to be solved by the application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0068] In the embodiments of the application, the communication type between the Internet of Things collection system, the gateway and the background can be deployed in combination with the existing Internet of Things system, and the embodiments are not limited.

[0069] It should be noted that the embodiments of the present application can involve the use of user data. In actual application, user-specific personal data can be used in the schemes described herein in a manner that complies with the applicable laws and regulations of the country (for example, with the explicit consent of the user, with the actual notification to the user, etc.) and within the scope permitted by the applicable laws and regulations.

[0070] The embodiment of the application provides a community resident behavior prediction speculation method based on an Internet of Things gateway preprocessing, which can be realized by an electronic device, which can be a terminal or a server. Figure 1 As shown in the flowchart of the community resident behavior prediction speculation method based on the Internet of Things gateway preprocessing, the processing flow of the method can include the following steps:

[0071] S1, according to a preset sampling frequency, a plurality of source heterogeneous data sets of community residents are collected and uploaded to an industrial gateway unit, wherein the plurality of source heterogeneous data sets include spatial flow behavior images and / or time series flow sensing data;

[0072] S2, the industrial gateway unit receives the plurality of source heterogeneous data sets and pre-processes (cleans, normalizes heterogeneous formats, classifies, etc.) the plurality of source heterogeneous data sets (resident travel video images, action posture images, time series sensing information such as barrier / access control records at a certain time point, entry / exit door fingerprint verification behaviors, etc.); the industrial gateway unit supervises the plurality of source heterogeneous data sets after preprocessing, identifies the features of each modality data in the plurality of source heterogeneous data sets, and uploads the features to a background server in a single state;

[0073] S3, the background server supervises whether the data features in the single state are alarm behavior features:

[0074] If yes, a feature joint sampling instruction is issued to the industrial gateway unit, and step S4 is entered;

[0075] Otherwise, the data features of the next modality are continuously supervised for alarm;

[0076] S4, the industrial gateway unit responds to the feature joint sampling instruction, fuses the features of each modality data in the plurality of source heterogeneous data sets, generates corresponding community resident space-time features, and uploads the community resident space-time features to the background server;

[0077] S5, the background server speculates a behavior prediction result matched with the community resident space-time features based on historical travel sequence big data of the community residents and records and saves the behavior prediction result.

[0078] The accompanying drawings are combined with the above description. Figure 2As shown, the scheme needs to be pre-processed by several community distributed deployment of Internet of Things collection system and industrial gateway unit when it is executed, and the Internet of Things collection system of each area (such as building) collects the multi-source heterogeneous data set of community residents according to the first sampling frequency allocated to it (according to the building / region setting corresponding sampling frequency, distributed data upload), and uploads to the industrial gateway unit of the region, and then the industrial gateway unit performs preprocessing. Each distributed industrial gateway unit reports the preprocessed data according to the preset data reporting priority (such as residential area priority to community gate area) and reports the background.

[0079] As shown in the background configuration and operation management page of the industrial gateway unit. Figure 3 and Figure 4 The background can configure and monitor the information of each deployed industrial gateway unit. For example, for the industrial gateway unit with monitoring number 52153148, the background will record the network management number and its running state (running time, CPU usage, memory occupation, network delay, data packet volume of feature data reported each time, etc.) in its system, and also record its monitoring period, state monitoring score and its identified abnormal behavior, record its abnormal processing state, times and operation behavior. In this way, through gateway data reporting and behavior abnormality identification and judgment (feature supervision), the background can save and update the running records of each industrial gateway unit deployed in the community, and the background administrator can check the work log of each industrial gateway unit at any time.

[0080] The multi-source heterogeneous data collection scheme of the Internet of Things collection system can refer to the following hardware configuration:

[0081] 1. Spatial data collection:

[0082] Model: Hikvision DS-2CD3326DWA-I (1080P@30fps);

[0083] Deployment: community entrance / unit door / public area;

[0084] Frequency: residential area 15fps / key area 30fps.

[0085] 2. Time series data collection:

[0086] Access control record: Langui intelligent access control F7+ (RS485 interface);

[0087] Fingerprint record: Entropy-based technology CF80 (TCP / IP protocol);

[0088] Environmental sensor: TI CC2650 multifunctional sensor terminal.

[0089] Implementation process:

[0090] Real-time transmission of video stream to gateway through RTSP protocol; access control / fingerprint device reports JSON format log every 500ms.

[0091] Establish device-resident ID mapping table (code example as follows):

[0092] {device_id: "GATE_A", resident_id: "RC2024001"}.

[0093] Data encapsulation format:

[0094] "timestamp": "2024-06-18T09:30:25.000Z",

[0095] "data_type": "door_event",

[0096] "payload": {

[0097] "device": "GATE_A",

[0098] "action": "enter",

[0099] "verify_mode": "fingerprint"

[0100] }

[0101] }

[0102] In order to reduce the background pressure, the industrial gateway unit first needs to be pretreated: the data classifier configured in each gateway unit and the space-time dual-channel feature processing model are used to extract and classify the collected multi-source data features, form a plurality of single modal behavior features (or time sequence sensor features), upload a single modal feature to the background, and perform background behavior warning judgment (the background can extract and record various warning behavior features through deep learning technology by collecting and recording historical warning behavior big data, and store them, and judge whether to trigger a warning by comparing features, such as finding that the resident's access to the access control is abnormal (matching / similar to the recorded historical access control alarm behavior feature), and determining that the resident's single modal data feature reported by the gateway is an alarm behavior feature), if there is a warning behavior feature, the gateway is notified to report the joint feature of the resident for post-processing; if not, the next time sequence behavior is supervised. In this way, the collected big data can be sent to the background for processing, greatly reducing the background's supervision and processing power of the resident behavior big data and the running load. When the background finds that a time sequence behavior has a behavior warning feature, it notifies the gateway to report the joint behavior feature (fusion feature) of the resident. This step is completed by the gateway in each region, so the background does not need to perform feature fusion, greatly reducing the operating pressure of the background.

[0103] The subsequent background can predict the behavior prediction result matching the space-time feature of the community resident based on the historical travel sequence big data of the community resident based on big data technology. In this way, the behavior of the community resident is orderly managed in the background, and the behavior information of the resident's appearance and going out is predicted, so as to better control and track the resident's activity progress and location.

[0104] As shown in Figure 5 Preferably, the industrial gateway unit comprises:

[0105] (1) a processor for providing data processing and logical communication;

[0106] (2) a data classifier for classifying the multi-source heterogeneous data set into corresponding space flow behavior images or time sequence flow sensor data, and inputting the model channel of the space-time dual-channel feature processing model according to the type;

[0107] (3) a space-time dual-channel feature processing model comprising:

[0108] a space feature processing channel based on MobileNetV3, for identifying and extracting space flow behavior image features in the space flow behavior image;

[0109] a time sequence feature processing channel based on BiLSTM network and 1D-CNN network, for identifying and extracting time sequence flow sensor features in the time sequence flow sensor data;

[0110] (4) Feature fusioner, for using a gated feature fusion mechanism to perform feature fusion on the spatial flow behavior image features and the time series flow sensing features, to generate community resident spatiotemporal features;

[0111] (5) An industrial communication gateway module, configured to upload the community resident spatiotemporal features to the background server in real time.

[0112] By reducing the burden of edge computing, the industrial gateway performs data preprocessing and feature extraction, which can greatly reduce the amount of raw data transmission and processing in the background.

[0113] The gateway realizes multi-modal hierarchical processing:

[0114] Spatial data (image / video) → MobileNetV3 lightweight feature extraction;

[0115] Time series data (sensors) → BiLSTM+1D-CNN fusion processing.

[0116] Dynamic triggering mechanism: single-modal feature preliminary screening → background early warning judgment → triggering multi-modal fusion upload;

[0117] Spatiotemporal feature fusion: gated mechanism dynamically weights spatial and time series feature weights.

[0118] The specific industrial gateway hardware configuration scheme can refer to Table 1 below (for other interfaces, power supplies, etc., not described here):

[0119]

[0120] Table 1

[0121] The working principles of each module are as follows:

[0122] 1. Processor unit (data processing core)

[0123] Hardware selection: NXP i.MX 8M Plus (integrated NPU accelerator)

[0124] Working mechanism:

[0125] Heterogeneous computing architecture: Cortex-A53 handles control logic, and NPU is dedicated to AI model inference (supports MobileNetV3 / BiLSTM INT8 quantization acceleration);

[0126] Hardware-level isolation: Cortex-M7 real-time core independently runs industrial protocol stack to ensure low communication latency;

[0127] Energy efficiency ratio control: DVFS dynamic frequency adjustment technology, typical power consumption ≤ 3W@full load.

[0128] 2. Data classifier (heterogeneous data routing)

[0129] Hardware support: running lightweight classification algorithm model (such as FFT-based real-time spectrum analysis algorithm to identify sensor data types) with NPU;

[0130] Processing flow: original data type determination:

[0131] If it is an RGB image, input the spatial stream as an image channel;

[0132] If it is sensor time series data, input the time series stream sensor data channel.

[0133] 3. Spatial-temporal dual-channel feature processing model

[0134] (1) Spatial feature channel (optimized version of MobileNetV3)

[0135] Hardware acceleration: NPU running quantized model (TensorRT deployment);

[0136] Performance indicators: inference time ≤ 15 ms at 224 × 224 resolution.

[0137] The specific spatial stream branch design is as follows: an improved architecture based on MobileNetV3, containing 12 Bottleneck modules, using h-swish activation function and SE attention mechanism, and the input resolution is adjusted to 160 × 120 pixels. This branch processes static image features, and the calculation amount is controlled within 0.5B FLOPs; on the original MobileNetV3 model structure, the adaptive spatial pooling layer can be adjusted (dynamically adjusted to 160 × 160 to reduce calculation amount), and the channel attention mechanism is compressed to 4-bit precision.

[0138] (2) Time series feature channel (BiLSTM + 1D-CNN fusion architecture)

[0139] Hardware optimization: CPU uses ARM SIMD instructions to accelerate matrix operations;

[0140] Processing flow: sensor data → 1D-CNN (3 layers of depth separable convolution) → BiLSTM (128 units) → time series feature vector.

[0141] The specific time series stream branch design is as follows: BiLSTM network cooperates with 1D depth separable convolution, the time window is set to 15 frames, and the hidden layer dimension is 64. Causal convolution is introduced to ensure real-time performance, and the processing time delay of time series behavior features is < 20 ms. Sliding window mechanism (5-second window / 1-second step).

[0142] The specific space-time dual-channel feature processing model is designed as follows:

[0143] The space feature processing channel based on MobileNetV3 is used to identify and extract the space flow behavior image features Fv in the space flow behavior image:

[0144] ,

[0145] For the input image (from the current sampling image set R, resolution HxW), is the depth separable convolution kernel weight, represents the depth separable convolution operation, is a preset bias term, and GeLU is a Gaussian error linear unit activation function.

[0146] The time sequence feature processing channel based on BiLSTM network and 1D-CNN network is used to identify and extract the time sequence flow sensing features h t :

[0147] ,

[0148] is the sensor data at time t, is the hidden state at time t, is the LSTM parameter (including input / forgetting / output gate parameters).

[0149] The default functions and parameters of the above model features can be adjusted and set by the user according to actual needs.

[0150] 4. Gating feature fusioner

[0151] Hardware implementation: processor floating point unit calculation;

[0152] Fusion algorithm: used to fuse features to generate the community resident spatiotemporal features Fspatiotemporal of residents:

[0153] ,

[0154] are the space flow behavior image and the time sequence flow sensing data, respectively;

[0155] is the gating weight matrix, represents Hadamard (element-wise multiplication), and σ is the sigmoid activation function (output [0, 1]); [] represents the feature splicing operation (concatenation, addition, global pooling, etc.). The above fusion means that the fusion feature at time t is represented.

[0156] Bandwidth optimization: Output feature dimension compressed to 256 dimensions (FP16 format)

[0157] 5. Industrial communication gateway module

[0158] Intelligent transmission strategy:

[0159] Baseline data: Full upload per hour (<10KB);

[0160] Abnormal event: Real-time MQTT push (CoAP protocol compression, bandwidth <2KB / s);

[0161] Protocol support:

[0162] Industrial layer: OPC UA over TSN (Time-Sensitive Networking);

[0163] IoT layer: MQTT 3.1.1 with TLS 1.3.

[0164] Its data processing efficiency is shown in Table 2 below:

[0165]

[0166] Table 2

[0167] Each of the above distributed setups supports access from 200+ sensors per gateway, with a spatiotemporal feature extraction latency of ≤200ms, reducing the daily cloud data processing volume by 82.6%, perfectly meeting the real-time and reliability requirements of industrial scenarios.

[0168] Preferably, in step S2, identifying the features of each modality in the multi-source heterogeneous dataset and uploading them to the backend server in a single state specifically involves:

[0169] The industrial gateway unit randomly extracts a single-modal data feature from the spatial flow image features or the temporal flow sensing features generated in this sampling, and uploads it to the backend server by the industrial communication gateway module.

[0170] First, the gateway reports a resident's single-modal data feature, such as spatial flow behavior image features (e.g., entry and exit image features in the access control space, or behavioral image features when entering the access control area), to the backend. The backend then determines whether the behavior is abnormal (e.g., not swiping the card as required or carrying unidentified persons / prohibited pets, etc., which can be understood in conjunction with existing AI recognition and early warning technologies). If it is abnormal, the gateway will be notified again to upload the resident's fused behavioral feature F spatiotemporal.

[0171] Preferably, S5, the background server is based on the historical travel sequence big data of community residents, infers the behavior prediction result matched with the space-time characteristics of the community residents and records and saves, including:

[0172] Based on the historical travel sequence big data of a plurality of community residents collected and stored on the background server, a resident behavior prediction model is constructed, and the construction method of the resident behavior prediction model comprises the following steps:

[0173] Collecting historical travel sequence big data of a plurality of community residents;

[0174] Extracting the space-time characteristics of each community resident at different space-time points in the historical travel sequence big data, and labeling the behavior prediction characteristics of the next space-time point;

[0175] Statistically analyzing each labeled behavior prediction characteristic to obtain a feature set, and dividing the feature set into a training set and a validation set according to a preset proportion;

[0176] Inputting the training set into a preset LSTM model to generate an initial resident behavior prediction model;

[0177] Verifying the behavior prediction accuracy of the initial resident behavior prediction model using the validation set:

[0178] If the verification is qualified, the resident behavior prediction model is deployed and applied to the background server;

[0179] Otherwise, the resident behavior prediction model is retrained;

[0180] Inputting the space-time characteristics of the community residents into the resident behavior prediction model to infer the behavior prediction result of the community residents at the next space-time point matched with the space-time characteristics of the community residents;

[0181] Binding the behavior prediction result of the community residents at the next space-time point with the community resident ID and recording and saving to the background database.

[0182] The present application constructs a resident behavior prediction model based on LSTM (Long Short Term Memory), which uses historical travel sequence big data of community residents to predict the behavior of residents at the next space-time point (such as location change, time change or specific activity type).

[0183] The administrator can obtain data from the database of the background server (storing historical travel data of community residents) to construct a resident behavior prediction model; subsequently, by inputting the space-time characteristics generated by the historical sequence of residents, the model outputs the behavior prediction at the next space-time point, and the prediction result is bound with the resident ID and then stored persistently.

[0184] LSTM-based sequence prediction model, good at processing time series data, its input / output definition:

[0185] Input: Spatio-temporal feature sequence of community residents (represented as a sequence of vectors)

[0186] Output: "behavior prediction features" of the next spatio-temporal point (defined as a probability distribution of the next location ID, which can be extended to time or behavior type)

[0187] In the prediction phase, input a resident's historical sequence (variable length), and the model outputs a single behavior prediction result (prediction at the last time step).

[0188] Data flow architecture:

[0189] Data source: Database stores historical travel sequences (e.g., MySQL / PostgreSQL table, or HDFS file)

[0190] Model side: Background server (Python / Flask framework) processes data preprocessing, model training and prediction

[0191] Storage side: Database table records prediction results (structure like: resident ID, prediction timestamp, prediction result, confidence)

[0192] Performance indicators: Model prediction accuracy verification threshold (e.g., ≥85% accuracy), otherwise retrain

[0193] The model construction steps will be described in detail as follows:

[0194] The model construction process is as follows: collect data → extract spatio-temporal features and label → divide dataset → train LSTM model → verify accuracy → deploy model

[0195] The LSTM model is a preset model, and its architecture is not described here. The model output of the present application is classification data (predicted next location ID), which needs to be encoded into category labels (discrete values) for output.

[0196] 1. Model training

[0197] Step 1: Collect historical travel sequence big data of community residents

[0198] (1) Prepare data source: resident travel record table stored in the background database (table schema: resident_id, timestamp, location_id). location_id is a discrete location identifier (e.g., POI ID, community area code), and timestamp is the travel time.

[0199] (2) Dataset scope: Collect complete historical sequence data of multiple residents (e.g., 1000+), and each resident sequence is a time-ordered list of [timestamp, location ID], with no limit on sequence length.

[0200] (3) Data cleaning (Pre-processing):

[0201] Handle missing values: remove sequence breakpoints (e.g., 2+ hours of no data) or interpolate (based on historical average).

[0202] De-noising: filter invalid timestamps (e.g., time <1970) or location IDs (e.g., outlier GPS coordinates).

[0203] Normalization: sequence is sorted by time, ensuring reasonable interval between each sequence point (e.g., average 1 hour per point).

[0204] Output: Cleaned data is stored as CSV or directly loaded as Pandas DataFrame, format like [resident_id, [timestamp, location_id] sequence].

[0205] Step 2: Extract spatio-temporal features and label behavior prediction features

[0206] (1) Spatio-temporal feature definition (can be understood in conjunction with the previous "time series behavior encoding "): Extract feature vectors from each sequence point (timestamp, location_id) and encode them as numerical vectors. Features include:

[0207] Time features: based on timestamp, encoded as periodic features (avoid one-hot redundancy).

[0208] Hour: use sine-cosine encoding , which can handle time series periodicity.

[0209] Day of the week: encoded as an integer 1-7 (Monday to Sunday).

[0210] Day type: 0 (weekday) or 1 (weekend).

[0211] (2) Spatial feature: location ID is directly used as an integer index (if the location ID is not continuous, it needs to be mapped to an integer label). For coordinate type data, it is encoded as [lat, lon] normalized to [-1, 1]. It can be combined with the previous spatial flow behavior image feature Fv for extraction processing, so as to obtain the behavior characteristics of residents in different spaces, such as access behavior of access control, and specific images are used to extract multi-modal features of different convolution layers.

[0212] Feature vector example: each sequence point feature is a 4-dimensional vector, such as: [hour time point, location ID, behavior type, other behavior additional information (additional features identified by visual technology, such as residents often access access control at 10.00 am)].

[0213] (3) Label behavior prediction feature: behavior feature is defined as the location ID (classification label) of the next spatiotemporal point. For sequence point i, the target label is the location ID of sequence point i+1. Labeling rules:

[0214] Sequence length n: the label of each valid sequence point (from i=1 to n-1) is location ID_{i+1}.

[0215] Invalid processing: the end point of the sequence (without the next point) is not labeled, and short sequences (length <3) are removed.

[0216] Output format: each sample is (input sequence, target label), where:

[0217] Input sequence: [feature_vector_t0, feature_vector_t1,..., feature_vector_tk] (time ordered);

[0218] Target label: [location_index_{t1}, location_index_{t2},..., location_index_{tk+1}], actually, the model is trained on the shifted version of the sequence (predicting the next point).

[0219] Feature extraction example:

[0220] import numpy as np

[0221] Spatiotemporal feature extraction function

[0222] def extract_features(timestamp, location_id):

[0223] from datetime import datetime

[0224] dt = datetime.fromtimestamp(timestamp)

[0225]

[0226]

[0227] day_of_week = dt.weekday() + 1 # 1-7 for Monday-Sunday

[0228] day_type = 1 if day_of_week > 5 else 0 # 0=weekday, 1=weekend

[0229] location_index = location_id_map[location_id] # Pre-built mapping dictionary

[0230] return [hour_sin, hour_cos, day_of_week, day_type, location_index]

[0231] Sequence labeling functions

[0232] all_samples = [] # Store all samples

[0233] for seq in sequences: # sequences is a list of sequences for each resident

[0234] if len(seq) < 3: # Skip if sequence is too short

[0235] continue

[0236] features_seq = [extract_features(ts, loc) for ts, loc in seq]

[0237] targets = [loc_index for _, loc in seq[1:]] # Target sequence: Position ID starts from point 2

[0238] all_samples.append((np.array(features_seq), np.array(targets))).

[0239] Feature dimension: Set input feature vector size D=5 (4-dim time features + 1-dim location), number of location ID labels is total number of locations (integer index).

[0240] Step 3: Statistic behavior prediction features to get feature set, divide training set and validation set

[0241] Statistical behavior prediction features (labels): Frequency distribution of location labels, used for class balancing and data insight (e.g., hotspots).

[0242] Feature set: Aggregate all samples (multiple resident sequences), sample size is total number of sequence points (e.g., 1 million samples).

[0243] Dataset division: For example, use Scikit-learn division method, according to the preset ratio: 8:2 (training set: validation set), ensure that residents do not overlap (avoid data leakage). Random division or division by resident: 80% of resident data for training, 20% for validation.

[0244] Sequence processing: Each sequence is an independent unit, using variable-length sequence support (avoid padding pollution).

[0245] Output: Training set (X_train, y_train) and validation set (X_val, y_val), where X_train is the input sequence list, and y_train is the target label list.

[0246] Processing variable-length sequences: Use Python list to store sequences (Keras supports variable-length input).

[0247] Step 4: Input the training set into the preset LSTM model to generate the initial resident behavior prediction model

[0248] Model input / output functions are as follows:

[0249] Input: Sequence data (variable-length sequence), each time step feature vector dimension D=5 (defined in step 2);

[0250] Output: Behavior prediction features (location ID), classification problem. Set the number of categories K = number of location indexes (e.g., 100 categories).

[0251] LSTM layer definition:

[0252] Type: Sequence-to-sequence but only focus on the last step prediction (because the goal is to predict the next point).

[0253] Layer structure: 2 stacked LSTM layers to capture long-term dependencies; 1 Dense output layer (softmax activation).

[0254] Parameter presets (set to default values):

[0255] LSTM1 layer: 128 units, activation function 'tanh', return sequence output (return_sequences=True), input supports variable length.

[0256] LSTM2 layer: 64 units, activation function 'tanh'.

[0257] Dense layer: K units, softmax activation, output probability distribution for each position.

[0258] Optimizer: Adam (learning rate 0.001).

[0259] Loss function: Sparse categorical cross-entropy (sparse_categorical_crossentropy, since the label is an integer index).

[0260] Output: Probability distribution vector, predicted position index is argmax(prob_vector).

[0261] When training, the model architecture can be configured as follows:

[0262] Input (samples, sequence_length, D=5) (supports variable length sequences);

[0263] Masking layer (handle padding, no padding in this scheme but reserved);

[0264] LSTM(128, return_sequences=True) → Output sequence (same length as input);

[0265] LSTM(64, return_sequences=True) → Output sequence (same length as input);

[0266] (Optional: Dropout 0.2 to prevent overfitting)

[0267] Dense(K, activation='softmax') → Output sequence.

[0268] During training, the model outputs a sequence of the same length as the input sequence, and the target is the shifted version of the sequence (y_train). The loss is computed based on all time steps, but only the output of the last time step is used to predict the next point during inference.

[0269] The training process is as follows:

[0270] Input data format: X_train is a list of lists (each element is a variable-length sequence), and y_train is the corresponding label sequence;

[0271] Batch training: batch size 32, sequence length normalized by padding (but not recommended, use variable-length support instead). Variable-length processing uses Keras' pad_sequences but does not mask.

[0272] Training loop:

[0273] Epochs: 10-100 (early stop based on validation error).

[0274] Training log: output training loss and accuracy per epoch.

[0275] Step 5: Verify the prediction accuracy of the model using the validation set

[0276] Evaluation metrics: accuracy (the proportion of model output argmax(prob) consistent with the true label), weighted F1-score (handle class imbalance).

[0277] Validation criteria: validation set accuracy ≥ threshold (e.g., 85%, based on business requirements).

[0278] If not qualified (accuracy < 85%), retraining strategy:

[0279] Hyperparameter optimization: adjust learning rate, LSTM layer size, add Dropout (to prevent overfitting), increase data volume.

[0280] Retraining loop: new training set or data augmentation (such as sequence cropping), new number of epochs.

[0281] Revalidation: until the threshold is met.

[0282] If qualified, save the model file (.h5 or SavedModel format).

[0283] Step 6: Deploy and apply the model to the background server

[0284] Deployment operation: load the model file to the server (Python environment) and set up the inference API.

[0285] API Interface Design: POST request input (JSON format: {"resident_id": id, "sequence":[[feat_t0], [feat_t1], ...]}), output predicted location ID.

[0286] Output generation function: When the model makes predictions, it takes a historical sequence (of variable length) as input, calculates the output sequence of the entire sequence, and takes the prediction of the last time step as the result.

[0287] 2. Model-based prediction execution process

[0288] Input preparation: Extract the historical sequence of the target resident from the database (by querying resident_id), extract spatiotemporal features (same as step 2), and assemble the feature sequence;

[0289] Model inference: Calling the `predict_next_behavior(sequence)` function outputs:

[0290] next_loc_index: The predicted position ID of the next spatiotemporal point (integer);

[0291] prob_vector: Confidence distribution (used for reliability assessment).

[0292] Output format: The prediction results are discrete behaviors (location IDs), which can be extended to other behaviors (such as time-varying predictions, which require modification of the output layer).

[0293] Error handling: Return the default value (e.g., the hottest position) when the sequence is invalid.

[0294] 3. Prediction Result Recording and Saving Process

[0295] The prediction results are linked to resident IDs and persisted to the database.

[0296] like Figure 6 The prediction page shown, for example, for the target resident with ID 61205771, the system will predict the resident's behavior on 2025-05-16 based on their historical spatiotemporal behavior records (the behavioral sequence characteristics of a certain time period). It will also combine the confidence level of the prediction results set for their temporal characteristics to make a prediction of the behavior at this time (that is, the behavior at the next spatiotemporal point mentioned above). This means that it is predicted that the resident may "stay at home and watch TV" on that day. The system can also generate a behavior suggestion based on the set travel suggestions and bind it with the resident's ID (in this case, the resident's model number in the system) and push it to the resident's APP or the terminal device of their family members to notify them of community activities.

[0297] Therefore, the single state reporting can be performed by the gateway, the single state behavior characteristics of the residents are pre-identified by the background, whether the single state data characteristics are alarm behavior characteristics, if the single state data characteristics are alarm behavior characteristics, the community resident space-time characteristics of the residents are notified to be reported to the gateway, and based on the historical travel sequence big data of the community residents, a behavior prediction result matched with the community resident space-time characteristics is speculated. Therefore, not only the background pressure can be reduced, but also the possible behavior mode of the user in the future can be tracked and speculated in time, so that the resident behavior is facilitated to be tracked and managed and the security subscription management is facilitated.

[0298] In another aspect, a community resident behavior prediction speculation device based on Internet of Things gateway preprocessing is provided, which is applied to a community resident behavior prediction speculation method based on Internet of Things gateway preprocessing, and the device comprises:

[0299] (1) an Internet of Things collection system, configured to collect a multi-source heterogeneous data set of community residents according to a preset sampling frequency and upload the multi-source heterogeneous data set to an industrial gateway unit, wherein the multi-source heterogeneous data set comprises spatial flow behavior images and / or time series flow sensing data;

[0300] (2) an industrial gateway unit, configured to receive the multi-source heterogeneous data set and pre-process the multi-source heterogeneous data set, and perform feature supervision on the pre-processed multi-source heterogeneous data set, identify each modality data characteristic in the multi-source heterogeneous data set, and upload the single state to a background server; and / or, in response to a feature joint sampling instruction issued by the background server, fuse each modality data characteristic in the multi-source heterogeneous data set, generate corresponding community resident space-time characteristics, and upload the community resident space-time characteristics to the background server;

[0301] (3) a background server, configured to supervise whether the single state data characteristic is an alarm behavior characteristic:

[0302] if yes, a feature joint sampling instruction is issued to the industrial gateway unit; and / or, based on historical travel sequence big data of community residents, a behavior prediction result matched with the community resident space-time characteristics is speculated and recorded and saved;

[0303] otherwise, the data characteristic of the next modality is continuously supervised for alarm;

[0304] The Internet of Things collection system and the industrial gateway unit are in communication connection;

[0305] The industrial gateway unit and the background server are in communication connection.

[0306] The specific composition and interaction process of the device can be understood in combination with the above method steps, and will not be repeated here.

[0307] Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application, as Figure 7As shown, electronic device 410 may include a first processor 2001.

[0308] Optionally, the electronic device 410 may also include a memory 2002 and a transceiver 2003.

[0309] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0310] The following is combined Figure 4 A detailed description of each component of the electronic device 410 is provided below:

[0311] The first processor 2001 is the control center of the electronic device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0312] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0313] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0314] In a specific implementation, as one example, the electronic device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0315] The memory 2002 is configured to store a software program for implementing the scheme of the present application, and the first processor 2001 is configured to control the execution of the software program. The specific implementation can refer to the method embodiments described above, and will not be described here.

[0316] Alternatively, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magneto-optical disk storage (including a compact disk, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory 2002 can be integrated with the first processor 2001, or can exist independently and be coupled to the first processor 2001 through an interface circuit (not shown) of the electronic device 410. The embodiments of the present application are not limited in this regard. Figure 7

[0317] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0318] Alternatively, the transceiver 2003 can include a receiver and a transmitter (not shown separately in the figure). The receiver is configured to implement the receiving function, and the transmitter is configured to implement the transmitting function. Figure 7

[0319] Alternatively, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and be coupled to the first processor 2001 through an interface circuit (not shown) of the electronic device 410. The embodiments of the present application are not limited in this regard. Figure 7 It should be noted that the structure of the electronic device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0320] Figure 7 The structure of the electronic device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0321] ​​​In addition, the technical effects of the electronic device 410 can refer to the technical effects of the inference method for community resident behavior prediction based on gateway pre-processing described in the above method embodiments, which will not be repeated here.

[0322] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0323] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

[0324] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0325] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0326] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0327] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0328] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0329] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0330] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0331] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0332] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0333] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0334] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A speculation method for community resident behavior prediction based on Internet of Things gateway preprocessing, characterized in that, The method comprises: S1, collecting a multi-source heterogeneous data set of community residents according to a preset sampling frequency and uploading an industrial gateway unit, wherein the multi-source heterogeneous data set comprises a spatial flow behavior image and / or time series flow sensing data; S2, receiving the multi-source heterogeneous data set by the industrial gateway unit and preprocessing the multi-source heterogeneous data set, supervising the preprocessed multi-source heterogeneous data set, identifying each modal data feature in the multi-source heterogeneous data set and uploading the single modal data feature to the background server; S3, supervising by the background server whether the single modal data feature is an alarm behavior feature: If yes, a feature joint sampling instruction is issued to the industrial gateway unit, and step S4 is entered; Otherwise, the alarm supervision is continued for the data feature of the next modal; S4, fusing each modal data feature in the multi-source heterogeneous data set by the industrial gateway unit in response to the feature joint sampling instruction, generating corresponding community resident space-time features and uploading them to the background server; S5, the background server infers a behavior prediction result matched with the community resident space-time feature based on historical travel sequence big data of community residents and records and saves it. 2.The method of claim 1, wherein, The industrial gateway unit comprises: (1) a processor for providing data processing and logical communication; (2) a data classifier for classifying the multi-source heterogeneous data set into corresponding spatial flow behavior images or time series flow sensing data and inputting them into the model channel of the spatial-time dual-channel feature processing model according to the type; (3) the spatial-time dual-channel feature processing model comprises: a spatial feature processing channel based on MobileNetV3, for identifying and extracting spatial flow behavior image features in the spatial flow behavior image; a time series feature processing channel based on BiLSTM network and 1D-CNN network, for identifying and extracting time series flow sensing features in the time series flow sensing data; (4) a feature fusion device for using a gated feature fusion mechanism to fuse the spatial flow behavior image features and the time series flow sensing features to generate community resident space-time features; (5) an industrial communication gateway module for uploading the community resident space-time features to the background server in real time. 3.The method of claim 2, wherein, In S2, the identification of each modal data feature in the multi-source heterogeneous data set and the single modal upload to the background server are as follows: From the modal data features generated by the industrial gateway unit this time: the spatial flow behavior image features or the time series flow sensing features, a single modal data feature is randomly extracted and uploaded to the background server by the industrial communication gateway module.

4. The method of claim 2, wherein the method further comprises: In S5, the background server infers a behavior prediction result matched with the community resident space-time feature based on historical travel sequence big data of community residents and records and saves it, which comprises: Based on the historical travel sequence big data of a plurality of community residents collected and stored on the background server, a resident behavior prediction model is constructed; Input the spatiotemporal characteristics of the community residents into the resident behavior prediction model, and predict the behavior prediction result of the community residents matching the spatiotemporal characteristics of the community residents at the next spatiotemporal point by the resident behavior prediction model; Bind the behavior prediction result of the community residents at the next spatiotemporal point with the community resident ID, and record and save it to the background database.

5. The method of claim 4, wherein the method further comprises: The method for constructing the resident behavior prediction model comprises: Collecting historical travel sequence big data of a plurality of community residents; Extracting the spatiotemporal characteristics of each community resident at different spatiotemporal points in the historical travel sequence big data, and labeling the behavior prediction characteristics at the next spatiotemporal point; Counting each labeled behavior prediction characteristic to obtain a feature set, and dividing it into a training set and a validation set according to a preset proportion; Inputting the training set into a preset LSTM model to generate an initial resident behavior prediction model; Verifying the behavior prediction accuracy of the initial resident behavior prediction model by using the validation set: If the verification is qualified, deploy and apply the resident behavior prediction model to the background server; Otherwise, retrain the resident behavior prediction model.

6. A speculation device for community resident behavior prediction based on Internet of Things gateway pre-processing, the speculation device for community resident behavior prediction based on Internet of Things gateway pre-processing is used to realize the speculation method for community resident behavior prediction based on Internet of Things gateway pre-processing as any one of claims 1-5, characterized in that, The device comprises: (1) an Internet of Things collection system for collecting a multi-source heterogeneous data set of community residents according to a preset sampling frequency and uploading it to an industrial gateway unit, wherein the multi-source heterogeneous data set comprises spatial flow behavior images and / or time series flow sensing data; (2) an industrial gateway unit for receiving the multi-source heterogeneous data set and preprocessing the multi-source heterogeneous data set; and supervising the preprocessed multi-source heterogeneous data set to identify the data characteristics of each modality in the multi-source heterogeneous data set and upload them to the background server in a single state; and / or, in response to the feature joint sampling instruction issued by the background server, fuse the data characteristics of each modality in the multi-source heterogeneous data set to generate corresponding spatiotemporal characteristics of community residents and upload them to the background server; (3) a background server for supervising whether the data characteristics of the single state are alarm behavior characteristics: If yes, issue a feature joint sampling instruction to the industrial gateway unit; and / or, based on the historical travel sequence big data of community residents, predict the behavior prediction result matching the spatiotemporal characteristics of the community residents and record and save it; Otherwise, continue to supervise the data characteristics of the next modality for alarm; The Internet of Things collection system is in communication connection with the industrial gateway unit; The industrial gateway unit is in communication connection with the background server.

7. An electronic device, comprising: The electronic device comprises: a processor; a memory having computer readable instructions stored thereon, wherein the computer readable instructions are executed by the processor to implement the method of any one of claims 1 to 5.

8. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code, which can be called and executed by the processor to implement the method of any one of claims 1 to 5.

Images