A gate management method, device, equipment and medium based on multimodal data
Through the multimodal data management method, deep learning and attention mechanism are used to extract the multimodal data characteristics of the gate, and combined with Q learning and state transfer matrix model, the problem of difficulty in monitoring a single mode data is solved and the gate is improved Accuracy and responsiveness of abnormal monitoring.
Patent Information
- Application Number
- CN202411144825.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-20
AI Technical Summary
At this stage, gate abnormality monitoring relies on single mode data, making it difficult to capture comprehensive and complex gate data, resulting in low monitoring accuracy.
Using the gate management method based on multimodal data, by obtaining the multimodal data of the gate, using deep learning models such as gated loop network and attention mechanism to extract feature representations, and dynamically adjust them through the Q learning reward mechanism, combined with The state transition matrix and Bayesian fault prediction model perform exception monitoring.
The accuracy and dynamic response capabilities of gate abnormality monitoring are improved, the accuracy of multimodal data feature extraction and data fusion are enhanced, and the abnormality monitoring capabilities of gate abnormality monitoring are improved.
Smart Images

Figure CN118656794B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present application relate to the technical field of gate safety management, and in particular, to a gate management method, device, equipment and medium based on multimodal data. Background Art
[0002] The gate is a channel blocking device used to manage the flow of people and regulate the entry and exit of pedestrians. It is mainly used in subway gate systems and toll gate systems. Its core function is to allow only one person to pass at a time, and it can be used at the entrance of toll and access control occasions.
[0003] At the current stage, the data feature extraction of gate abnormality monitoring is often not comprehensive enough, that is, it relies on data of a single mode and it is difficult to capture comprehensive and complex gate data, which leads to low accuracy of gate abnormality monitoring. Summary of the invention
[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
[0005] The main purpose of the embodiments of the present disclosure is to propose a gate management method, device, equipment and medium based on multimodal data, which can improve the accuracy of gate abnormality monitoring.
[0006] A first aspect of an embodiment of the present application provides a gate management method based on multimodal data, and the gate management method based on multimodal data includes:
[0007] Obtain multi-modal gate data of the target gate;
[0008] The multimodal gate data is fused according to the fusion model to obtain the multimodal feature representation of the target gate, wherein the multimodal gate data is fused according to the fusion model to obtain the multimodal feature representation of the target gate, including:
[0009] Extracting a first feature representation corresponding to each mode of gate data from the multimodal gate data through a first gated recurrent network or a long short-term memory network;
[0010] The first feature representation of the multimodal gate data is fused through a first attention mechanism to obtain a second feature representation of the target gate;
[0011] By using a Q-learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data to obtain a multimodal feature representation of the target gate;
[0012] Based on a second gated recurrent network and a second attention mechanism, the target gate is monitored for abnormalities according to the multimodal feature representation.
[0013] The embodiments of the present application have at least the following beneficial effects:
[0014] This embodiment extracts multimodal gate data, extracts the feature representation of each modal gate data, and fuses the feature representation of each modal gate data using feature fusion technology, ultimately achieving gate abnormality monitoring of multimodal gate data. The embodiment adds a more comprehensive data source and improves the accuracy of gate abnormality monitoring.
[0015] The fusion model of this embodiment extracts the feature representation of multimodal gate data by setting a deep learning model, thereby improving the fusion model's ability to capture time series data such as multimodal gate data; by setting an attention mechanism to fuse the feature representations of gate data of different modalities, the dynamic fusion capability of feature representations of gate data of different modalities is improved; dynamic feature learning is set to dynamically adjust the feature representations fused by the attention mechanism to improve the dynamic optimization capability of the model. Through the above methods, the accuracy of the model's feature extraction of multimodal data can be improved, the accuracy of data fusion can be improved, and ultimately the accuracy of gate abnormality monitoring can be improved. Moreover, the method of this embodiment uses a gated recurrent network and an attention mechanism to learn multimodal feature representations, which can also improve the accuracy of gate abnormality monitoring.
[0016] In some embodiments of the present application, before adjusting the second feature representation according to the multimodal gate data through the Q learning reward mechanism, the gate management method based on multimodal data further includes:
[0017] Optimizing the second feature representation according to the triplet loss to obtain an optimized second feature representation;
[0018] The adjusting the second feature representation according to the multimodal gate data by using the Q learning reward mechanism includes:
[0019] The optimized second feature representation is adjusted according to the multimodal gate data through a Q-learning reward mechanism.
[0020] In some embodiments of the present application, before adjusting the second feature representation according to the multimodal gate data through the Q learning reward mechanism, the gate management method based on multimodal data further includes:
[0021] The fusion model is subjected to stochastic gradient descent optimization or adaptive optimization.
[0022] In some embodiments of the present application, the multimodal gate data includes gate pass records;
[0023] After acquiring the multimodal gate data of the target gate, the gate management method based on multimodal data further includes:
[0024] Through the time series prediction model, the future gate pass records are predicted based on the historical gate pass records:
[0025] ;
[0026] in, Indicated in The gate pass record at that time, express Upper limit value, Indicated in The gate pass record at that time, represents the parameters of the time series forecasting model, represents the weight of external influencing factors, represents the external influencing factor, represents the error term;
[0027] The target gate is monitored for abnormalities based on the future gate passage record.
[0028] In some embodiments of the present application, the multimodal gate data includes historical gate status data;
[0029] In the step of acquiring multimodal gate data of the target gate, the gate management method based on multimodal data further includes:
[0030] Through the state transfer matrix model, the future gate state is predicted based on the historical gate state data:
[0031] ;
[0032] in, Indicated in The gate status data at that time, Indicated in The gate status data at that time, represents the state transfer matrix, represents the control input vector, represents the control matrix, represents the noise vector;
[0033] By using the Bayesian fault prediction model, the target gate is monitored for abnormalities according to the future gate state.
[0034] In some embodiments of the present application, the multimodal gate data includes gate environment data, and the gate environment data includes temperature, humidity and light intensity;
[0035] After acquiring the multimodal gate data of the target gate, the gate management method based on multimodal data further includes:
[0036] The operating status of the target gate is calculated according to the temperature, humidity and light intensity through a linear regression model:
[0037] ;
[0038] in, Indicates the running status. Indicates temperature, Indicates humidity, Indicates the light intensity, represents a constant term, represents the regression coefficient, represents the error term;
[0039] Through the support vector machine model, the target gate is monitored for abnormalities according to the operating status of the target gate:
[0040] ;
[0041] in, represents the failure probability, represents the Lagrange multiplier, The category labels of the training samples in represents the kernel function, represents the support vector, represents the gate environment data, Represents the bias term.
[0042] In some embodiments of the present application, the multimodal gate data includes visitor data; the visitor data includes face recognition data, fingerprint data and iris data;
[0043] After acquiring the multimodal gate data of the target gate, the gate management method based on multimodal data further includes:
[0044] Verify the identity of the visitor based on the visitor data:
[0045] ;
[0046] in, Indicates the verification result of the visitor's identity. Represents face recognition data, Represents fingerprint data, Represents iris data, represents the corresponding weight coefficient;
[0047] The target gate is monitored for abnormalities according to the identity of the visiting person.
[0048] A second aspect of an embodiment of the present application provides a gate management device based on multimodal data, and the gate association device based on multimodal data includes:
[0049] A data acquisition unit, used for acquiring multi-modal gate data of a target gate;
[0050] A data fusion unit is used to fuse the multimodal gate data according to a fusion model to obtain a multimodal feature representation of the target gate, wherein the fusing of the multimodal gate data according to the fusion model to obtain the multimodal feature representation of the target gate includes:
[0051] Extracting a first feature representation corresponding to each mode of gate data from the multimodal gate data through a first gated recurrent network or a long short-term memory network;
[0052] The first feature representation of the multimodal gate data is fused through a first attention mechanism to obtain a second feature representation of the target gate;
[0053] By using a Q-learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data to obtain a multimodal feature representation of the target gate;
[0054] An anomaly monitoring unit is used to perform anomaly monitoring on the target gate according to the multimodal feature representation based on a second gated recurrent network and a second attention mechanism.
[0055] The third aspect of an embodiment of the present application proposes an electronic device, comprising at least one controller and a memory for communicating with the controller; the memory stores instructions that can be executed by the at least one controller, and the instructions are executed by the at least one controller so that the at least one controller executes the gate management method based on multimodal data as described above.
[0056] A fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the gate management method based on multimodal data as described above is implemented.
[0057] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0059] Figure 1 It is a flowchart of an embodiment of a gate management method based on multimodal data provided by the present application;
[0060] Figure 2 It is a thermal map of gate passage records provided in an embodiment of the present application;
[0061] Figure 3 is a gate state change frequency diagram provided by an embodiment of the present application;
[0062] Figure 4 is a random variation diagram of gate failure probability provided by an embodiment of the present application;
[0063] Figure 5 is a gate failure probability prediction diagram provided by an embodiment of the present application;
[0064] Figure 6 It is a pie chart of visitor distribution provided in an embodiment of the present application;
[0065] Figure 7 It is a Sankey diagram of visitor access paths provided in an embodiment of the present application;
[0066] Figure 8 It is a historical behavior pattern heat map provided by an embodiment of the present application;
[0067] Fig. 9 This is a diagram of a dynamic contrast enhancement network structure of a gate provided in an embodiment of the present application;
[0068] Fig.10 It is a structural schematic diagram of a gate management device based on multimodal data provided in an embodiment of the present application;
[0069] Fig.11 It is a schematic diagram of the structure of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION
[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0071] like Figure 1As shown, an embodiment of the present application provides a gate management method based on multimodal data, and the method includes the following steps S110 to S130:
[0072] Step S110, obtaining multi-modal gate data of the target gate;
[0073] Step S120, fusing the multimodal gate data according to the fusion model to obtain a multimodal feature representation of the target gate;
[0074] Step S130: Based on the second gated recurrent network and the second attention mechanism, perform abnormal monitoring on the target gate according to the multimodal feature representation.
[0075] In step S110 of this embodiment, the gate is a channel blocking device. The gate type and installation environment are not specifically limited here. The target gate refers to the gate that needs to be monitored. The multimodal gate data here includes but is not limited to: gate pass records, gate status data, gate environment data, and gate visitor data. The following briefly describes the source of the data:
[0076] (1) Gate pass records: Sensors are installed on the gate, including RFID readers, infrared sensors and cameras, and then these sensors record the time, location and identity information of each pass in real time.
[0077] (2) Gate status data: A door magnetic switch sensor, infrared sensor and status indicator sensor are installed on each gate, and then these sensors are used to record the gate's switch status, traffic conditions and fault information in real time.
[0078] (3) Gate environmental data: use temperature sensors, humidity sensors and light sensors to monitor and record environmental parameters in real time.
[0079] (4) The gate access personnel data, using high-precision facial recognition technology to collect and verify the identity of the person, and using fingerprint, iris and other sensors to collect data.
[0080] In step S120, the fusion model includes: a deep neural network (including a first gated recurrent network or a long short-term memory network), a first attention mechanism, and a Q learning reward mechanism; multimodal gate data is fused according to the fusion model to obtain a multimodal feature representation of the target gate, including the following steps S1210 to S1230:
[0081] Step S1210: extracting a corresponding first feature representation from the multimodal gate data through a first gated recurrent network or a long short-term memory network;
[0082] Step S1220: The first feature representation is fused through the first attention mechanism to obtain a second feature representation of the target gate;
[0083] Step S1230: Through the Q learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data to obtain the multimodal feature representation of the target gate.
[0084] In this embodiment S1210, it is necessary to perform feature extraction on the multimodal gate data, that is, to use a deep neural network to extract high-level feature representation, first convert the multimodal gate data into a time series, and then use GRU (i.e., gated recurrent network) or LSTM (i.e., long short-term memory network) to capture the long-term dependencies in the data sequence. Taking GRU (i.e., gated recurrent network) as an example, it specifically includes:
[0085] (1)
[0086] in, represents the hidden state at the current time step, represents the weight matrix, represents the input of the current time step, Represents the bias term. The input of each time step contains all the multimodal gate data at the current moment (for example, travel time, identity of the passerby, gate status, environmental parameters, etc.). These inputs are operated through the weight matrix and hidden state, and the hidden state of the current time step is output, which represents the feature representation at the current moment.
[0087] This embodiment uses a recurrent neural network to extract the feature representation (i.e., the first feature representation) of the multimodal gate data, which improves the ability to capture time series data compared to the current sequence processing formula.
[0088] After extracting the feature representation, step S1220 needs to fuse the extracted features, using the attention mechanism (Attention) to fuse the first feature representation of the extracted multi-modal gate data together to form the second feature representation of the target gate (the feature representation after fusion of features) to adapt to the changes in data features in different environments. Among them, the feature fusion uses the attention mechanism formula, which is:
[0089] (2)
[0090] in, represents the query matrix, represents the bond matrix, represents the value matrix, Represents the dimension of the key matrix. The hidden state is used as the input of the query matrix, key matrix and value matrix, and the weighted feature representation is calculated through the attention mechanism formula. These weighted feature representations integrate the features of different time steps to form a more representative feature vector of multi-modal gate data. Compared with the traditional feature fusion formula, this embodiment adopts the attention mechanism, which integrates the important information of each time step in the sequence data and improves the dynamic fusion capability of different modal features.
[0091] After obtaining the second feature representation, step S1230 uses reinforcement learning to optimize the feature representation through interaction with the environment. A reward mechanism is designed to adjust the feature representation based on the feedback of multimodal gate data, such as good decisions made based on the current state based on the real-time feedback of traffic records and environmental data, so as to enable the fusion model to learn better feature representation. Among them, the reward mechanism uses the Q learning formula, which is:
[0092] (3)
[0093] in, represents the state-action value function, represents the learning rate, Represents the reward value, Represents the discount factor.
[0094] Here, the state-action value function is updated according to the feedback value of the real-time environment and traffic conditions. Here, traffic records and environmental information are used as states, and the feature representation extracted by the model is used as actions, which are updated through the Q learning formula. The design of the reward mechanism enables the fusion model to continuously optimize the feature representation according to the actual situation, improving the dynamic response ability and adaptability of the fusion model.
[0095] In step S130 of the present embodiment, based on the second gated recurrent network and the second attention mechanism, the target gate is monitored for abnormalities according to the multimodal feature representation, that is, abnormal behaviors are monitored in real time through the fused data.
[0096] Here, anomaly monitoring uses an anomaly monitoring algorithm based on GRU (gated recurrent network) and attention mechanism, specifically:
[0097] (4)
[0098] (5)
[0099] in, represents the attention weight, represents the weight matrix, represents the hidden state, represents the bias term, Represents the context vector.
[0100] The beneficial effects of this embodiment include:
[0101] Traditional methods often rely on data from a single modality. This embodiment extracts multi-modal gate data of the gate, and then uses feature extraction technology to extract the feature representation of each modality of gate data. Then, feature fusion technology is used to fuse the feature representation of each modality of gate data, and finally, multi-modal gate data can be used to monitor gate anomalies, increase more comprehensive data sources, and improve the accuracy of gate anomaly monitoring.
[0102] The fusion model of this embodiment extracts the feature representation of multimodal gate data by setting a deep learning model, thereby improving the fusion model's ability to capture time series data such as multimodal gate data; by setting an attention mechanism to fuse the feature representations of gate data of different modalities, the dynamic fusion capability of feature representations of gate data of different modalities is improved; dynamic feature learning is set to dynamically adjust the feature representations fused by the attention mechanism to improve the dynamic optimization capability of the model. Through the above methods, the accuracy of the model's feature extraction of multimodal data can be improved, the accuracy of data fusion can be improved, and ultimately the accuracy of gate abnormality monitoring can be improved. Moreover, the method of this embodiment uses a gated recurrent network and an attention mechanism to learn multimodal feature representations, which can also improve the accuracy of gate abnormality monitoring.
[0103] In some embodiments of the present application, before adjusting the second feature representation according to the multimodal gate data by the Q learning reward mechanism in step S1230, the method further includes the following steps:
[0104] The fusion model is optimized according to the triplet loss to obtain a second feature representation after optimizing the second feature representation;
[0105] Through the Q-learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data, including:
[0106] Through the Q-learning reward mechanism, the optimized second feature representation is adjusted according to the multimodal gate data.
[0107] This embodiment uses a contrastive learning framework to enable the fusion model to learn the correlation between different modalities. The contrastive loss function is used to optimize the model so that similar samples are closer in the feature space and dissimilar samples are farther away. Among them, contrastive learning uses the triple loss formula, which is:
[0108] (6)
[0109] in, represents the distance between samples, represents the anchor point sample, represents the positive sample, represents negative samples, Represents the interval value. The input samples of contrastive learning include anchor samples, positive samples, and negative samples; specifically, anchor samples and positive samples come from similar traffic records or the same environmental state, while negative samples come from different traffic records or environmental states. The fusion model is optimized by the triplet loss formula, so that similar samples are closer in the feature space and dissimilar samples are farther away, thereby enhancing the model's ability to distinguish features.
[0110] In some embodiments of the present application, before adjusting the second feature representation according to the multimodal gate data through the Q learning reward mechanism, the gate management method based on multimodal data further includes:
[0111] Perform stochastic gradient descent optimization or adaptive optimization on the ensemble model.
[0112] In this embodiment, stochastic gradient descent (SGD) or an adaptive optimization algorithm (such as Adam) can be used to optimize model parameters. Taking the adaptive optimization algorithm Adam as an example, the specific method is:
[0113] (7)
[0114] (8)
[0115] (9)
[0116] (10)
[0117] (11)
[0118] in, and represent the momentum and square momentum of the gradient, respectively. represents the model parameters, represents the learning rate, and represents the exponential decay rate of the momentum term, is a decimal value that prevents division by zero.
[0119] This embodiment adopts momentum and adaptive learning rate adjustment to improve optimization efficiency and stability.
[0120] like Figures 2 to 10 In order to facilitate the understanding of those skilled in the art, some embodiments of the present application provide a gate management method based on multimodal data, and the method includes the following steps S910 to S950:
[0121] Step S910: Collect multi-modal gate data, and use the multi-modal gate data to monitor gate abnormalities.
[0122] Step S910 includes the following process:
[0123] Step S9110: Multi-modal gate data includes gate pass records. The data can be obtained in the following ways:
[0124] First, Internet of Things (IoT) sensors are installed at each pass point. These sensors include RFID readers, infrared sensors and cameras. These sensors then record the time, location and identity information of each pass in real time.
[0125] Then, the gate traffic records are analyzed to predict the peak traffic period and monitor abnormal behavior. The peak period prediction uses a time series prediction model, which is as follows:
[0126] (12)
[0127] in, Indicated in The gate pass record at that time, express Upper limit value, Indicated in The gate pass record at that time, represents the parameters of the time series forecasting model, represents the weight of external influencing factors, represents the external influencing factor, represents the error term.
[0128] In this embodiment, the time series prediction model uses external influencing factors and dynamic model parameter adjustment to improve the accuracy of the prediction.
[0129] The results of the time series forecasting model are as follows Figure 2 As shown in the figure, the heat map shows the number of pass records for each gate at different times of the day. For example, Gate A has 11 pass records in a certain hour, indicating that there is a significant peak in the period. By using past pass records and external influencing factors, the time series prediction model can predict future peak periods and monitor abnormal behaviors, such as Gate B's pass records suddenly increasing to 10 times in a certain hour. At the same time, the accuracy of the time series prediction model can be evaluated by comparing the pass records predicted by the formula with the actual number of records in the heat map. If the predicted values for certain periods are highly consistent with the actual values, it means that the model parameters are too high. and external factors If the settings are reasonable, the prediction effect is good. On the contrary, if there is a significant deviation, it is necessary to adjust the model parameters or consider more external influencing factors to improve the prediction accuracy.
[0130] Step S9120: The multi-modal gate data includes gate status data. A variety of sensors are installed on each gate, including a door magnetic switch sensor, an infrared sensor, and a status indication sensor, and then the switch status, traffic conditions, and fault information of the gate are recorded in real time through these sensors.
[0131] The future gate state is predicted through the state transfer matrix model:
[0132] (13)
[0133] in, Indicated in The gate status data at that time, Indicated in The gate status data at that time, represents the state transfer matrix, represents the control input vector, represents the control matrix, represents the noise vector.
[0134] This embodiment uses a state transition matrix to improve the accuracy of recording and the dynamic response capability to state changes.
[0135] Then, the Bayesian fault prediction model is used to monitor the target gate for abnormalities according to the future gate status. The Bayesian reasoning method is used to improve the accuracy and adaptability of fault prediction.
[0136] The final result is Figure 3 , Figure 4 As shown, Figure 3 The bar chart shows the frequency of the gate's "open" and "closed" states during the recording period. It can be seen that the gate's "open" state changes more frequently than the "closed" state, indicating that the gate was in the open state more frequently during this time period. Figure 4 The line graph shows the change of failure probability at different time points. The graph shows that the failure probability increases significantly at certain time points (such as 02:00 and 06:00), indicating that there may be a higher failure risk at these time points, suggesting that maintenance and inspection need to be strengthened.
[0137] Step S9130: The multi-modal gate data includes gate environment data. The temperature sensor, humidity sensor and light sensor are used to monitor and record the environmental parameters in real time.
[0138] First, the operating status of the target gate is calculated based on temperature, humidity and light intensity through a linear regression model:
[0139] (14)
[0140] in, Indicates the running status. Indicates temperature, Indicates humidity, Indicates the light intensity, represents a constant term, represents the regression coefficient, represents the error term.
[0141] This embodiment uses regression analysis, which improves the accuracy of prediction of the impact of environmental parameters compared to a single parameter analysis method.
[0142] Then, the support vector machine model is used to monitor the abnormality of the target gate according to its operating status:
[0143] (15)
[0144] in, represents the failure probability, represents the Lagrange multiplier, The category labels of the training samples in represents the kernel function, represents the support vector, Indicates the gate environment data (temperature, humidity and light intensity), Represents the bias term.
[0145] This embodiment adopts the kernel method, which can better perform classification in high-dimensional space and improve the accuracy and adaptability of prediction.
[0146] The final result is Figure 5 As shown in the figure, the failure probability predicted by the support vector machine (SVM) model is shown. Each point in the figure represents a sample. The vertical axis represents the predicted failure probability, and the horizontal axis represents the sample number. The figure shows that the model can effectively distinguish between faulty and normal samples. Most faulty samples have a high predicted failure probability, and most normal samples have a low predicted failure probability, indicating that the model has a high prediction accuracy.
[0147] Step S9150: The multi-modal gate data includes gate access personnel data, and the personnel identity is collected and verified by using high-precision face recognition, fingerprint, iris, etc.
[0148] First, verify the identity of the visitor based on the visitor data:
[0149] (16)
[0150] in, Indicates the verification result of the visitor's identity. Represents face recognition data, Represents fingerprint data, Represents iris data, Represents the corresponding weight coefficient.
[0151] The above formula adopts multiple verifications. Compared with a single verification method, this embodiment improves the accuracy and security of verification.
[0152] Then, the target gate is monitored for abnormalities based on the identity of the visitor. The specific process is as follows:
[0153] Use the access permission management model to record and manage the access permissions of each person to ensure that only authorized persons can pass. Among them, the access permission uses the role permission comprehensive formula, which is:
[0154] (17)
[0155] in, Represents access authority, Representative identity, Represents role permissions, Compared with the traditional static authority formula, this embodiment adopts dynamic role weight, which improves the flexibility and accuracy of authority management.
[0156] Then, the position and department information model is used to record the position and department information of each passer, so as to facilitate the authority management and analysis of the pass records. Among them, the position and department information uses the department weight comprehensive formula, which is as follows:
[0157] (18)
[0158] in, Representative position and department information, Representative identity, Representative department information, is the department weight coefficient. Compared with the traditional simple recording formula, this embodiment adopts the department weight to improve the accuracy of information recording.
[0159] Then use the visitor information management model to record visitor information in detail, including visit time, visit reason, etc. Among them, the visitor information uses the visit reason comprehensive formula, which is:
[0160] (19)
[0161] in, Representative visitor information, Represents the access time, Representative's reason for visit, is the visit reason weight coefficient. Compared with the traditional visitor information recording formula, this embodiment adopts the reason weight, improves the comprehensiveness of visitor information recording, and innovatively solves the singleness of visitor information recording.
[0162] Then the gate usage monitoring model monitors the use of the equipment by personnel, such as the frequency and duration of use, in order to conduct behavioral analysis. Among them, the gate usage uses the gate usage comprehensive formula, which is:
[0163] (20)
[0164] in, Represents the use of the gate, Represents the frequency of use, Represents the usage time, , The present embodiment adopts the frequency and duration weights to improve the comprehensiveness of the gate usage record.
[0165] The final result is Figure 6 , Figure 7 , Figure 8 As shown, Figure 6 The distribution of personnel in each department is shown. The pie chart clearly shows the proportion of personnel in each department. For example, the technical department accounts for 30%, the marketing department accounts for 20%, the sales department accounts for 25%, the human resources department accounts for 15%, and the finance department accounts for 10%.
[0166] Figure 7 Shows the visitor's access path. The Sankey diagram shows the flow of visitors entering and leaving different areas. For example, after entering the system, a visitor can visit area A or area B and then leave. This kind of diagram can help managers understand the visitor's access flow and optimize visitor management and arrangement.
[0167] Figure 8 The heat map shows the historical behavior patterns of different people. The heat map uses color depth to indicate the frequency of behavior.
[0168] This embodiment can identify possible faults in advance by analyzing and predicting the gate machine data of multiple modes collected, so as to perform preventive maintenance. Through early prediction and analysis, the operating parameters and configuration of the gate machine system can be adjusted in real time to adapt to the ever-changing environment and needs.
[0169] Step S920, data cleaning and preprocessing: Before data analysis, the traffic records, gate status, environmental information and personnel information need to be cleaned and preprocessed. First, eliminate data noise, monitor and delete outliers, and ensure data accuracy. Then, unify the data format, standardize timestamps, text and numeric fields. Finally, handle missing data, fill or delete missing records, and ensure data integrity. Through these steps, data quality and consistency can be ensured, laying the foundation for subsequent analysis.
[0170] Step S930: construct a fusion model, and then implement multi-modal gate data fusion based on the model.
[0171] like Fig. 9 As shown, feature extraction is performed on the data preprocessed in step S920, and a deep learning model is used to extract high-level feature representation. In order to achieve dynamic feature learning, when processing sequence data, a gated recurrent unit (GRU) is used for feature extraction (ie, the first feature representation in the above step).
[0172] The extracted feature representations are fused together using the attention mechanism to form a multimodal feature representation (i.e., the second feature representation in the above step).
[0173] The contrastive learning framework enables the model to learn the correlation between different modalities. The contrastive loss function is used to optimize the model so that similar samples are closer in the feature space and dissimilar samples are farther away. Among them, contrastive learning uses the triple loss formula.
[0174] Introduce reinforcement learning to optimize feature representation through interaction with the environment. Design a reward mechanism to adjust feature representation based on feedback from traffic conditions, such as real-time feedback from traffic records and environmental information, to reward the system for making good decisions based on the current state, thereby prompting the model to learn better feature representation.
[0175] The collected multimodal gate data is used for model training to optimize the contrastive learning framework. The model parameters can be optimized using an adaptive optimization algorithm.
[0176] Step S940: real-time monitoring and early warning.
[0177] The fused data is used to monitor abnormalities in real time and send out early warning signals. At the same time, access control and management are implemented, and the gate switch strategy is dynamically adjusted according to the fused data to improve security.
[0178] Real-time monitoring of the fused data uses deep learning models and intelligent analysis algorithms to detect anomalies. Then, anomalies are detected and early warning signals are issued. For example, if the switching frequency of a gate is detected to be abnormally high within a certain period of time, the system will issue an early warning signal.
[0179] Next, the data is monitored in real time and the gate opening and closing strategies are adjusted dynamically. For example, based on the current traffic records and environmental information, the system can automatically adjust the gate opening and closing frequency and strategy to ensure safety and efficiency. Access control uses reinforcement learning algorithms, specifically:
[0180] (twenty one)
[0181] in, Represents in state Next select action The probability of Represents the state-action value function. Compared with the traditional static rules, this embodiment adopts reinforcement learning to improve the adaptive ability.
[0182] Through the above steps, the gate machine can be monitored and warned, abnormalities can be detected and warning signals can be issued. At the same time, the gate machine's opening and closing strategy can be dynamically adjusted according to the fusion data to improve safety and efficiency.
[0183] Step S950, decision support based on fused data: generating reports and suggestions for reference by managers.
[0184] This embodiment has at least the following beneficial effects:
[0185] The introduction of a fusion model, combined with deep learning and multimodal fusion technology, can achieve the fusion of multimodal gate data, which can more comprehensively extract data features and more effectively monitor abnormal behavior.
[0186] By using multimodal data fusion technology, information from different data sources is fused together to form a more comprehensive and accurate feature representation. Such feature representation can better capture the characteristics and behavior patterns of passers-by and improve the ability to monitor abnormal behavior.
[0187] The fusion contrast learning framework and the reinforcement learning method enable the fusion model to learn the correlation between different modes and dynamically adjust the feature representation according to the feedback of multi-modal gate data. This can further improve the accuracy of the fusion model and thus improve the accuracy of gate abnormality monitoring.
[0188] In one embodiment of the present application, a gate management device based on multimodal data includes: a data acquisition unit 1100, a data fusion unit 1200, and an abnormality monitoring unit 1300, specifically:
[0189] The data acquisition unit 1100 is used to acquire multi-modal gate data of a target gate;
[0190] The data fusion unit 1200 is used to fuse the multimodal gate data according to the fusion model to obtain the multimodal feature representation of the target gate, wherein the fusion model includes: a deep neural network, a first attention mechanism and a Q learning reward mechanism; fusing the multimodal gate data according to the fusion model to obtain the multimodal feature representation of the target gate includes:
[0191] Extracting a corresponding first feature representation from the multimodal gate data through a first gated recurrent network or a long short-term memory network;
[0192] The first feature representation is fused through the first attention mechanism to obtain the second feature representation of the target gate;
[0193] Through the Q-learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data to obtain the multimodal feature representation of the target gate;
[0194] The abnormality monitoring unit 1300 is used to monitor the target gate for abnormality based on the second gated recurrent network and the second attention mechanism according to the multimodal feature representation. It should be noted that the present device embodiment and the above method embodiment are the same inventive concept, so the content of the above method embodiment is applicable to the present device embodiment and will not be described in detail here.
[0195] like Fig.11 The present application also provides an electronic device, which includes:
[0196] at least one memory;
[0197] at least one processor;
[0198] at least one program;
[0199] The program is stored in the memory, and the processor executes at least one program to implement the gate management method based on multimodal data implemented in the present disclosure.
[0200] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.
[0201] The electronic device according to the embodiment of the present application is described in detail below.
[0202] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0203] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the program code is stored in the memory 1700, and the processor 1600 calls and executes the gate management method based on multimodal data in the embodiment of this application.
[0204] Input / output interface 1800, used to implement information input and output;
[0205] Communication interface 1900, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0206] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );
[0207] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .
[0208] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned gate management method based on multimodal data.
[0209] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0210] The embodiments described in this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0211] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0212] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0213] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0214] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0215] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0216] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0217] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0218] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0219] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.
[0220] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.
Claims
1. A gate management method based on multimodal data, characterized in that: The gate management method based on multimodal data includes: Obtain multi-modal gate data of the target gate; The multimodal gate data is fused according to the fusion model to obtain the multimodal feature representation of the target gate, wherein the multimodal gate data is fused according to the fusion model to obtain the multimodal feature representation of the target gate, including: Extracting a first feature representation corresponding to each mode of gate data from the multimodal gate data through a first gated recurrent network or a long short-term memory network; The first feature representation of the multimodal gate data is fused through a first attention mechanism to obtain a second feature representation of the target gate; By using a Q-learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data to obtain a multimodal feature representation of the target gate; Based on a second gated recurrent network and a second attention mechanism, performing abnormal monitoring on the target gate according to the multimodal feature representation; The multimodal gate data includes visitor data; the visitor data includes face recognition data, fingerprint data and iris data; After acquiring the multimodal gate data of the target gate, the gate management method based on multimodal data further includes: Verify the identity of the visitor based on the visitor data: ; in, Indicates the verification result of the visitor's identity. Represents face recognition data, Represents fingerprint data, Represents iris data, represents the corresponding weight coefficient; The target gate is monitored for abnormalities according to the identity of the visiting person.
2. The gate management method based on multimodal data according to claim 1 is characterized in that: Before adjusting the second feature representation according to the multimodal gate data through the Q learning reward mechanism, the gate management method based on multimodal data further includes: Optimizing the second feature representation according to the triplet loss to obtain an optimized second feature representation; The adjusting the second feature representation according to the multimodal gate data by using the Q learning reward mechanism includes: The optimized second feature representation is adjusted according to the multimodal gate data through a Q-learning reward mechanism.
3. The gate management method based on multimodal data according to claim 2 is characterized in that: Before adjusting the second feature representation according to the multimodal gate data through the Q learning reward mechanism, the gate management method based on multimodal data further includes: The fusion model is subjected to stochastic gradient descent optimization or adaptive optimization.
4. A gate management device based on multimodal data, characterized in that: The gate association device based on multimodal data includes: A data acquisition unit, used for acquiring multi-modal gate data of a target gate; A data fusion unit is used to fuse the multimodal gate data according to a fusion model to obtain a multimodal feature representation of the target gate, wherein the fusing of the multimodal gate data according to the fusion model to obtain the multimodal feature representation of the target gate includes: Extracting a first feature representation corresponding to each mode of gate data from the multimodal gate data through a first gated recurrent network or a long short-term memory network; The first feature representation of the multimodal gate data is fused through a first attention mechanism to obtain a second feature representation of the target gate; By using a Q-learning reward mechanism, the second feature representation is adjusted according to the multimodal gate data to obtain a multimodal feature representation of the target gate; an anomaly monitoring unit, configured to perform anomaly monitoring on the target gate according to the multimodal feature representation based on a second gated recurrent network and a second attention mechanism; The multimodal gate data includes visitor data; the visitor data includes face recognition data, fingerprint data and iris data; After acquiring the multimodal gate data of the target gate, the gate management method based on multimodal data further includes: Verify the identity of the visitor based on the visitor data: ; in, Indicates the verification result of the visitor's identity. Represents face recognition data, Represents fingerprint data, Represents iris data, represents the corresponding weight coefficient; The target gate is monitored for abnormalities according to the identity of the visiting person.
5. An electronic device, characterized in that: It includes at least one controller and a memory for communicating with the controller; the memory stores instructions that can be executed by the at least one controller, and the instructions are executed by at least one of the controllers to enable at least one of the controllers to execute the gate management method based on multimodal data as described in any one of claims 1 to 4.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the gate management method based on multimodal data as described in any one of claims 1 to 4.
