Intelligent security task processing method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, can be applied to the fields of intelligent security and protection, finance and medical treatment, and discloses an intelligent security and protection task processing method, device, equipment and medium, the method comprises the following steps: obtaining a multi-modal feature of a target object from a first video stream, the multi-modal feature comprising a visual feature, a language feature and an action feature; extracting scene features associated with the multi-modal task from the multi-modal features; obtaining a correlation degree between each modal feature and the scene feature; calculating the weight of each modal feature according to the correlation degree; determining a target fusion routing path of the multi-modal features according to the weight of each modal feature; fusing the multi-modal features according to the target fusion routing path to obtain a first target feature; performing enhancement processing on the first target feature to obtain a second target feature; and reasoning based on the multi-modal task instruction and the second target feature to obtain a target decision instruction. According to the scheme, deep scene understanding can be realized through multi-modal feature collaboration, and the reasonability of the target decision instruction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, which can be applied to the fields of intelligent security, finance and medical treatment, and in particular to a smart security task processing method, device, equipment and medium. BACKGROUND

[0002] Vision-Language-Action Model (VLA) is usually used in intelligent security, and since intelligent security deals with multi-modal tasks, the traditional multi-modal fusion method usually adopts static weight distribution or simple feature splicing, so when a VLA is used to process complex multi-modal tasks, it cannot dynamically adjust the fusion strategy according to different task scenarios and data characteristics. For example, in the intelligent security scenario, when the ambient light changes or abnormal behavior occurs, the model has difficulty in flexibly adjusting the fusion weights of the visual, language and action modalities, resulting in a decline in the ability to identify and respond to abnormal events. In addition, since the traditional VLA model lacks deep perception and understanding of the task context, the execution of the task often depends on specific context information such as time, place and participating subjects in actual application, and the traditional VLA model cannot effectively integrate these context factors, so that the decision of the VLA model in different contexts lacks pertinence and adaptability.

[0003] Therefore, the existing VLA model cannot well handle abnormal behavior and adapt to data changes and complex scenarios when processing multi-modal tasks. SUMMARY

[0004] The embodiments of the present application provide a smart security task processing method, device, equipment and medium, aiming to solve the problem that the VLA model in the prior art cannot well handle abnormal behavior and adapt to data changes and complex scenarios when processing multi-modal tasks.

[0005] In a first aspect, the embodiments of the present application provide a smart security task processing method, which comprises:

[0006] preprocessing a first video stream to be analyzed to obtain multi-modal features of a target object from the first video stream, the multi-modal features including visual features, language features and action features;

[0007] extracting context features associated with multi-modal tasks from the multi-modal features;

[0008] obtaining the correlation between each modality feature and the context features based on a multi-head self-attention mechanism;

[0009] calculating the weight of each modality feature according to the correlation between each modality feature and the context features;

[0010] determine a target fusion routing path of the multi-modal features according to the weights of the modal features;

[0011] fuse the multi-modal features according to the target fusion routing path to obtain first target features;

[0012] perform enhancement processing on the first target features to obtain second target features;

[0013] perform inference based on a multi-modal task instruction and the second target features to obtain a target decision instruction corresponding to the multi-modal features.

[0014] In a second aspect, an embodiment of the present application provides an intelligent security task processing apparatus, which comprises:

[0015] an input / output unit configured to acquire a first video stream to be analyzed;

[0016] a processing unit configured to perform preprocessing on the first video stream acquired by the input / output unit, and acquire multi-modal features of a target object from the first video stream, the multi-modal features comprising visual features, language features and action features;

[0017] The processing unit is further configured to extract a scene feature associated with a multi-modal task from the multi-modal features, acquire an association degree between each modal feature and the scene feature based on a multi-head self-attention mechanism, calculate a weight of each modal feature according to the association degree between each modal feature and the scene feature, determine a target fusion routing path of the multi-modal features according to the weights of the modal features, fuse the multi-modal features according to the target fusion routing path to obtain first target features, perform enhancement processing on the first target features to obtain second target features, and perform inference based on a multi-modal task instruction and the second target features to obtain a target decision instruction corresponding to the multi-modal features and output the target decision instruction through the input / output unit.

[0018] In a third aspect, an embodiment of the present application further provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the intelligent security task processing method of the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program causes a processor to execute the intelligent security task processing method of the first aspect when the computer program is executed by the processor.

[0020] The embodiment of the present application provides an intelligent security task processing method, device, equipment and medium, the method comprises the following steps: acquiring a multi-modal feature of a target object from a first video stream, the multi-modal feature comprising a visual feature, a language feature and a motion feature; extracting a scene feature associated with a multi-modal task from the multi-modal feature; acquiring an association degree between each modal feature and the scene feature; calculating the weight of each modal feature according to the association degree between each modal feature and the scene feature; determining a target fusion routing path of the multi-modal feature according to the weight of each modal feature; fusing the multi-modal feature according to the target fusion routing path to obtain a first target feature; performing enhancement processing on the first target feature to obtain a second target feature; and performing reasoning based on a multi-modal task instruction and the second target feature to obtain a target decision instruction. The present application can improve the accuracy of scene recognition by dynamically routing path adaptive fusion features, can realize deep scene understanding by introducing scene features, and thus can improve the rationality of the target decision instruction. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The flowchart of the intelligent security task processing method provided by an embodiment of the present application is shown in the figure.

[0023] Figure 2 is Figure 1 The flowchart of a specific embodiment of step S120 in the figure.

[0024] Figure 3 is Figure 1 The flowchart of a specific embodiment of step S170 in the figure.

[0025] Figure 4 Another flowchart of the intelligent security task processing method provided by an embodiment of the present application is shown in the figure.

[0026] Figure 5 The schematic block diagram of the intelligent security task processing device provided by an embodiment of the present application is shown in the figure.

[0027] Figure 6 The schematic block diagram of the computer equipment provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0029] It should be understood that the terms "include" and "contain" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0031] It should be further understood that the term "and / or" as used in the specification and the appended claims of the present application refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0032] Referring to Figure 1 , which is a flowchart of an intelligent security task processing method provided by an embodiment of the present application, the intelligent security task processing method is applied in an intelligent security system.

[0033] As Figure 1 shown, the intelligent security task processing method provided by the embodiment of the present application includes the following steps S110-S180.

[0034] S110, pre-processing a first video stream to be analyzed to obtain multi-modal features of a target object from the first video stream.

[0035] The multi-modal features include visual features, language features and action features.

[0036] Processing of visual features: For the input visual image or video stream, first use YOLOv8 target detection algorithm to quickly identify various objects in the image, obtain the class, position and bounding box information of the object. Then use the improved Swin Transformer to extract the features of the detected target region, get the visual feature vector FV\in\mathbb{R}^{m\times d_V}, where m is the number of features, and d_V is the dimension of the visual feature. At the same time, extract the global scene features of the image for subsequent context awareness analysis.

[0037] Processing of language features: For the input language text, first perform preprocessing operations such as word segmentation, part-of-speech tagging and named entity recognition. Use the BERT model to encode the text and obtain the word vector representation. Then through syntactic analysis and semantic role labeling, extract the syntactic structure and semantic relationship of the sentence, get the language feature vector FL\in\mathbb{R}^{n\times d_L}, n is the number of language features, and d_L is the dimension of the language feature.

[0038] Processing of action features: For the data collected by the action sensor (such as robot joint angle, human motion posture data), first perform normalization and filtering to remove noise interference. Then use Temporal Convolutional Network (TCN) to extract the time sequence dynamic features of the action, get the action feature vector FA\in\mathbb{R}^{p\times d_A}, p is the number of action features, and d_A is the dimension of the action feature. Integrate the feature vectors of the three modalities to get the preprocessed multi-modal feature F_{pre}=[FV;FL;FA].

[0039] For example, in the financial field, when high-frequency transaction risk control, intelligent investment research report generation, anti-money laundering monitoring, etc. need to be performed through the intelligent security task processing method in the embodiment of the application, the aforementioned visual features can be market data, such as stock K-line chart, transaction volume heat map (image / time series data); language features can be customer instructions, such as natural language instructions (such as "find low volatility technology stocks"); action features can be risk signals, such as real-time risk indicators (such as volatility index, leverage rate change). Correspondingly, the context features can be macroeconomic indicators (interest rate, inflation), customer historical behavior, market sentiment, so as to perform subsequent processes based on these data, realize the flexible balance of "market trend, customer demand, risk control", and avoid single signal misjudgment.

[0040] For example, in the medical field, when intelligent diagnosis needs to be achieved through the intelligent security task processing method in this embodiment of the invention, the aforementioned visual features can be replaced with medical images, such as CT scans, X-rays, and pathological slide images; the aforementioned linguistic features can be replaced with electronic medical record text, such as patient complaints, medical history records, and doctor's notes; the aforementioned action features can be replaced with physiological signals, such as electrocardiograms (ECG), electroencephalograms (EEG), and time-series data of blood oxygen saturation. Contextual features can be replaced with patient age / gender, epidemiological data (such as flu season), and hospital resource status (such as ICU bed shortages), thereby enabling subsequent processes based on this data to achieve real-time decision-making by the surgical robot, epidemic prediction, personalized medication recommendations, and a balance of diagnostic reliability.

[0041] S120. Extract contextual features associated with the multimodal task from the multimodal features.

[0042] The scenario feature Fctx is a vectorized representation of the task context extracted from multimodal data and external knowledge. It is used to dynamically guide multimodal fusion and includes multiple scenario elements. For example, in a smart security scenario, the basic scenario includes: time (2 AM), location (back door restricted area), and lighting conditions (low light); external knowledge includes: historical anomaly records for the area and mall security rules. The number of scenario elements (e.g., time, location, lighting, etc., a total of q elements) is specified.

[0043] In one embodiment, such as Figure 2 As shown, step S120 includes:

[0044] S1201. Extract the first scene element from visual features, the first scene element including scene type, ambient lighting conditions, and object layout; extract task keywords from multimodal task instructions from language features; and extract the location of the action and the subject involved in the action from action features.

[0045] Specifically, when extracting task-related contextual elements from preprocessed multimodal features, contextual elements can be extracted according to different modalities. For example, in the visual modality, information such as scene type (e.g., indoor, outdoor), ambient lighting conditions, and object layout can be extracted; in the linguistic modality, keywords such as time, location, and target object in task instructions can be extracted; and in the action modality, information such as the location of the action and the participating entities can be extracted. Simultaneously, contextual elements can be supplemented and improved by combining external sensor data (e.g., temperature and humidity sensors) and a pre-set knowledge base (e.g., common sense knowledge, domain knowledge).

[0046] S1202. Obtain supplementary elements, which are derived from external sensor data and a preset knowledge base.

[0047] S1203, fuse the supplementary element, the first scene element, the task keyword, the action occurrence position and the action participant subject and perform vectorization coding to obtain the scene feature.

[0048] Specifically, when the scene elements extracted in S1201 and S1202 are vectorized and coded, for a category type scene element (such as a scene type), a one-hot encoding or embedding encoding method is used to convert it into a vector; for a numerical type scene element (such as temperature), after normalization, it is directly used as an element of a vector. Then, a multi-layer perception (MLP) is used to fuse the coded scene elements to obtain a scene feature vector F_{ctx}∈\mathbb{R}^{q×d_{ctx}}, where q is the number of scene features, and d_{ctx} is the dimension of the scene feature.

[0049] In this embodiment, by extracting the scene from the multi-modal data (scene type, instruction keyword, sensor data) and the external knowledge base, the problem of insufficient scene understanding can be solved, thereby improving the matching degree of the VLA model decision and the scene. For example, in the intelligent security scene, the visual feature weight is adjusted in combination with the light condition.

[0050] In this embodiment, in order to more clearly understand the application scenarios in the present application, the following will describe a specific application scenario of monitoring abnormal behavior of a shopping mall after closing at night by an intelligent security system. The intelligent security system deploys the VLA model of the present application, which can process the following multi-modal inputs:

[0051] Visual features: for example, a monitoring camera captures a person lingering at the back door (nighttime is a low light condition). The "person" target is detected, and the SwinTransformer extracts local features (a blurred figure) and global scene features (low light).

[0052] Language features: for example, the voice instruction "check whether there is an anomaly in the back door area". The BERT parses the keywords "anomaly" and "back door" in the instruction and identifies them as high-priority tasks.

[0053] Action features: for example, the sensor of a patrol robot detects a change in the door lock state (angle sensor data). The door lock angle mutation time series data is analyzed to determine "abnormal opening".

[0054] Contextual features: e.g. time (2 a.m.), location (rear door access area), historical records (no normal activity in this area). The extracted contextual elements are time (2 a.m. -> high-risk period), area (access area), and sensor data (abnormal door lock). These contextual elements are vectorized and encoded as a vector Fctx with [high-risk period = 1, access area = 1, historical abnormality = 0]. The contextual features of the mall security rules are combined to reinforce the mall's nighttime closing scenario.

[0055] After obtaining the above multi-modal features, the weights of each modal feature are calculated, for example, SV = 0.8 (visual key in low light), SL = 0.6 (clear instructions), and SA = 0.9 (abnormal door lock action is significant).

[0056] After obtaining the weights, since SA is the largest, the action priority fusion path is selected based on the weights of each modal feature, and the action feature FA and the visual feature FV (e.g. door lock data correction human detection frame) are fused by weighting to obtain the fused first target feature.

[0057] Correspondingly, the Bi-LSTM associates the time series of "person wandering" and "door opening", and the GNN constructs a semantic graph: person - close to door lock -> infer "attempt to break in". According to the time series and the semantic graph, the target decision instruction "trigger alarm and notify security personnel" is generated.

[0058] It can be understood that since it is a nighttime scenario after closing, it can be confirmed whether it is a real intrusion when executing the target decision instruction, and if it is a false alarm, it is labeled as "light interference". The visual weight SV in low light and the action modal confidence threshold are adjusted accordingly.

[0059] As can be seen, in the mall's nighttime closing scenario, the model prioritizes processing action sensor data through dynamic routing, enhances decision rationality by combining contextual awareness (time, area rules), and finally achieves accurate alarm by correlating scattered clues (human + door lock) through multi-modal reasoning. In contrast, the traditional model may miss detection due to static fusion or neglect of context, and may be judged as "normal patrol". It can be seen that the present scheme significantly improves the security response capability in complex scenarios.

[0060] S130, obtaining the correlation between each modal feature and the contextual feature based on a multi-head self-attention mechanism.

[0061] Specifically, the context feature vector F_{ctx} and the pre-processed multi-modal feature F_{pre} are input into the modal importance evaluation network. The network is based on the Transformer architecture and calculates the correlation between different modal features and contextual features through a multi-head self-attention mechanism.

[0062] For example, in the smart security scenario, the context feature Fctx: [time = 2am, area = no-entry zone, light = low]

[0063] The visual feature FV: blurry figure (low confidence)

[0064] The language feature FL: instruction "check the back door"

[0065] The action feature FA: door lock angle mutation

[0066] Then, the visual feature correlation degree is: the "light = low" in the context feature is highly correlated with the visual feature "blurry figure", but the target detection confidence is low → SV = 0.6

[0067] The language feature correlation degree is: the instruction "back door" matches the context "no-entry zone" → SL = 0.7

[0068] The action modality correlation degree is: the door lock mutation is strongly related to the context "early morning + no-entry zone" → SA = 0.9

[0069] The normalized score: [SV, SL, SA] = Softmax([0.6, 0.7, 0.9]) = [0.23, 0.27, 0.50]

[0070] Therefore, it is concluded that the action feature weight is the highest (SA = 0.5), so the action priority fusion path is triggered.

[0071] S140, according to the correlation degree between each modality feature and the context feature, calculating the weight of each modality feature.

[0072] In an embodiment, S140 includes:

[0073] Linearly projecting each modality feature and the visual feature respectively;

[0074] According to the correlation degree between each modality feature and the context feature, respectively calculating the attention weight of each modality feature.

[0075] The correlation degree between each modality feature and the context feature is averaged according to the dimension of each modality feature to obtain the global weight of each modality feature.

[0076] Wherein, the purpose of linear projection is to map the context feature and the modality feature to the same space, so as to facilitate the calculation of the correlation degree.

[0077] For each modality feature, its weight (also called importance score) visual feature weight SV, language feature weight SL, and action feature weight SA are calculated, and the calculation formula is as follows:

[0078] SV = \text{Attn}(F_{ctx},FV)

[0079] SL = \text{Attn}(F_{ctx}, FL)

[0080] SA = \text{Attn}(F_{ctx}, FA)

[0081] where \text{Attn} denotes an attention calculation function.

[0082] In this embodiment, the weight of each modality feature calculated by the cross-attention mechanism can realize the weight distribution based on the correlation degree with the scene perception, provide a quantitative basis for the selection of the dynamic routing path, and enable the VLA model to realize adaptive fusion, thereby realizing multi-modal collaborative decision-making.

[0083] S150, determining a target fusion routing path of the multi-modal features according to the weights of the modality features.

[0084] In an embodiment, the target fusion routing path includes a first routing path with visual feature priority, a second routing path with language feature priority, and a third routing path with action feature priority.

[0085] Correspondingly, S150 includes:

[0086] When the first weight of the visual feature is the largest, the first routing path is the main path, and the second routing path and the third routing path are auxiliary paths.

[0087] When the second weight of the language feature is the largest, the second routing path is the main path, and the first routing path and the third routing path are auxiliary paths.

[0088] When the third weight of the action feature is the largest, the third routing path is the main path, and the first routing path and the second routing path are auxiliary paths.

[0089] The selection strategy of the dynamic routing path in this embodiment can refer to the following Table 1:

[0090]

[0091] Table 1

[0092] For example, taking the action feature as the reference, i.e., taking the third routing path as the main path and the first and second routing paths as auxiliary paths, the feature fusion manner is as follows:

[0093] The action feature FA (door lock mutation timing feature) is encoded by the TCN and used as the fusion reference. The visual feature FV is extracted by the Swin Transformer and cross-attention is performed with the FA to correct the position of the shadow (for example, associating a blurred shadow with the position of the door lock). The "back door" keywords extracted by the BERT language feature FL strengthen the regional no-entry attribute.

[0094] For example, dynamically select the visual priority path (SV highest), and a person carrying dangerous goods is captured by a high-definition camera during the day. The fusion strategy is: the visual feature FV (dangerous goods detection box) is used as the main feature, the language instruction (such as "pay attention to suspicious packages") is used to assist classification, and the action sensor (such as no abnormalities) is used to reduce the weight.

[0095] For another example, dynamically select the language priority path (SL highest), and the voice instruction "monitor the southeast corner locker first". The fusion strategy is: the language feature FL ("locker" position encoding) drives the visual camera to turn, and the action sensor (such as cabinet door vibration) is used as a secondary verification.

[0096] S160, fuse the multi-modal features according to the target fusion routing path to obtain a first target feature.

[0097] For example, according to the weight of each modal feature, the fusion routing path of the multi-modal feature is dynamically selected. A plurality of different fusion modules (such as early fusion modules, late fusion modules, and hierarchical fusion modules) are set, when the visual modal weight SV is high, the routing path that preferentially fuses the visual feature is selected; when the language modal importance is prominent, the path that focuses on the language feature processing is selected. In this way, dynamic and adaptive fusion of multi-modal features is realized, and the fused feature F_{fusion} is obtained.

[0098] S170, performing enhancement processing on the first target feature to obtain a second target feature.

[0099] In an embodiment, as shown in Figure 3 S170 includes:

[0100] S1701, capturing the dependency relationship of the visual feature, the language feature, and the action feature in the first target feature in the time sequence.

[0101] S1702, constructing the semantic association relationship and the structural relationship among the visual feature, the language feature, and the action feature.

[0102] S1703, obtaining the second target feature according to the dependency relationship, the semantic association relationship, and the structural relationship.

[0103] In this embodiment, the above step S170 can be implemented by using a multi-modal collaborative inference network. The multi-modal collaborative inference network adopts a structure combining a bidirectional long short-term memory network (Bi-LSTM) and a graph neural network (GNN). The Bi-LSTM is used to capture the dependency relationship of the multi-modal features in the time sequence, and the GNN is used to model the semantic association and structural relationship between different modal features. Specifically, the fused first target feature F fusion is input into the multi-modal collaborative inference network. The multi-modal collaborative inference network adopts a structure combining a bidirectional long short-term memory network (Bi-LSTM) and a graph neural network (GNN). The Bi-LSTM captures the dependency relationship of the visual features, language features and action features in the first target feature in the time sequence. The GNN constructs the semantic association relationship and structural relationship between the visual features, language features and action features.

[0104] In this embodiment, through the multi-layer Bi-LSTM and GNN layers, the depth interaction and enhancement of the multi-modal features are realized, and the enhanced second target feature F enhanced is obtained, so that the problem of insufficient modal interaction can be solved, the depth of interaction between modal features is enhanced, and the robustness of the VLA model under noisy data (such as correcting ambiguous language features by GNN) is also improved.

[0105] S180, performing inference based on the multi-modal task instruction and the second target feature to obtain a target decision instruction corresponding to the multi-modal feature and output the target decision instruction.

[0106] The multi-modal task instruction includes action prediction, language generation, etc.

[0107] In the inference phase, the enhanced second target feature F enhanced is input into a decoder based on Transformer, and inference is performed in combination with the multi-modal task instruction. Through the multi-head self-attention mechanism, the deep semantic information in the feature is mined to generate a preliminary decision result. Then, through a fully connected layer and a corresponding activation function (such as a Softmax function for a classification task and a linear function for a regression task), the final action instruction or language reply is output.

[0108] In this embodiment, by adaptively fusing features through a dynamic routing path, the accuracy of scene recognition can be improved, scene features are introduced, deep scene understanding can be realized, and the rationality of the target decision instruction is improved.

[0109] In an embodiment, as shown in FIG. 18, after S180, the method further includes: Figure 4

[0110] S210, executing the target decision instruction.

[0111] ​S220, acquire an execution result, and record actual scene change data in the process of executing the target decision instruction.

[0112] In the embodiment, by recording the action execution result and the scene change data, and then optimizing the dynamic routing and the reasoning network parameter by taking the feedback as a new sample, the problems of insufficient generalization ability and non-closed loop feedback can be solved, and the performance of the VLA model can be continuously optimized with the increase of the use scenarios.

[0113] For example, in the financial field, when high-frequency transaction risk control, intelligent investment research report generation and the like need to be performed by the intelligent security task processing method in the embodiment, the aforementioned visual features can be replaced by market data such as stock K-line charts and transaction volume heat maps (image / sequential data); the language features can be replaced by customer instructions such as natural language instructions (such as “find low-volatility technology stocks”); and the action features can be replaced by risk signals such as real-time risk indicators (such as volatility indexes and leverage rate changes). Correspondingly, the scene features can be replaced by macroeconomic indicators (interest rates and inflation), customer historical behaviors and market sentiment, so that subsequent processes are performed based on these data, the flexible balance of “market trends, customer needs and risk control” is achieved, and single signal misjudgment is avoided.

[0114] For example, in the medical field, when intelligent diagnosis needs to be implemented by the intelligent security task processing method in the embodiment, the aforementioned visual features can be replaced by medical images such as CT scans, X-ray films and pathological section images; the language features can be replaced by electronic medical record texts such as patient complaints, medical history records and doctor notes; and the action features can be replaced by physiological signals such as electrocardiograms (ECG), electroencephalograms (EEG) and blood oxygen saturation time series data. The scene features can be replaced by patient age / sex, epidemiological data (such as flu season) and hospital resource status (such as ICU bed shortage), so that subsequent processes are performed based on these data, and the real-time decision of a surgical robot, the prediction of an epidemic, the personalized drug recommendation and the reliability of diagnosis basis are balanced.

[0115] In an embodiment, after S140 and before S150, the method further includes:

[0116] The light condition is acquired, and the first weight of the visual feature is adjusted according to the light condition. For example, in low light, the weight of the visual feature is reduced to avoid false positives.

[0117] The intelligent security task processing method disclosed in the application can improve the accuracy of scene recognition by adaptively fusing features through a dynamic routing path, can realize deep scene understanding by introducing scene features, and thus can improve the rationality of the target decision instruction, and is suitable for intelligent security task processing scenarios in fields such as finance, medicine and public transportation.

[0118] The embodiment of the present application also provides an intelligent security task processing device 100, which is used for executing any one of the foregoing embodiments of the intelligent security task processing method. Specifically, refer to Figure 5 , Figure 5 The embodiment of the present application provides a schematic block diagram of the intelligent security task processing device, the intelligent security task processing device 100 provided by the embodiment of the present application, which comprises an input and output unit 110, a processing unit 120 and a display unit 130.

[0119] The input and output unit 110 is used for acquiring a first video stream to be analyzed;

[0120] The processing unit 120 is used for pre-processing the first video stream acquired by the input and output unit 110, acquiring a multi-modal feature of a target object from the first video stream, and the multi-modal feature comprises a visual feature, a language feature and a motion feature;

[0121] The processing unit 120 is also used for extracting a scene feature associated with a multi-modal task from the multi-modal feature, acquiring an association degree between each modal feature and the scene feature based on a multi-head self-attention mechanism, calculating a weight of each modal feature according to the association degree between each modal feature and the scene feature, determining a target fusion routing path of the multi-modal feature according to the weight of each modal feature, fusing the multi-modal feature according to the target fusion routing path to obtain a first target feature, performing enhancement processing on the first target feature to obtain a second target feature, performing inference based on a multi-modal task instruction and the second target feature to obtain a target decision instruction corresponding to the multi-modal feature, and outputting and displaying the target decision instruction by the display unit 130 through the input and output unit 110.

[0122] In an embodiment, the target fusion routing path comprises a first routing path with visual feature priority, a second routing path with language feature priority and a third routing path with motion feature priority; and the processing unit 120 is also used for:

[0123] When the first weight of the visual feature is the largest, the first routing path is used as a main path, and the second routing path and the third routing path are used as auxiliary paths;

[0124] When the second weight of the language feature is the largest, the second routing path is used as a main path, and the first routing path and the third routing path are used as auxiliary paths;

[0125] When the third weight of the motion feature is the largest, the third routing path is used as a main path, and the first routing path and the second routing path are used as auxiliary paths.

[0126] In an embodiment, the processing unit 120 is specifically used for:

[0127] linearly project each modality feature to the visual feature respectively;

[0128] calculate an attention weight of each modality feature according to a correlation degree between the modality feature and the scene feature;

[0129] average the correlation degree between each modality feature and the scene feature according to a dimension of each modality feature to obtain a global weight of each modality feature;

[0130] normalize the global weights of all modality features.

[0131] In an embodiment, the processing unit 120 is specifically configured to:

[0132] extract a first scene element from the visual feature, the first scene element including a scene type, an ambient light condition, and an object layout; extract a task keyword in the multi-modal task instruction from the language feature; and extract an action occurrence position and an action participating subject from the action feature;

[0133] obtain a supplementary element, the supplementary element being from external sensor data and a preset knowledge base;

[0134] fuse the supplementary element, the first scene element, the task keyword, the action occurrence position, and the action participating subject and perform vectorization coding to obtain the scene feature.

[0135] In an embodiment, the processing unit 120 is further configured to:

[0136] obtain a light condition;

[0137] adjust a first weight of the visual feature according to the light condition.

[0138] In an embodiment, the processing unit 120 is specifically configured to:

[0139] capture a dependency relationship of the visual feature, the language feature, and the action feature in a time sequence in the first target feature;

[0140] construct a semantic correlation relationship and a structural relationship among the visual feature, the language feature, and the action feature;

[0141] obtain the second target feature according to the dependency relationship, the semantic correlation relationship, and the structural relationship.

[0142] In an embodiment, the processing unit 120 is further configured to:

[0143] execute the target decision instruction;

[0144] Obtain the execution results and record the actual scenario change data during the execution of the target decision instruction.

[0145] For example, in the financial sector, when the intelligent security task processing device described in this invention is needed for high-frequency trading risk control, intelligent investment research report generation, etc., the aforementioned visual features can be replaced with market data, such as stock candlestick charts and trading volume heatmaps (images / time-series data); language features can be replaced with customer instructions, such as natural language instructions (e.g., "find low-volatility technology stocks"); and action features can be replaced with risk signals, such as real-time risk indicators (e.g., volatility index, leverage ratio changes). Correspondingly, scenario features can be replaced with macroeconomic indicators (interest rates, inflation), customer historical behavior, and market sentiment, thereby enabling subsequent processes based on this data to achieve a flexible balance between "market trends, customer needs, and risk control," avoiding misjudgment based on a single signal.

[0146] For example, in the medical field, when intelligent diagnosis is required using the intelligent security task processing device in this embodiment of the invention, the aforementioned visual features can be replaced with medical images, such as CT scans, X-rays, and pathological slide images; the aforementioned linguistic features can be replaced with electronic medical record text, such as patient complaints, medical history records, and doctor's notes; the aforementioned action features can be replaced with physiological signals, such as electrocardiograms (ECG), electroencephalograms (EEG), and time-series data of blood oxygen saturation. Contextual features can be replaced with patient age / gender, epidemiological data (such as flu season), and hospital resource status (such as ICU bed shortages), thereby enabling subsequent processes based on this data to achieve real-time decision-making by the surgical robot, epidemic prediction, personalized medication recommendations, and a balance between the reliability of diagnostic evidence.

[0147] The intelligent security task processing device disclosed in this invention is used to execute any embodiment of the aforementioned intelligent security task processing method. Through multimodal feature collaboration, it can achieve deep contextual understanding and improve the rationality of target decision instructions.

[0148] The aforementioned intelligent security task processing method can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the computer device shown.

[0149] Please see Figure 6 , Figure 6 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a device bus 501, wherein the memory may include a storage medium 503 and internal memory 504.

[0150] The storage medium 503 can store an operating device 5031 and a computer program 5032. The computer program 5032, when executed, can cause the processor 502 to perform the intelligent security task processing method.

[0151] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0152] The memory 504 provides an environment for the execution of the computer program 5032 in the storage medium 503. The computer program 5032, when executed by the processor 502, can cause the processor 502 to perform the intelligent security task processing method.

[0153] The network interface 505 is configured to perform network communication, such as providing transmission of data information, etc. Those skilled in the art can understand that the network interface 505 can be configured to perform network communication by using a wired or wireless network. Figure 6 The structure shown in FIG. 5 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. Specifically, the computer device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the intelligent security task processing method disclosed in the embodiments of the present application.

[0155] Those skilled in the art can understand that the computer device shown in FIG. 5 is only a specific implementation of the computer device, and the application of the scheme of the present application is not limited to the computer device shown in FIG. 5. Figure 6 The embodiments of the computer device shown in FIG. 5 do not constitute a limitation on the specific structure of the computer device. In other embodiments, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device can only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are consistent with those of the memory 504 and the processor 502 shown in the embodiments, and will not be described here. ​

[0156] ​It should be understood that, in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0157] In another embodiment of the present application, a computer readable storage medium is provided. The computer readable storage medium can be a non-volatile computer readable storage medium or a volatile computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the intelligent security task processing method disclosed in the embodiments of the present application.

[0158] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein. Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in a general manner in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0159] In several embodiments provided by the present application, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic, for example, the division of units is merely logical function division, and actual implementation can have another division manner, or units with the same function can be combined into one unit, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, systems or units, and can also be electrical, mechanical or other form of connection.

[0160] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0161] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0162] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the present application, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk, and various program code storage media.

[0163] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A smart security task processing method, characterized in that, The method comprises the following steps: preprocessing a first video stream to be analyzed to obtain multi-modal features of a target object from the first video stream, the multi-modal features comprising visual features, language features and action features; extracting scene features associated with a multi-modal task from the multi-modal features; obtaining correlation degrees between each modal feature and the scene features based on a multi-head self-attention mechanism; calculating weights of each modal feature according to the correlation degrees between each modal feature and the scene features; determining a target fusion routing path of the multi-modal features according to the weights of each modal feature; fusing the multi-modal features according to the target fusion routing path to obtain first target features; performing enhancement processing on the first target features to obtain second target features; performing inference based on a multi-modal task instruction and the second target features to obtain a target decision instruction corresponding to the multi-modal features and outputting the target decision instruction. 2.The intelligent security task processing method of claim 1, wherein, The target fusion routing path comprises a first routing path with visual features as priority, a second routing path with language features as priority and a third routing path with action features as priority. When the first weight of the visual features is the largest, the first routing path is used as a main path, and the second routing path and the third routing path are used as auxiliary paths. When the second weight of the language features is the largest, the second routing path is used as a main path, and the first routing path and the third routing path are used as auxiliary paths. When the third weight of the action features is the largest, the third routing path is used as a main path, and the first routing path and the second routing path are used as auxiliary paths. The method further comprises the following steps: 3.The intelligent security task processing method of claim 2, wherein, performing linear projection of each modal feature on the visual features respectively; calculating attention weights of each modal feature according to the correlation degrees between each modal feature and the scene features respectively; averaging the correlation degrees between each modal feature and the scene features according to the dimensions of each modal feature to obtain global weights of each modal feature; normalizing the global weights of all modal features. The method further comprises the following steps:

4. The intelligent security task processing method of claim 3, wherein, extracting a first scene element from the visual features, the first scene element comprising a scene type, an ambient light condition and an object layout; extracting a task keyword in a multi-modal task instruction from the language features; and extracting an action occurrence position and an action participant from the action features; obtaining supplementary elements from external sensor data and a preset knowledge base; fusing the supplementary elements, the first scene element, the task keyword, the action occurrence position and the action participant and performing vectorization coding to obtain the scene features.

5. The intelligent security task processing method of claim 3, wherein, The method further comprises the following steps: obtaining a light condition; adjusting the first weight of the visual features according to the light condition. 6.The intelligent security task processing method of any one of claims 1-5, wherein, The first target feature is enhanced to obtain a second target feature, including: capturing the dependency relationship of visual features, language features and action features in the first target feature in a time sequence; constructing semantic association relationships and structural relationships among the visual features, language features and action features; obtaining the second target feature according to the dependency relationship, the semantic association relationship and the structural relationship.

7. The intelligent security task processing method according to any one of claims 1-6, characterized in that, After the target decision instruction corresponding to the multi-modal feature is obtained and output, it further includes: executing the target decision instruction; obtaining the execution result and recording the actual scene change data in the execution process of the target decision instruction.

8. An intelligent security task processing apparatus, characterized by comprising: including: an input and output unit configured to obtain a first video stream to be analyzed; a processing unit configured to preprocess the first video stream obtained by the input and output unit, and obtain multi-modal features of a target object from the first video stream, the multi-modal features including visual features, language features and action features; the processing unit is further configured to extract scene features associated with a multi-modal task from the multi-modal features, obtain an association degree between each modal feature and the scene features based on a multi-head self-attention mechanism, calculate a weight of each modal feature according to the association degree between each modal feature and the scene features, determine a target fusion routing path of the multi-modal features according to the weight of each modal feature, and fuse the multi-modal features according to the target fusion routing path to obtain a first target feature; enhancing the first target feature to obtain a second target feature; and performing inference based on a multi-modal task instruction and the second target feature to obtain a target decision instruction corresponding to the multi-modal feature and output the target decision instruction through the input and output unit.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the intelligent security task processing method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program causes the processor to execute the intelligent security task processing method of any one of claims 1-7 when the processor executes the computer program.

Images