Image review method fusing semantic comprehension and visual identification

Abstract

The invention belongs to the technical field of machine room safety monitoring, and discloses an image review method fusing semantic understanding and visual recognition, which comprises the following steps: calculating visual / semantic feature dynamic credibility in real time through an exponential weighted moving average algorithm in combination with environment interference and equipment state parameters; resNet50 is adopted to extract visual features in global and local branches, and a BERT model is adopted to encode semantic features; correcting the feature correlation degree according to the scene, calculating a dynamic weight, carrying out heterogeneous calibration through an attention mechanism, and calling priority rules such as'physical security features are higher than behavior features' for arbitration during conflicts; the method is advantaged in that low-credibility feature interference fusion is avoided, abnormity identification accuracy in a complex scene of a machine room is improved, multi-modal data cooperation demands are adapted, scene labels are marked based on time, work orders and historical data dimensions, an exclusive sub-model is constructed for a high-density scene through DBSCAN density clustering, and low-frequency scene parameters are migrated to a similar model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine room safety monitoring, and specifically relates to an image review method fusing semantic understanding and visual recognition. BACKGROUND

[0002] As a data storage and computing core hub, the machine room is rapidly evolving towards high-density deployment, unmanned operation and multi-tenant sharing with the development of digitalization. Physical safety monitoring has become a key link to protect equipment and data security. Image review, as a core technical means, needs to integrate visual and semantic information to meet the safety monitoring needs in complex scenarios. In the current physical safety monitoring of the machine room, image review relies on the cooperation of visual and semantic multi-modal data, but the existing technical solutions have the following technical problems: The cross-modal features have essential heterogeneity, leading to fusion distortion. Visual data focuses on pixel-level spatial features, and semantic data focuses on text-level logical features. Existing technologies mostly use simple feature splicing or fixed weight weighting, without considering the dynamic reliability difference of features. For example, the visual recognition rate decreases from 95% to 60% in a strong light environment, which is still equally weighted with high-confidence semantic features. False alarms in access control failure in semantic logs directly participate in fusion, ultimately leading to a long-term low level of abnormal recognition accuracy.

[0003] Scene dynamics cause feature correlation failure. The machine room has different scenarios such as maintenance period, business peak period, and midnight quiet period. The existing global unified model lacks scene adaptation capability. Unauthorized access + unfamiliar faces in the maintenance period are easily misjudged as illegal intrusion, and there are no semantic alarms but visual shadows in the midnight period, which are prone to missed judgments. Due to the lack of dynamic adjustment mechanism for scene and feature correlation, the core correlation logic fails with the change of scene.

[0004] The lack of evidence chain closure leads to insufficient traceability. Machine room safety review needs to form a complete evidence chain of abnormal triggering, visual evidence and semantic tracing. The existing scheme can only output abnormal results, and cannot associate key tracing information. For example, when a server displacement alarm occurs, the snapshot before displacement, corresponding access records and maintenance work orders cannot be retrieved, making it difficult to distinguish between equipment failure and human movement, and unable to locate the responsible person, which does not meet the core demands of safety supervision traceability, falsifiability and auditability. SUMMARY

[0005] The purpose of the present application is to provide an image review method fusing semantic understanding and visual recognition to solve the problems raised in the background.

[0006] In order to achieve the above purpose, the present application provides the following technical scheme: an image review method fusing semantic understanding and visual recognition, the specific steps of which are as follows: Step S1: Collecting color and infrared dual-mode images and visual reliability parameters, synchronously collecting device semantic log and semantic reliability parameters, and collecting device position data; Step S2: Image space alignment is performed when pre-processing visual data, and visual feature dynamic reliability is calculated; a machine room safety professional dictionary is established when pre-processing semantic data, key information is extracted and converted into a structured feature table, and semantic feature dynamic reliability is calculated; scene labels are labeled for data sets, and a conflict priority rule library is established; Step S3: Feature vector extraction is performed on visual data through global and local feature branches, and the visual dynamic reliability is converted into a reliability vector; basic feature extraction is performed on semantic data, and the semantic dynamic reliability and log source weight are converted into a reliability vector; Step S4: The feature correlation degree is corrected according to the scene type, the effective coefficients of visual features and semantic features are calculated, and dynamic weights are obtained; finally, visual features and semantic features are calibrated, and a scene and reliability calibration fusion feature vector is finally generated; Step S5: Sub-models are established according to scene types, corresponding data sets are trained to ensure the accuracy of abnormality recognition; feature vectors are input, and abnormal information is output, triggering the abnormality review process; Step S6: For the abnormality review process, three-dimensional correlation evidence is generated from time, space, and logic, and an abnormality traceability report is generated; then, a three-level linkage response is triggered, and the evidence chain is uploaded to the management platform; Step S7: A feature reliability feedback library, a scene rule feedback library, and an evidence chain feedback library are established; feature fusion level, scene rule level, and evidence chain level are iteratively optimized based on feedback library data.

[0007] Preferably, the specific steps of step S1 are as follows: A 4K infrared camera deployed in a machine room collects color and infrared dual-mode images, labels camera ID and monitoring area, and synchronously collects visual reliability parameters including image clarity and face detection confidence; real-time captures access control system log, device vibration sensor log, security terminal alarm text, and device state semantic log, converts these data into JSON format and associates with device ID, and synchronously collects semantic reliability parameters including log source reliability and data generation timeliness; In addition, real-time position data of the device is collected through a GPS module built in the cabinet, and the visual data, semantic data, GPS position data, and reliability parameters are integrated into a unified data set in combination with time stamp, spatial position, and device ID three-dimensional alignment rules.

[0008] Preferably, the specific steps of step S2 are as follows: Based on the unified data set of step S1, cross-modal data preprocessing is performed. In visual data processing, after removing noise by Gaussian filtering, image space alignment is realized based on fixed markers in the machine room. The face region is detected and cropped by MTCNN. Then, the exponential weighted moving average algorithm is used to calculate the dynamic reliability of visual features in combination with environmental interference parameters and equipment state parameters. In semantic data processing, a machine room safety professional dictionary containing keywords is established. The event type, location and time key information are extracted by jieba segmentation and BiLSTM entity recognition, and converted into a structured feature table. The dynamic reliability of semantic features is calculated by the EWMA algorithm combined with data aging and log entry method. Based on the time dimension, work order dimension and historical data dimension, scene labels are labeled for the data set. A mapping table of scenes and data is established, and a multi-modal conflict priority rule library is established. Priority 1 is that physical security features are higher than behavior features, and behavior features are higher than environmental features. Priority 2 is that real-time collected data is higher than historical data.

[0009] Preferably, the specific steps of step S3 are as follows: Based on the preprocessed visual data and semantic data of step 2, ResNet50 model is used to process visual data in two branches. The global feature branch extracts the overall scene features of the image including cabinet arrangement and personnel posture, and outputs a global feature vector. The local feature branch extracts a face feature vector from the face region. The two outputs are combined to obtain a visual basic feature vector. The visual feature dynamic reliability and image clarity are converted into a reliability vector, which is combined with the visual basic feature vector to generate a visual and reliability fusion feature vector. For semantic data, the event type, location and time text sequence in the structured feature table are encoded by the fine-tuned BERT model, and a semantic basic feature vector is output. Then, the semantic feature dynamic reliability and log source weight are converted into a reliability vector, which is combined with the semantic basic feature vector to generate a semantic and reliability fusion feature vector.

[0010] Preferably, the specific steps of step S4 are as follows: The scene labels labeled in step S2 are called, and the cross-modal features of scene perception are weighted and fused in combination with the fusion feature vector of step S3. First, the feature correlation degree is corrected according to the scene type, and the feature correlation degree calculated by the original cosine similarity is corrected to 0.6. In the maintenance period scene, the rules of increasing the weight of semantic work order information by 25% and the weight of visual personnel authorization verification by 20% are executed. The correlation degree calculation result is also corrected. Then, the effective coefficients of the two types of features are calculated respectively, wherein the visual effective coefficient is equal to the visual dynamic credibility multiplied by the scene visual weight, and the semantic effective coefficient is equal to the semantic dynamic credibility multiplied by the scene semantic weight. After normalization processing of the effective coefficients, the dynamic weights are obtained; Finally, the visual and semantic features are calibrated through an attention mechanism, the common dimension of time and position matching is highlighted, if a modal conflict occurs, the multi-modal conflict priority rule library is called for arbitration, high-priority features are preferentially adopted, the calibrated features are weighted and summed according to the dynamic weights, and a scene and reliability calibrated fusion feature vector is generated.

[0011] Preferably, the specific steps of the step S5 are as follows: Based on the scene and reliability calibrated fusion feature vector of the step S4, in combination with the scene label of the step S2, the scene label, the abnormal type and the feature contribution degree in the historical data are clustered by using a DBSCAN density clustering algorithm to identify the scene type. For a scene with a clustering density not lower than 500 samples per class, a dedicated sub-model is constructed. For a low-frequency scene with a clustering density lower than 100 samples per class, the parameters thereof are migrated to a similar scene sub-model. Each sub-model is trained by using a labeled data set corresponding to the scene, takes the fusion feature vector and the scene label as input, and optimizes the hidden layer parameters through a cross-entropy loss function, so as to ensure the abnormal recognition accuracy of the sub-model under the corresponding scene. In the real-time inference stage, the corresponding sub-model is called according to the scene label of the step S2. After the fusion feature vector generated in the step S4 is input into the sub-model, the abnormal type, the abnormal confidence and the feature contribution degree are output. When the abnormal confidence is not lower than 0.9, the abnormal process under review is triggered, and the trajectory integrity verification is performed at the same time. If there is data missing, the adjacent cabinet camera and GPS position data are called for supplementation.

[0012] Preferably, the specific steps of the step S6 are as follows: For the abnormality under review of the step S5, in combination with the original data of the step S1 and the preprocessing result of the step S2; In the time dimension, the continuous visual frames and the synchronous semantic logs 30 seconds before and after the abnormality are called, a trajectory time sequence diagram is generated, and the feature dynamic credibility decay curve of each time node is labeled; In the spatial dimension, the device ID of the abnormal area is associated, the historical position snapshot and the door lock record of the device 24 hours before the abnormality are called, and a cabinet position change heat map is generated; In the logic dimension, the computer room safety knowledge graph is called to query information such as the tenant to which the abnormal device belongs, the recent maintenance record, the relocation application and the like to complete the logical link; An abnormality traceability report is generated, which contains an abnormality timeline, multi-view image evidence, semantic traceability record and device state change; and then a three-level linkage response is triggered, including a machine room sound and light alarm, pushing of alarm information and abnormal image evidence to a security terminal, locking of an abnormal area access control, and synchronous uploading of a complete evidence chain to a machine room security management platform.

[0013] Preferably, the specific steps of the step S7 are as follows: Collecting various feedback information of the step S6, establishing a three-dimensional feedback library, wherein a feature reliability feedback library collects cases of misjudgment caused by low-credibility features, a scene rule feedback library collects cases of scene adaptation deviation, and an evidence chain feedback library collects cases of traceability failure caused by missing evidence; Then, multi-dimensional iterative optimization is performed based on the feedback library data every month, the EWMA algorithm parameters are adjusted at the feature fusion level, and the feature weight threshold is set; new scenes are mined through unsupervised learning at the scene rule level, the feature association rules of the scenes are automatically generated and supplemented to the scene and feature association rule library; and the data acquisition dimension is supplemented according to the feedback at the evidence chain level.

[0014] The beneficial effects of the present application are as follows: 1、The present application uses an exponential weighted moving average algorithm to combine environmental interference and device state parameters to calculate the dynamic credibility of visual / semantic features in real time; ResNet50 is used to extract visual features in global and local branches, and a BERT model is used to encode semantic features; then the feature correlation is corrected according to the scene, the dynamic weight is calculated, and the heterogeneous calibration is performed through an attention mechanism; when there is a conflict, priority rules such as "physical security features are higher than behavior features" are called to arbitrate; low-credibility features are avoided to interfere with fusion, and the accuracy of abnormality recognition in complex scenes of a machine room is improved, and the collaborative needs of multi-modal data are adapted.

[0015] 2、The present application labels scene tags based on time, work order and historical data dimensions, uses DBSCAN density clustering to construct a dedicated sub-model for high-density scenes, and migrates low-frequency scene parameters to similar models; during training, the parameters are optimized using the corresponding scene data set, such as adjusting the number of neurons in the personnel authorization matching layer during peak hours, calling the matching sub-model in real time, and executing scene-based weight rules, such as increasing the weight of semantic work orders by 25% during maintenance periods and increasing the weight of visual authorization verification by 20%, so that the model accurately adapts to different scenes and effectively reduces the misjudgment and omission rates.

[0016] 3, The application aims at the abnormality to be examined, calls 30-second visual frames before and after the abnormality from the time dimension, generates a time sequence diagram and a reliability attenuation curve, associates the equipment ID from the space dimension, calls 24-hour historical position snapshots and door lock records, generates a position change heat map, calls the machine room safety knowledge graph from the logic dimension, supplements information such as the tenant to which the equipment belongs, maintenance records, relocation applications, and finally generates a traceability report containing an abnormal time line and multi-perspective evidence, triggers a three-level response of machine room sound and light alarm, security terminal evidence push and access control locking, and uploads a complete evidence chain to the management platform, can locate the person in charge, and fills the blank of machine room safety review traceability capability. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The image review method flowchart of the application fuses semantic understanding and visual recognition. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0019] As shown in Figure 1 The application embodiment provides an image review method fusing semantic understanding and visual recognition, and the specific steps of the method are as follows: Step S1: color and infrared dual-mode images and visual reliability parameters are collected, device semantic logs and semantic reliability parameters are collected synchronously, and device position data is collected; Step S2: image space alignment is performed when visual data is preprocessed, and visual feature dynamic reliability is calculated; a machine room safety professional dictionary is established when semantic data is preprocessed, key information is extracted and converted into a structured feature table, and semantic feature dynamic reliability is calculated; scene labels are labeled for data sets, and a conflict priority rule library is established; Step S3: feature vector extraction is performed on visual data through global and local feature branches, and the visual dynamic reliability is converted into a reliability vector; basic feature extraction is performed on semantic data, and the semantic dynamic reliability and log source weight are converted into a reliability vector; Step S4: the feature correlation degree is corrected according to the scene type, the effective coefficients of visual features and semantic features are calculated, and dynamic weights are obtained; finally, heterogeneous calibration is performed on visual features and semantic features, and finally a scene and reliability calibration fusion feature vector is generated; Step S5: According to the scene type, a sub-model is established, a corresponding data set is trained to ensure the accuracy of anomaly recognition; the feature vector is input, and the abnormal information is output to trigger the abnormal process to be reviewed; Step S6: For the abnormal process to be reviewed, an abnormality traceability report is generated from the time, space, and logical three-dimensional correlation evidence, and then a three-level linkage response is triggered to upload the evidence chain to the management platform; Step S7: Establish a feature reliability feedback library, a scene rule feedback library, and an evidence chain feedback library; based on the feedback library data, iterative optimization is performed at the feature fusion level, the scene rule level, and the evidence chain level.

[0020] The specific steps of step S1 are as follows: A 4K infrared camera deployed in the machine room collects color and infrared dual-mode images, labels the camera ID and the monitoring area, and synchronously collects visual reliability parameters such as image clarity and face detection confidence; real-time capture of access control system logs, device vibration sensor logs, security terminal alarm text, and device status semantic logs, including server indicator light status, cabinet door lock switch records, and maintenance work order reservation information, convert these data into JSON format and associate with device ID, and collect semantic reliability parameters such as log source credibility and data generation timeliness; In addition, the real-time position data of the device is collected through the GPS module built-in the cabinet, and the visual data, semantic data, GPS position data, and reliability parameters are integrated into a unified data set based on the three-dimensional alignment rules of time stamp, spatial position, and device ID.

[0021] The specific steps of step S2 are as follows: Based on the unified data set of step S1, the cross-modal data is preprocessed. In the visual data processing, after removing noise by Gaussian filtering, the image space is aligned based on the fixed markers in the machine room, the face area is detected and cropped by MTCNN, and the cabinet or device area is retained when there is no face. The image size is uniformly adjusted to 224x224 pixels, and then the exponential weighted moving average algorithm is used to calculate the dynamic reliability of visual features combined with environmental interference parameters (such as strong light intensity, dust concentration), and device state parameters (camera occlusion rate, sensor fault code). The image with a dynamic reliability lower than 0.6 is removed, and the image with a face detection confidence lower than 0.7 is labeled as low reliability visual features; In semantic data processing, a professional dictionary of machine room safety containing keywords such as unauthorized personnel, illegal movement, and access control triggering is established. Jieba segmentation and BiLSTM entity recognition are used to extract event type, location, and time key information, and convert them into a structured feature table. The EWMA algorithm is used to calculate the dynamic credibility of semantic features combined with data time effectiveness and log entry mode. The credibility is marked as high, medium, and low when it is not less than 0.9, between 0.7 and 0.9, and less than 0.7, respectively. Based on time dimension, work order dimension, and historical data dimension, scene labels are marked for data sets. A mapping table of scenes and data is established, and a multi-modal conflict priority rule library is established to clearly define that priority 1 is physical security features (device vibration, door lock status) higher than behavior features (person posture), and behavior features higher than environment features (light). Priority 2 is real-time data higher than historical data.

[0022] EWMA dynamic credibility calculation formula: ; In the formula: represents the dynamic credibility of the feature at time t, with a value range of [0, 1], which is used to quantify the reliability of visual features (such as face detection credibility, image clarity) or semantic features (such as log credibility) at the current time. For example, in the late night quiet period, dust causes the visual feature =0.7; represents the credibility decay coefficient, with a value range of [0.1, 0.95], which is dynamically adjusted by environmental interference (such as strong light, dust) or device status (such as camera obstruction, sensor failure). In a dusty environment, from 0.95 to 0.9, The larger the value, the more significant the impact of the current data quality on credibility. represents the original data quality score at the current time, with a value range of [0, 1], which is directly output by the data acquisition device or calculated after preprocessing. For example, the original clarity score of the image captured by the camera =0.8, and the original credibility score of the access log =0.98; represents the dynamic credibility of the feature at time t-1, with a value range of [0, 1], which reflects the time continuity of credibility. For example, within 10 minutes after the log is generated, =0.98, and after 1 hour due to time decay =0.8; t represents the time stamp, with a unit of seconds, ensuring that the credibility calculation is synchronized with the data acquisition time.

[0023] The specific steps of step S3 are as follows: Based on the preprocessed visual data and semantic data in step 2, single-modal feature vector extraction is carried out respectively: ResNet50 model is used to process visual data in two branches, wherein the global feature branch extracts the overall scene features including cabinet arrangement and personnel posture, and outputs a 2048-dimensional global feature vector; the local feature branch extracts a 128-dimensional face feature vector for the face region, and outputs a no-face identification vector when there is no face; the outputs of the two branches are combined to obtain a 2176-dimensional visual basic feature vector; then the visual feature dynamic reliability (for example, 0.85 calculated by EWMA) and the image clarity (for example, 0.78) are converted into a 2-dimensional reliability vector, which is combined with the visual basic feature vector to generate a 2178-dimensional visual and reliability fusion feature vector; For semantic data, the event type, location and time text sequence in the structured feature table are encoded by using the fine-tuned BERT model, and a 2176-dimensional semantic basic feature vector is output; then the semantic feature dynamic reliability (for example, 0.92 calculated by EWMA) and the log source weight (the access control log weight is set to 1.0, and the manual input weight is set to 0.85) are converted into a 2-dimensional reliability vector, which is combined with the semantic basic feature vector to generate a 2178-dimensional semantic and reliability fusion feature vector, so as to ensure that the dimensions of the visual and semantic feature vectors are kept uniform.

[0024] The specific steps of step S4 are as follows: The scene label labeled in step S2 and the scene and feature association rule base are called, and the cross-modal features of scene perception are weighted and fused in combination with the fusion feature vector in step S3. First, the feature association degree is corrected according to the scene type, for example, in the deep night silent period scene, the visual feature association weight is increased by 30%, and the rule of focusing on checking the vision when there is no alarm in the semantic is executed, and the feature association degree calculated by the original cosine similarity is corrected to 0.6; in the maintenance period scene, the semantic work order information weight is increased by 25%, and the visual personnel authorization verification weight is increased by 20%, and the calculation result of the association degree is also corrected; Then, the effective coefficients of the two types of features are calculated respectively, wherein the visual effective coefficient is equal to the visual dynamic reliability multiplied by the scene visual weight, and the semantic effective coefficient is equal to the semantic dynamic reliability multiplied by the scene semantic weight. Taking the “deep night silent period” scene as an example, the scene visual weight is set to 1.3, and the semantic weight is set to 0.7. If the visual dynamic reliability is 0.8 and the semantic dynamic reliability is 0.98, the visual effective coefficient is 1.04 and the semantic effective coefficient is 0.686. After normalization processing of the effective coefficient, the dynamic weight is obtained, such as the visual weight 0.6 and the semantic weight 0.4. Finally, the visual and semantic features are calibrated by attention mechanism to highlight the common dimensions of time and location matching. If there is a modal conflict, such as the semantic display authorization maintenance but the visual detects abnormal displacement of the device, the multi-modal conflict priority rule base is called for arbitration, and high-priority features are preferred, such as vibration sensor data credibility 0.95 higher than semantic work order credibility 0.9, vibration data is adopted, the calibrated features are weighted and summed according to the dynamic weight, and a 2178-dimensional scene and reliability calibration fusion feature vector is generated.

[0025] Feature effective coefficient calculation formula: ; ; Visual feature effective coefficient, value range [0, 1.5], quantifies the effective contribution of visual features in the current scene, for example, during the night quiet period =0.8x1.3=1.04; Semantic feature effective coefficient, value range [0, 1.5], quantifies the effective contribution of semantic features in the current scene, for example, during the night quiet period =0.98x0.7=0.686; Visual feature dynamic credibility at time t, value range [0, 1], obtained by the EWMA dynamic credibility calculation formula, corresponding to the reliability of visual data (such as face detection, image clarity); Semantic feature dynamic credibility at time t, value range [0, 1], also obtained by the EWMA dynamic credibility calculation formula, corresponding to the reliability of semantic data (such as access control log, work order information); Visual feature weight under scene s, value range [0.5, 1.5], defined by the scene and feature association rule base, for example, during the night quiet period =1.3, maintenance period =0.8; Semantic feature weight under scene s, value range [0.5, 1.5], also defined by the scene and feature association rule base, for example, during the night quiet period =0.7, maintenance period =1.25; s represents the scene label, time dimension night quiet period, maintenance period, peak period, etc. Scene, marked by the scene and data mapping table in step S2.

[0026] Feature dynamic weight calculation formula: ; ; wherein represents the dynamic weight of visual features, the value range is [0, 1], which is used as the weighting coefficient of visual features in cross-modal fusion, for example, in the late-night quiet period ; represents the dynamic weight of semantic features, the value range is [0, 1], which is used as the weighting coefficient of semantic features in cross-modal fusion, for example, in the late-night quiet period ; represents the effective coefficient of visual features, the value range is [0, 1.5], which is obtained by the feature effective coefficient calculation formula; represents the effective coefficient of semantic features, the value range is [0, 1.5], which is obtained by the feature effective coefficient calculation formula, and together determine the weight distribution; The formula constraint is: , which ensures that the dimensions of the weighted sum of the visual and reliability fusion feature vectors and the semantic and credibility fusion feature vectors are unified after cross-modal feature fusion.

[0027] The specific steps of the step S5 are as follows: Based on the scene and reliability calibration fusion feature vector of the step S4, combined with the scene label of the step S2, the DBSCAN density clustering algorithm is used to cluster the scene label, abnormal type and feature contribution degree in the historical data to identify the scene type. For the scenes with a clustering density not less than 500 samples per class (such as maintenance period, peak period, late-night quiet period), a dedicated sub-model is constructed. For low-frequency scenes with a clustering density less than 100 samples per class, such as machine room cleaning period, the parameters are migrated to a similar scene sub-model, such as maintenance period sub-model; Each sub-model is trained using the labeled data set corresponding to the scene, taking the fusion feature vector and the scene label as input, and optimizing the hidden layer parameters through the cross-entropy loss function. For example, when the high-peak period sub-model has a false alarm bias due to dense human flow, the number of neurons in the personnel authorized list matching layer is adjusted to 256, ensuring that each sub-model has high abnormal recognition accuracy in the corresponding scene; In the real-time inference stage, the corresponding sub-model is called according to the scene label of step S2, the fusion feature vector generated in step S4 is input into the sub-model, and the output is an abnormal type, an abnormal confidence and a feature contribution degree, such as a person breaking in, a confidence of 0.95 and a visual feature contribution degree of 0.6. When the abnormal confidence is not lower than 0.9, a to-be-reviewed abnormal process is triggered, and a trajectory integrity check is performed at the same time. If there is data loss, such as no image data from 14:33 to 14:34 due to camera disconnection, adjacent cabinet camera and GPS position data are called to supplement, so as to avoid misjudgment due to data loss.

[0028] The specific steps of step S6 are as follows: For the to-be-reviewed abnormality of step S5, the original data of step S1 and the preprocessing result of step S2 are combined; In the time dimension, the continuous visual frames and the synchronous semantic logs 30 seconds before and after the abnormality are called, a trajectory time sequence diagram is generated, and a feature dynamic confidence decay curve of each time node is labeled, such as a visual confidence of 0.9 at 14:32 and a decrease to 0.7 at 14:33 due to dust influence; In the spatial dimension, the device ID of the abnormal area is associated, the historical position snapshot and the door lock record of the device 24 hours before the abnormality are called, and a cabinet position change heat map is generated; In the logic dimension, the knowledge graph of the machine room safety is called, including entities: personnel, equipment, work order; relationship: authorization, maintenance, ownership, information such as tenant ownership, recent maintenance record and relocation application of the abnormal device is queried to complete the logical link, such as when there is no work order record, the tenant application information is queried, a relocation work order not recorded for 3 days is found, and then the judgment result is corrected to be compliant; An abnormality traceability report is generated, including an abnormality timeline, multi-perspective image evidence, semantic traceability record and device state change, for example, 14:32, the east access control is not authorized (semantic confidence 0.98), 14:33, camera 101 captures a stranger's face (visual confidence 0.92), 14:34, cabinet A1-01 vibration sensor alarm (confidence 0.95), 14:35, the door lock is not opened (device state log), through the knowledge graph query, it is confirmed that there is no maintenance or relocation work order, and finally it is determined to be illegal intrusion; then a three-level linkage response is triggered, including machine room sound and light alarm, alarm information and abnormal image evidence are pushed to the security terminal, the abnormal area access control is locked, and the complete evidence chain is synchronized to the machine room safety management platform.

[0029] The specific steps of step S7 are as follows: Collecting the various types of feedback information of the step S6, establishing a three-dimensional feedback library, wherein the characteristic reliability feedback library collects cases of misjudgment caused by low reliability characteristics, such as misjudgment caused by visual clarity 0.5, records the upper limit of the weight of such characteristics, the scene rule feedback library collects cases of scene adaptation deviation, such as missed judgment during the tenant relocation period, records the characteristics of the scene, and the evidence chain feedback library collects cases of traceability failure caused by missing evidence, such as no vibration data to distinguish between faults and human movement, records the types of sensors that need to be supplemented; Subsequently, multi-dimensional iterative optimization is performed based on the feedback library data every month, EWMA algorithm parameters are adjusted at the feature fusion level, such as adjusting the reliability decay coefficient from 0.95 to 0.9 in a dusty environment, and setting the feature weight threshold, such as setting the upper limit of the weight of visual features with a clarity lower than 0.6 to 0.4; at the scene rule level, new scenes are mined through unsupervised learning, and the feature association rules of the scene are automatically generated, such as lowering the weight of unauthorized access to 0.2, raising the weight of device moving out verification to 0.5, and supplementing to the scene and feature association rule library; at the evidence chain level, data acquisition dimensions are supplemented according to feedback, such as adding cabinet vibration frequency sensors and tenant application information interfaces, and finally through iterative optimization, the system false alarm rate is kept at a low level.

[0030] It should be noted that in this text, relational terms such as first and second are used merely to distinguish one entity or action from another, and do not necessarily require or imply that there is any such actual relationship or order between these entities or actions. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0031] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An image review method that fuses semantic understanding and visual recognition, characterized in that: The specific steps of the method are as follows: Step S1: Collecting color and infrared dual-mode images and visual reliability parameters, synchronously collecting device semantic log and semantic reliability parameters, and collecting device position data; Step S2: Image space alignment is performed during visual data preprocessing, and visual feature dynamic reliability is calculated; During semantic data preprocessing, a machine room safety professional dictionary is established, key information is extracted and converted into a structured feature table, and semantic feature dynamic reliability is calculated; Scene labels are annotated for the data set, and a conflict priority rule library is established; Step S3: Feature vector extraction is performed on visual data through global and local feature branches, and the visual dynamic reliability is converted into a reliability vector; basic feature extraction is performed on semantic data, and the semantic dynamic reliability and log source weight are converted into a reliability vector; Step S4: According to the scene type, the feature correlation degree is corrected, the effective coefficient of visual feature and semantic feature is calculated, and the dynamic weight is obtained; finally, the visual feature and the semantic feature are calibrated, and the scene and reliability calibration fusion feature vector is generated; Step S5: According to the scene type, a sub-model is established, and the corresponding data set is trained to ensure the accuracy of anomaly recognition; Input feature vector, output abnormal information, trigger the abnormal process under review; Step S6: For the abnormal process under review, generate an abnormality traceability report from time, space, and logical three-dimensional related evidence; Then trigger a three-level linkage response, upload the evidence chain to the management platform; Step S7: Establish a feature reliability feedback library, a scene rule feedback library, and an evidence chain feedback library; Based on the feedback library data, iterative optimization is performed at the feature fusion level, scene rule level, and evidence chain level. 2.The image review method of fusing semantic understanding and visual recognition according to claim 1, characterized in that: The specific steps of step S1 are as follows: A 4K infrared camera deployed in the machine room collects color and infrared dual-mode images, labels the camera ID and the monitoring area, and synchronously collects visual reliability parameters including image clarity and face detection confidence; real-time capture of access control system log, device vibration sensor log, security terminal alarm text and device status semantic log, convert these data into JSON format and associate with device ID, and collect semantic reliability parameters including log source reliability and data generation timeliness; In addition, through the GPS module built-in the cabinet, the real-time position data of the device is collected, combined with the timestamp, spatial position and device ID three-dimensional alignment rules, the visual data, semantic data, GPS position data and reliability parameters are integrated into a unified data set. 3.The image review method of fusing semantic understanding and visual recognition according to claim 2, characterized in that: The specific steps of step S2 are as follows: Based on the unified data set of step S1, the cross-modal data is preprocessed, after removing noise by Gaussian filtering, the image space is aligned based on the fixed markers in the machine room, the face area is detected and cropped by MTCNN, and then the exponential weighted moving average algorithm is used to calculate the visual feature dynamic reliability combined with the environmental interference parameters and device state parameters; In semantic data processing, a computer room safety professional dictionary containing keywords is established, event type, location and time key information are extracted by using jieba segmentation and BiLSTM entity recognition, and are converted into a structured feature table, and the dynamic credibility of the semantic features is calculated by combining the data time effectiveness and the log entry mode through the EWMA algorithm; Based on the time dimension, the work order dimension, and the historical data dimension, the scene labels are marked for the data set, a mapping table of scenes and data is established, and a multi-modal conflict priority rule library is established, which clearly defines that the physical safety class features are higher than the behavior class features, and the behavior class features are higher than the environment class features, and the real-time collected data is higher than the historical data.

4. The image review method of claim 3, wherein: The specific steps of the step S3 are as follows: Based on the visual data and semantic data preprocessed in the step 2, the ResNet50 model is used to process the visual data in two branches, wherein the global feature branch extracts the overall scene features of the image including the cabinet arrangement and the personnel posture, and outputs the global feature vector, the local feature branch extracts the face feature vector from the face region, and the two branch outputs are combined to obtain the visual basic feature vector, and then the visual feature dynamic credibility and the image clarity are converted into the reliability vector, and the visual basic feature vector is combined to generate the visual and reliability fusion feature vector; For the semantic data, the event type, location and time text sequence in the structured feature table are encoded by using the fine-tuned BERT model, and the semantic basic feature vector is output, and then the semantic feature dynamic credibility and the log source weight are converted into the credibility vector, and the semantic basic feature vector is combined to generate the semantic and credibility fusion feature vector.

5. The image review method of claim 4, wherein: The specific steps of the step S4 are as follows: The scene labels labeled in the step S2 are called, and the fusion feature vector in the step S3 is combined to perform weighted fusion on the scene-aware cross-modal features, first, the feature correlation degree is corrected according to the scene type, and the feature correlation degree calculated by the original cosine similarity is corrected to 0.6, in the maintenance period scene, the rules of increasing the semantic work order information weight by 25% and the visual personnel authorization verification weight by 20% are executed, and the correlation degree calculation result is also corrected; Then, the effective coefficients of the two types of features are calculated respectively, wherein the visual effective coefficient is equal to the visual dynamic credibility multiplied by the scene visual weight, and the semantic effective coefficient is equal to the semantic dynamic credibility multiplied by the scene semantic weight, and the effective coefficients are normalized to obtain the dynamic weight; Finally, the visual and semantic features are calibrated through the attention mechanism, the common dimension of time and location matching is highlighted, if there is a modal conflict, the multi-modal conflict priority rule library is called for arbitration, the high priority features are preferentially adopted, the calibrated features are weighted and summed according to the dynamic weight, and the scene and reliability calibration fusion feature vector is generated.

6. The image review method of claim 5, wherein: The specific steps of the step S5 are as follows: Based on the scene and reliability calibration fusion feature vector of step S4, combined with the scene label of step S2, the DBSCAN density clustering algorithm is used to cluster the scene label, anomaly type and feature contribution degree in the historical data to identify the scene type. For the scene with a clustering density of not less than 500 samples per class, a dedicated sub-model is constructed. For the low-frequency scene with a clustering density of less than 100 samples per class, the parameters are migrated to the similar scene sub-model; Each sub-model is trained using the labeled data set of the corresponding scene, taking the fusion feature vector and scene label as input, and optimizing the hidden layer parameters through the cross-entropy loss function to ensure the accuracy of the sub-model in identifying anomalies in the corresponding scene; In the real-time inference stage, the corresponding sub-model is called according to the scene label of step S2. After the fusion feature vector generated in step S4 is input into the sub-model, the output is the anomaly type, anomaly confidence and feature contribution degree. When the anomaly confidence is not less than 0.9, the abnormal review process is triggered, and the trajectory integrity check is performed at the same time. If there is data missing, the adjacent cabinet camera and GPS position data are called for supplementation.

7. The image review method of claim 6, wherein: The specific steps of step S6 are as follows: For the abnormal review of step S5, the original data of step S1 and the preprocessing result of step S2 are combined; In the time dimension, the continuous visual frames and synchronous semantic logs 30 seconds before and after the anomaly are called to generate a trajectory timing diagram and label the feature dynamic confidence decay curve of each time node; In the spatial dimension, the device ID of the abnormal area is associated, and the historical position snapshot and door lock record of the device 24 hours before the anomaly are called to generate a cabinet position change heat map; In the logic dimension, the data center security knowledge graph is called to query the tenant to which the abnormal device belongs, the latest maintenance record and the relocation application information to complete the logical link; An abnormal traceability report is generated, which includes the abnormal timeline, multi-perspective image evidence, semantic traceability record and device state change. Then a three-level linkage response is triggered, including data center sound and light alarm, pushing alarm information and abnormal image evidence to security terminal, locking the abnormal area access control, and synchronously uploading the complete evidence chain to the data center security management platform.

8. The image review method of claim 7, wherein: The specific steps of step S7 are as follows: Collect various feedback information of step S6 to establish a three-dimensional feedback library, including a feature reliability feedback library collecting cases of misjudgment caused by low-confidence features, a scene rule feedback library collecting cases of scene adaptation deviation, and an evidence chain feedback library collecting cases of traceability failure caused by missing evidence; Then, based on the feedback library data, multi-dimensional iterative optimization is performed every month. In the feature fusion layer, the EWMA algorithm parameters are adjusted, and the feature weight threshold is set. In the scene rule layer, new scenes are mined through unsupervised learning to automatically generate feature association rules for the scene and supplement them to the scene and feature association rule library. In the evidence chain layer, the data collection dimension is supplemented according to the feedback.

Images