Early warning analysis method based on intelligent vision and server

Abstract

The invention provides an early warning analysis method based on intelligent vision and a server, and the method comprises the steps: obtaining a multi-mode video stream containing a visible light and infrared thermal imaging image sequence through a vision collection device in a monitoring region, and carrying out the spatial-temporal feature extraction of the multi-mode video stream; the method comprises the steps of generating dynamic behavior sequence characteristics including a target object moving track, posture change and thermodynamic distribution characteristics, analyzing the dynamic behavior sequence characteristics through a preset abnormal behavior judgment model in a multi-dimensional matching mode, judging whether the target object triggers abnormal behavior characteristics or not, and if yes, generating a graded early warning signal in combination with an association weight. In addition, according to the spatial distribution and the time length of the abnormal behavior characteristics, the parameters of the acquisition equipment and the coverage range of the monitoring area are dynamically adjusted, and the subsequent video stream anomaly detection precision is optimized, so that intelligent and accurate early warning analysis is realized, and the security efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an intelligent vision-based early warning analysis method and a server. Background Art

[0002] In today's era, the importance of security is becoming increasingly prominent, and various places have a strong demand for efficient and accurate early warning analysis technology. Traditional early warning analysis methods mostly rely on a single sensor or limited monitoring methods, which makes it difficult to comprehensively and accurately monitor and analyze the behavior of target objects in complex environments.

[0003] Related technologies attempt to use multiple sensors for monitoring, but these methods often simply superimpose or process the data from different sensors separately, without achieving true data fusion and collaborative analysis. For example, simply combining visible light cameras and infrared sensors does not fully explore the inherent connections between different modal data when analyzing data, and cannot comprehensively characterize the behavioral characteristics of the target object from multiple dimensions, making it difficult to effectively improve the accuracy and reliability of early warning analysis.

[0004] In terms of abnormal behavior judgment, traditional methods are usually based on simple rule matching or fixed threshold settings, which cannot adapt to complex and changeable actual scenarios. They lack sufficient flexibility and adaptability to the diverse behavior patterns of different target objects in different environments. Once new or complex abnormal behavior patterns are encountered, misjudgment or omission is likely to occur, which cannot meet the requirements for accurate early warning in practical applications. Summary of the invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a warning analysis method based on intelligent vision, the method comprising: Acquire a multimodal video stream of a target scene in real time through a visual acquisition device deployed in a monitoring area, wherein the multimodal video stream includes a visible light image sequence and an infrared thermal imaging image sequence; Extracting spatiotemporal features from the multimodal video stream to generate dynamic behavior sequence features of the target object in the monitoring area, wherein the dynamic behavior sequence features include movement trajectory, posture change, and thermal distribution features of the target object; Perform multi-dimensional matching analysis on the dynamic behavior sequence characteristics based on a preset abnormal behavior determination model to determine whether the target object triggers abnormal behavior characteristics in the monitoring area; If it is detected that the target object triggers the abnormal behavior feature in the monitoring area, a graded warning signal is generated by combining the dynamic behavior sequence feature with the association weight of the abnormal behavior feature, and a corresponding security response mechanism is triggered based on the graded warning signal; According to the spatial distribution and time duration of the abnormal behavior characteristics, the acquisition parameters of the visual acquisition device and the coverage of the monitoring area are dynamically adjusted to optimize the anomaly detection accuracy of subsequent video streams.

[0006] In a possible implementation of the first aspect, extracting spatiotemporal features from the multimodal video stream to generate dynamic behavior sequence features of the target object in the monitoring area includes: Inputting the visible light image sequence into a first convolutional neural network, extracting contour shape features and surface texture features of the target object, and generating a visible light behavior feature vector; Inputting the infrared thermal imaging image sequence into a second convolutional neural network, extracting the thermal radiation intensity distribution characteristics and temperature change gradient characteristics of the target object, and generating a thermal imaging behavior feature vector; Performing cross-modal fusion processing on the visible light behavior feature vector and the thermal imaging behavior feature vector to generate an initial fusion feature matrix; Based on the spatial position change of the target object in the continuous time frame, the initial fusion feature matrix is ​​modeled in time series to generate the dynamic behavior sequence feature; The time series modeling includes a continuity analysis of the movement speed, direction offset and posture angle of the target object in adjacent time frames to construct the spatiotemporal correlation between the movement trajectory and posture changes.

[0007] In a possible implementation of the first aspect, the step of performing time series modeling on the initial fusion feature matrix based on the spatial position change of the target object in the continuous time frame to generate the dynamic behavior sequence feature includes: Extracting the spatial position coordinates of the target object in continuous time frames from the initial fusion feature matrix to generate a spatial position coordinate sequence, and generating an instantaneous velocity vector sequence of the target object based on the position coordinate difference of adjacent time frames in the spatial position coordinate sequence, and generating a direction offset angle sequence based on the velocity vector direction angle difference of two consecutive time frames in the instantaneous velocity vector sequence; Extracting the posture angle parameters of the target object in continuous time frames from the initial fusion feature matrix to generate a posture angle sequence, and performing a differential operation on the posture angle parameters of adjacent time frames in the posture angle sequence to generate a posture change rate sequence; The instantaneous velocity vector sequence, the direction offset angle sequence and the attitude change rate sequence are aligned along the time axis to generate a joint motion feature matrix, and the joint motion feature matrix is ​​subjected to mean statistics and variance statistics using a sliding time window to generate a time window statistical feature matrix, and the time window statistical feature matrix is ​​spliced ​​with the initial fusion feature matrix in the channel dimension to generate an enhanced spatiotemporal feature matrix; Modeling the time dependency of the enhanced spatiotemporal feature matrix through a temporal convolutional network to generate dynamic behavior sequence features containing long-term and short-term motion correlations; The temporal convolutional network comprises a dilated convolutional layer and a causal convolutional layer, which are used to capture the multi-scale motion pattern of the target object in a continuous time frame; The size of the sliding time window is dynamically adjusted according to the historical movement frequency of the target object, so that a smaller window size is used during high-frequency movement and a larger window size is used during low-frequency movement; The channel dimension stitching includes performing feature channel expansion on the statistical indicators of the time window statistical feature matrix and the cross-modal features of the initial fusion feature matrix to form fusion data including motion statistical characteristics and original perceptual characteristics; The cross-modal fusion processing of the visible light behavior feature vector and the thermal imaging behavior feature vector to generate an initial fusion feature matrix includes: Mapping the visible light behavior feature vector and the thermal imaging behavior feature vector to a high-dimensional feature space respectively to generate a first high-dimensional feature vector and a second high-dimensional feature vector; Calculating a cross-modal weight matrix between the first high-dimensional feature vector and the second high-dimensional feature vector through an attention mechanism; Performing a weighted summation of the first high-dimensional feature vector and the second high-dimensional feature vector based on the cross-modal weight matrix to generate an intermediate fused feature vector; Inputting the intermediate fused feature vector into the fully connected layer for dimensionality reduction processing to generate the initial fused feature matrix; The attention mechanism determines the contribution of each feature dimension in the cross-modal weight matrix by calculating the cosine similarity between the visible light behavior feature vector and the thermal imaging behavior feature vector.

[0008] In a possible implementation of the first aspect, performing a multi-dimensional matching analysis on the dynamic behavior sequence features based on a preset abnormal behavior determination model to determine whether the target object triggers abnormal behavior features in the monitoring area includes: Dividing the dynamic behavior sequence feature into a plurality of time segment features, and performing local abnormality calculation on each time segment feature to obtain a local abnormality calculation result; Performing cumulative analysis on the local abnormality calculation results based on a sliding window mechanism to generate a global abnormality score; Comparing the global abnormality score with a preset abnormality threshold, and if the global abnormality score exceeds the abnormality threshold, determining that the target object triggers the abnormal behavior feature; Among them, the local abnormality calculation includes a comprehensive evaluation of the deviation distance between the movement trajectory in the dynamic behavior sequence characteristics and the preset normal trajectory template, the difference between the posture change frequency and the preset normal posture frequency, and the similarity between the thermal distribution characteristics and the preset normal thermal distribution.

[0009] In a possible implementation of the first aspect, the performing local abnormality calculation on each time segment feature to obtain a local abnormality calculation result includes: Extracting a moving trajectory point set of the target object from the dynamic behavior sequence features, and calculating the Hausdorff distance between the moving trajectory point set and a preset normal trajectory template; Extracting the posture change frequency of the target object from the dynamic behavior sequence features, and calculating the Euclidean distance between the posture change frequency and a preset normal posture frequency; Extracting a thermal distribution histogram of the target object from the dynamic behavior sequence characteristics, and calculating a Bhattacharyya coefficient between the thermal distribution histogram and a preset normal thermal distribution; Inputting the Hausdorff distance, the Euclidean distance and the Bhattacharyya coefficient into a normalization function to generate a local abnormality calculation result of the time segment feature; The normalization function is used to map abnormal indicators of different dimensions to a unified numerical range for weighted sum calculation.

[0010] In a possible implementation of the first aspect, the generating a graded warning signal by combining the association weights of the dynamic behavior sequence features and the abnormal behavior features includes: Determining the danger level of the abnormal behavior feature according to the duration and spatial coverage of the abnormal behavior feature; Matching the corresponding warning level from a preset warning strategy library based on the danger level, and generating the graded warning signal; Binding the graded warning signal with the target object identification information in the dynamic behavior sequence feature to generate a warning event record; The warning strategy library contains warning trigger conditions, warning information push objects and security equipment linkage rules corresponding to different danger levels; The security response mechanism includes sending an early warning notification to a designated terminal, activating an audible and visual alarm device, and controlling an intelligent access control system to lock the entrance and exit of the monitored area.

[0011] In a possible implementation of the first aspect, binding the graded warning signal with the target object identification information in the dynamic behavior sequence feature to generate a warning event record includes: Extracting facial features, clothing color features and body contour features of the target object from the dynamic behavior sequence features to generate identification information of the target object; The timestamp of the graded warning signal, the target object identification information and the location code of the monitoring area are stored in a warning event database; Generate a statistical report on abnormal behavior in the monitoring area based on the historical records in the warning event database; The abnormal behavior statistical report includes the time distribution of abnormal events, high-frequency triggering areas and the behavior pattern classification of target objects.

[0012] In a possible implementation of the first aspect, dynamically adjusting the acquisition parameters of the visual acquisition device and the coverage of the monitoring area according to the spatial distribution and time duration of the abnormal behavior characteristics includes: If the spatial distribution of the abnormal behavior characteristics is concentrated in the first sub-area of ​​the monitoring area, controlling the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area; If the duration of the abnormal behavior feature exceeds a preset threshold, the visual acquisition device is controlled to expand the shooting angle range of the monitoring area, and the backup camera is enabled to cover the blind spot; Reacquire the video stream of the monitoring area based on the adjusted acquisition parameters, and update the analysis result of the dynamic behavior sequence characteristics; The adjustment of the acquisition parameters includes real-time optimization of exposure time, focal length and white balance parameters to adapt to the abnormality detection requirements under different lighting conditions; The step of controlling the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area includes: Identify the bounding box coordinates of all active objects in the first sub-area by using a target detection algorithm, and calculate the center point position and coverage radius of the first sub-area according to the bounding box coordinates; Generate a pan / tilt control instruction based on the center point position and the coverage radius, adjust the pan / tilt rotation angle and zoom ratio of the visual acquisition device, and control the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area under the adjusted pan / tilt rotation angle and zoom ratio, perform local high-definition video stream acquisition on the first sub-area, and splice it with the video streams of other areas; The stitching process includes performing feature point matching and perspective transformation on the images in the overlapping areas to generate a seamlessly integrated panoramic monitoring picture.

[0013] In a possible implementation of the first aspect, the method further includes: Performing background modeling on the multimodal video stream to separate static background features and dynamic foreground features of the monitoring area; Constructing a three-dimensional spatial topological map of the monitoring area based on the static background features, and marking the location information of key security nodes; Overlaying and analyzing the dynamic foreground features with the three-dimensional space topology map to determine the real-time distance and movement trend of the target object relative to the key security node; If the real-time distance is less than the preset safety distance threshold and the movement trend points to the key security node, the detection sensitivity of the abnormal behavior feature is improved; wherein the key security node includes the safe location, power control box and emergency exit sign.

[0014] On the other hand, an embodiment of the present invention further provides a server, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0015] Based on the above aspects, the embodiment of the present application can capture target scene information from multiple dimensions by using visual acquisition equipment deployed in the monitoring area to obtain a multimodal video stream containing a visible light image sequence and an infrared thermal imaging image sequence. The visible light image provides detailed features such as the appearance and color of the target object, while the infrared thermal imaging image can effectively obtain the thermal distribution of the target object. The combination of the two makes the perception of the target scene more comprehensive and accurate, and greatly improves the ability to understand the behavioral characteristics of the target object. Secondly, when extracting spatiotemporal features from the multimodal video stream to generate the dynamic behavior sequence features of the target object, the included movement trajectory, posture changes, thermal distribution characteristics and other information can more completely and delicately portray the behavior pattern of the target object in the monitoring area. Compared with the traditional method that can only obtain limited behavior information, it can deeply explore the complexity and diversity of the target object's behavior. Furthermore, multi-dimensional matching analysis based on the preset abnormal behavior judgment model has changed the traditional simple rule matching mode. This multi-dimensional matching analysis not only considers a single feature, but also integrates multiple dimensions of dynamic behavior sequence features, which greatly improves the accuracy and reliability of abnormal behavior judgment, can effectively identify various abnormal behaviors in complex environments, and reduce the false alarm rate. After the abnormal behavior detection is triggered, a graded warning signal is generated by combining the correlation weights of the dynamic behavior sequence features and the abnormal behavior features, and based on this, the corresponding security response mechanism is triggered, realizing the accurate response to abnormal events and the reasonable allocation of resources. It can intelligently allocate security resources according to the severity and complexity of the abnormal behavior, and improve the response efficiency and pertinence of the security system. Finally, the acquisition parameters of the visual acquisition equipment and the coverage of the monitoring area are dynamically adjusted according to the spatial distribution and time duration of the abnormal behavior characteristics. The acquisition parameters and monitoring range of traditional monitoring systems are often fixed, and it is difficult to adapt to the dynamic changes of abnormal behavior in actual scenarios. This method can adaptively optimize the acquisition parameters and coverage range, which can not only effectively focus on the area where abnormal behavior occurs, but also promptly capture subsequent abnormal situations that may occur, further improving the anomaly detection accuracy of subsequent video streams and realizing intelligent and dynamic management of monitoring resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the execution flow of the early warning analysis method based on intelligent vision provided by an embodiment of the present invention.

[0017] Figure 2 It is a schematic diagram of the hardware architecture of the server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1It is a flow chart of an early warning analysis method based on intelligent vision provided by an embodiment of the present invention. The early warning analysis method based on intelligent vision is introduced in detail below.

[0019] Step S110, acquiring a multimodal video stream of a target scene in real time through a visual acquisition device deployed in a monitoring area, wherein the multimodal video stream includes a visible light image sequence and an infrared thermal imaging image sequence.

[0020] For example, consider a warehouse as a monitoring area, and deploy visual acquisition devices in different corners of the warehouse. The visual acquisition device has the ability to simultaneously acquire visible light image sequences and infrared thermal imaging image sequences. Various goods are stored inside the warehouse, and there are different areas such as shelves, cargo piles, and aisles. During normal working hours, workers carry out operations such as cargo handling and inventory in the warehouse. The visual acquisition device continuously acquires multimodal video streams of the target scene (inside the warehouse). For example, in the visible light image sequence, you can clearly see the workers wearing work clothes of different colors, the packaging color and shape of the goods, and the layout of the shelves. The infrared thermal imaging image sequence can show the thermal radiation of the workers' body temperature, as well as some goods or equipment that may have temperature differences (such as electric handling equipment that has just been started may have a slightly higher temperature). In this way, the visual acquisition device can capture the scene in the warehouse in real time and comprehensively, providing basic data for subsequent analysis.

[0021] Step S120, performing spatiotemporal feature extraction on the multimodal video stream to generate dynamic behavior sequence features of the target object in the monitoring area, wherein the dynamic behavior sequence features include movement trajectory, posture change, and thermal distribution features of the target object.

[0022] For the multimodal video stream obtained from the warehouse, the visible light image sequence is first input into the first convolutional neural network. Taking the staff as an example, the first convolutional neural network can accurately extract the contour shape features of the staff, such as the general outline of the human body, the extended shape of the limbs, etc., as well as the surface texture features, such as the texture of the work clothes, the texture of the tools in the hand, etc., thereby generating a visible light behavior feature vector.

[0023] At the same time, the infrared thermal imaging image sequence is input into the second convolutional neural network. Since the human body temperature is relatively stable and higher than the surrounding environment, the second convolutional neural network can extract the distribution characteristics of the thermal radiation intensity of the staff, such as the heat dissipation of various parts of the body, and the temperature change gradient characteristics, such as the temperature change gradient caused by a slight increase in the temperature of the muscle exertion part when carrying heavy objects, and then generate a thermal imaging behavior feature vector.

[0024] Next, the visible light behavior feature vector and the thermal imaging behavior feature vector are cross-modally fused. The two vectors are mapped to the high-dimensional feature space respectively to obtain the first high-dimensional feature vector and the second high-dimensional feature vector. The cross-modal weight matrix between the two is calculated by the attention mechanism. The attention mechanism determines the contribution of each feature dimension in the cross-modal weight matrix by calculating the cosine similarity between the visible light behavior feature vector and the thermal imaging behavior feature vector. For example, if a certain action of the staff causes the body temperature to rise, the contribution of the feature dimension in thermal imaging may increase. Based on the cross-modal weight matrix, the first high-dimensional feature vector and the second high-dimensional feature vector are weighted and summed to generate an intermediate fused feature vector, which is then input into the fully connected layer for dimensionality reduction to generate an initial fused feature matrix.

[0025] Then, based on the spatial position changes of the target object (staff) in continuous time frames, the initial fusion feature matrix is ​​modeled as a time series. The spatial position coordinates of the staff in continuous time frames are extracted from the initial fusion feature matrix. Assuming that the staff walks from one shelf to another in the warehouse, a spatial position coordinate sequence can be generated based on its position coordinates in each time frame, and an instantaneous velocity vector sequence is generated based on the position coordinate difference of adjacent time frames. For example, the staff walks faster in a flat aisle and walks slower between piles of goods. These speed changes are reflected in the instantaneous velocity vector sequence. Based on the difference in the direction angle of the velocity vector of two consecutive time frames in the instantaneous velocity vector sequence, a direction offset angle sequence is generated. At the same time, the posture angle parameters of the staff in continuous time frames are extracted from the initial fusion feature matrix to generate a posture angle sequence. For changes in posture angles such as bending over to carry goods and walking upright, the posture angle parameters of adjacent time frames in the posture angle sequence are differentially operated to generate a posture change rate sequence.

[0026] The instantaneous velocity vector sequence, the direction offset angle sequence and the posture change rate sequence are aligned along the time axis to generate a joint motion feature matrix. The sliding time window is used to perform mean statistics and variance statistics on the joint motion feature matrix to generate a time window statistical feature matrix, where the size of the sliding time window is dynamically adjusted according to the historical movement frequency of the staff. If the staff frequently moves quickly in a certain area (high-frequency movement), a smaller window size is used; if they move slowly in a certain area for a long time (low-frequency movement), a larger window size is used. Finally, the time window statistical feature matrix and the initial fusion feature matrix are spliced ​​in the channel dimension to generate an enhanced spatiotemporal feature matrix. The enhanced spatiotemporal feature matrix is ​​modeled for time dependence through a temporal convolutional network (including dilated convolutional layers and causal convolutional layers) to generate dynamic behavior sequence features containing long-term and short-term motion correlations. This feature can fully reflect the movement trajectory, posture changes, and thermal distribution characteristics of the staff in the warehouse.

[0027] Step S130: performing a multi-dimensional matching analysis on the dynamic behavior sequence features based on a preset abnormal behavior determination model to determine whether the target object triggers abnormal behavior features in the monitoring area.

[0028] In this embodiment, the dynamic behavior sequence features generated above can be divided into multiple time segment features. Taking the activities of staff in the warehouse as an example, each time segment may correspond to the time period when the staff completes a small task in the warehouse, such as picking up goods from a shelf and moving them to another designated area. The local abnormality degree is calculated for each time segment feature.

[0029] Extract the staff's moving trajectory point set from the dynamic behavior sequence features, and calculate its Hausdorff distance with the preset normal trajectory template (for example, the cargo handling route specified by the warehouse). Assuming that the normal route is from shelf A along the aisle to shelf B, if the staff deviates from this route and walks to other non-working areas of the warehouse, the Hausdorff distance will increase. Extract the staff's posture change frequency from the dynamic behavior sequence features, and calculate its Euclidean distance with the preset normal posture frequency (the frequency of bending, raising hands, etc. when carrying goods normally). If the staff suddenly maintains an abnormal posture for a long time (such as squatting for a long time without moving), the Euclidean distance will increase. Extract the staff's thermal distribution histogram from the dynamic behavior sequence features, and calculate its Bhattacharyya coefficient with the preset normal thermal distribution (normal body temperature and body heat distribution under normal working conditions). If the staff's body suddenly becomes abnormally hot (it may be because of physical discomfort or carrying a heat source), the Bhattacharyya coefficient will decrease.

[0030] The Hausdorff distance, Euclidean distance, and Bhattacharyya coefficient are input into the normalization function to generate the local anomaly calculation results of the time segment features. The local anomaly calculation results are cumulatively analyzed based on the sliding window mechanism to generate a global anomaly score. For example, if the staff member continuously deviates from the normal trajectory, has abnormal posture, and abnormal thermal distribution in multiple time segments, the global anomaly score will continue to increase. The global anomaly score is compared with the preset anomaly threshold. If the global anomaly score exceeds the anomaly threshold, it is determined that the staff member has triggered abnormal behavior characteristics in the warehouse.

[0031] Step S140: If it is detected that the target object triggers the abnormal behavior feature in the monitoring area, a graded warning signal is generated by combining the dynamic behavior sequence feature with the associated weight of the abnormal behavior feature, and a corresponding security response mechanism is triggered based on the graded warning signal.

[0032] In this embodiment, if it is detected that a staff member has triggered an abnormal behavior feature in the warehouse, for example, a staff member enters the valuables storage area of ​​the warehouse during non-working hours and behaves abnormally. The danger level of the abnormal behavior feature is determined according to the duration and spatial coverage of the abnormal behavior feature. If the staff member stays in the valuables storage area for a long time and has a large range of activities, it may be determined to be a higher danger level.

[0033] Based on the danger level, the corresponding warning level is matched from the preset warning strategy library, and a graded warning signal is generated. Assuming that for a high danger level, the warning trigger conditions in the warning strategy library are to immediately notify the security personnel, push detailed information to the warehouse management system, and start the sound and light alarm device. The objects of warning information push include warehouse managers, security department heads, etc., and the security equipment linkage rules are to block the entrances and exits of the area and turn on the emergency storage of monitoring records. The generated graded warning signal will trigger the corresponding security response mechanism according to these rules, send warning notifications to designated terminals (mobile phones or computer terminals of warehouse managers and security department heads), start the sound and light alarm device to warn other personnel, and control the intelligent access control system to lock the entrances and exits of the valuables storage area in the warehouse to prevent further development of abnormal behavior.

[0034] Bind the graded warning signal with the target object identification information in the dynamic behavior sequence features to generate a warning event record. Extract the facial features, clothing color features (such as blue work clothes) and body contour features of the staff from the dynamic behavior sequence features to generate the target object identification information. Store the timestamp of the graded warning signal, the target object identification information and the warehouse location code (such as the specific shelf area number) in the warning event database. Based on the historical records in the warning event database, a statistical report on abnormal behavior of the warehouse can be generated. For example, it is found that abnormal events of entering a specific area during non-working hours are mainly concentrated at night, the high-frequency triggering area is the valuables storage area, and the behavior pattern of the target object is classified as unauthorized entry into a specific area and abnormal activities.

[0035] Step S150, dynamically adjusting the acquisition parameters of the visual acquisition device and the coverage of the monitoring area according to the spatial distribution and time duration of the abnormal behavior characteristics, so as to optimize the anomaly detection accuracy of subsequent video streams.

[0036] In this embodiment, the acquisition parameters of the visual acquisition device and the coverage of the monitoring area can be dynamically adjusted according to the spatial distribution and time duration of the abnormal behavior characteristics. If it is found that the spatial distribution of the abnormal behavior characteristics of the staff is concentrated in a specific sub-area of ​​the warehouse, such as a corner of the valuables storage area, the visual acquisition device is controlled to increase the image acquisition frame rate and resolution of the first sub-area (the corner of the valuables storage area).

[0037] First, the bounding box coordinates of all active objects (staff members) in the sub-area are identified through the target detection algorithm, and the center point position and coverage radius of the sub-area are calculated based on the bounding box coordinates. The pan-tilt control instructions are generated based on the center point position and coverage radius, and the pan-tilt rotation angle and zoom multiple of the visual acquisition device are adjusted. Under the adjusted pan-tilt rotation angle and zoom multiple, the visual acquisition device is controlled to increase the image acquisition frame rate and resolution of the sub-area, and the local high-definition video stream of the sub-area is collected and spliced ​​with the video streams of other areas. During the splicing process, the images of the overlapping areas are matched with feature points and perspective transformed to generate a seamlessly integrated panoramic monitoring picture, so that the behavioral details of the staff in the area can be observed more clearly.

[0038] If the duration of the abnormal behavior feature exceeds the preset threshold, for example, the abnormal behavior of the staff in the warehouse lasts for a long time (more than the set 5 minutes), the visual acquisition device is controlled to expand the shooting angle range of the monitoring area (the entire warehouse), and the backup camera is enabled to cover the blind spot. Based on the adjusted acquisition parameters, the video stream of the warehouse is re-acquired, and the analysis results of the dynamic behavior sequence features are updated to more accurately detect abnormal behavior and optimize the accuracy of anomaly detection of subsequent video streams. The adjustment of acquisition parameters also includes real-time optimization of exposure time, focal length and white balance parameters to adapt to the anomaly detection requirements under different lighting conditions in the warehouse. For example, in darker areas of the warehouse, adjust the exposure time and white balance parameters to ensure that clear video images are obtained for anomaly detection.

[0039] Based on the above steps, the embodiment of the present application can capture target scene information from multiple dimensions by using visual acquisition equipment deployed in the monitoring area to obtain a multimodal video stream containing a visible light image sequence and an infrared thermal imaging image sequence. The visible light image provides detailed features such as the appearance and color of the target object, while the infrared thermal imaging image can effectively obtain the thermal distribution of the target object. The combination of the two makes the perception of the target scene more comprehensive and accurate, greatly improving the ability to understand the behavioral characteristics of the target object. Secondly, when extracting spatiotemporal features from the multimodal video stream to generate the dynamic behavior sequence features of the target object, the included movement trajectory, posture changes, thermal distribution characteristics and other information can more completely and delicately portray the behavior pattern of the target object in the monitoring area. Compared with the traditional method that can only obtain limited behavior information, it can deeply explore the complexity and diversity of the target object's behavior. Furthermore, multi-dimensional matching analysis based on the preset abnormal behavior judgment model has changed the traditional simple rule matching mode. This multi-dimensional matching analysis not only considers a single feature, but also integrates multiple dimensions of dynamic behavior sequence features, which greatly improves the accuracy and reliability of abnormal behavior judgment, can effectively identify various abnormal behaviors in complex environments, and reduce the false alarm rate. After the abnormal behavior detection is triggered, a graded warning signal is generated by combining the correlation weights of the dynamic behavior sequence features and the abnormal behavior features, and based on this, the corresponding security response mechanism is triggered, realizing the accurate response to abnormal events and the reasonable allocation of resources. It can intelligently allocate security resources according to the severity and complexity of the abnormal behavior, and improve the response efficiency and pertinence of the security system. Finally, the acquisition parameters of the visual acquisition equipment and the coverage of the monitoring area are dynamically adjusted according to the spatial distribution and time duration of the abnormal behavior characteristics. The acquisition parameters and monitoring range of traditional monitoring systems are often fixed, and it is difficult to adapt to the dynamic changes of abnormal behavior in actual scenarios. This method can adaptively optimize the acquisition parameters and coverage range, which can not only effectively focus on the area where abnormal behavior occurs, but also promptly capture subsequent abnormal situations that may occur, further improving the anomaly detection accuracy of subsequent video streams and realizing intelligent and dynamic management of monitoring resources.

[0040] In a possible implementation, step S120 includes: Step S121, inputting the visible light image sequence into a first convolutional neural network, extracting contour shape features and surface texture features of the target object, and generating a visible light behavior feature vector.

[0041] In this embodiment, in the warehouse scene, the first convolutional neural network can perform a detailed analysis of the visible light image of the staff. The image of the staff in the warehouse is clearly presented through the visible light image, such as the color, style and body contour of the work clothes they wear. The first convolutional neural network extracts the contour shape features of the target object (staff), such as the overall outline of the body, the general shape of the limbs, etc. These contour shape features help determine the posture and range of motion of the staff. At the same time, surface texture features are also extracted, such as the texture on the work clothes, the texture of the work shoes, and the texture of the surface of the handling tools in the hands. Through the extraction of these features, a visible light behavior feature vector is generated, which contains feature information related to various behaviors reflected by the staff under the visible light image.

[0042] Step S122: input the infrared thermal imaging image sequence into a second convolutional neural network, extract the thermal radiation intensity distribution characteristics and temperature change gradient characteristics of the target object, and generate a thermal imaging behavior feature vector.

[0043] In a warehouse environment, there is a temperature difference between the body temperature of the staff and the surrounding environment, and the distribution and change of their body heat during work have certain regularity. The second convolutional neural network processes these infrared thermal imaging images to extract the distribution characteristics of the thermal radiation intensity of the staff. For example, the thermal radiation intensity of the core parts of the staff's body (such as the torso) is relatively high, and the thermal radiation intensity of the limbs is relatively low. In addition, when doing physical labor, the thermal radiation intensity of the muscle parts may be enhanced due to faster blood circulation. At the same time, the temperature change gradient characteristics are also extracted. For example, when a staff member quickly walks from a colder warehouse aisle to a cargo storage area with a slightly higher temperature, a temperature change gradient may appear on the body surface. Based on the thermal radiation intensity distribution characteristics and the temperature change gradient characteristics, a thermal imaging behavior feature vector can be generated.

[0044] Step S123 , performing cross-modal fusion processing on the visible light behavior feature vector and the thermal imaging behavior feature vector to generate an initial fusion feature matrix.

[0045] Step S124, based on the spatial position change of the target object in the continuous time frame, the initial fusion feature matrix is ​​modeled in time series to generate the dynamic behavior sequence feature.

[0046] The time series modeling includes a continuity analysis of the movement speed, direction offset and posture angle of the target object in adjacent time frames to construct the spatiotemporal correlation between the movement trajectory and posture changes.

[0047] In a possible implementation, step S124 includes: Step S1241, extracting the spatial position coordinates of the target object in continuous time frames from the initial fusion feature matrix, generating a spatial position coordinate sequence, and generating an instantaneous velocity vector sequence of the target object based on the position coordinate difference of adjacent time frames in the spatial position coordinate sequence, and generating a direction offset angle sequence based on the velocity vector direction angle difference between two consecutive time frames in the instantaneous velocity vector sequence.

[0048] For example, when a worker walks from one shelf to another, the position coordinates of the worker in the warehouse are recorded in each time frame, and these position coordinates form a spatial position coordinate sequence. According to the position coordinate difference of adjacent time frames in the spatial position coordinate sequence, the instantaneous speed vector sequence of the worker is generated. For example, when a worker walks in a wide aisle, the position coordinate difference of adjacent time frames is large, and the value of the instantaneous speed vector is large, indicating a faster speed; while when walking in an area where goods are stacked densely, the position coordinate difference is small, the instantaneous speed vector value is also small, and the speed is slow. Based on the difference in the direction angle of the speed vector of two consecutive time frames in the instantaneous speed vector sequence, a direction offset angle sequence is generated. When a worker turns or changes the walking direction in the warehouse, the direction angle of the speed vector changes, and these changes are reflected by calculating the direction offset angle sequence.

[0049] Step S1242, extracting the posture angle parameters of the target object in continuous time frames from the initial fusion feature matrix, generating a posture angle sequence, and performing a differential operation on the posture angle parameters of adjacent time frames in the posture angle sequence to generate a posture change rate sequence.

[0050] For example, the postures of workers in the warehouse vary, such as when bending over to carry goods, walking upright to check the goods labels, and raising hands to put goods on the shelves. The angle parameters corresponding to these different postures are recorded to form a posture angle sequence. The posture angle parameters of adjacent time frames in the posture angle sequence are differentially calculated to generate a posture change rate sequence, which can reflect the speed of the posture change of the workers. For example, if the process from bending over to standing upright is completed in a short time, the posture change rate is large; if it is slowly standing up, the posture change rate is small.

[0051] Step S1243, align the instantaneous velocity vector sequence, the direction offset angle sequence and the posture change rate sequence according to the time axis to generate a joint motion feature matrix, and use a sliding time window to perform mean statistics and variance statistics on the joint motion feature matrix to generate a time window statistical feature matrix, and splice the time window statistical feature matrix with the initial fusion feature matrix in the channel dimension to generate an enhanced spatiotemporal feature matrix.

[0052] In detail, the joint motion feature matrix integrates information on the speed, direction, and posture changes of the staff. The joint motion feature matrix is ​​subjected to mean and variance statistics using a sliding time window to generate a time window statistical feature matrix. The size of the sliding time window is dynamically adjusted according to the historical movement frequency of the staff. For example, the staff often performs fast cargo handling tasks in the warehouse, which is a high-frequency movement. At this time, a smaller sliding time window size can more finely capture the changes in its motion features in a short period of time; while when performing relatively low-frequency movements such as cargo inventory, a larger sliding time window size can better reflect the average level of motion features over a longer period of time. The time window statistical feature matrix is ​​spliced ​​with the initial fusion feature matrix in the channel dimension to generate an enhanced spatiotemporal feature matrix. This channel dimension splicing is to expand the feature channels of the statistical indicators of the time window statistical feature matrix and the cross-modal features of the initial fusion feature matrix to form fused data containing motion statistical characteristics and original perceptual characteristics. For example, the mean and variance in the statistical indicators are combined with the original visible light and infrared thermal imaging features, so that the enhanced spatiotemporal feature matrix can more comprehensively reflect the motion characteristics of the staff in the warehouse.

[0053] Step S1244, performing time dependency modeling on the enhanced spatiotemporal feature matrix through a temporal convolutional network to generate dynamic behavior sequence features containing long-term and short-term motion correlations.

[0054] The temporal convolutional network includes a dilated convolutional layer and a causal convolutional layer, which are used to capture the multi-scale motion pattern of the target object in a continuous time frame.

[0055] The size of the sliding time window is dynamically adjusted according to the historical movement frequency of the target object, so that a smaller window size is used during high-frequency movement and a larger window size is used during low-frequency movement.

[0056] Among them, the channel dimension splicing includes performing feature channel expansion on the statistical indicators of the time window statistical feature matrix and the cross-modal features of the initial fusion feature matrix to form fusion data containing motion statistical characteristics and original perceptual characteristics.

[0057] In this embodiment, the temporal convolutional network includes an expanded convolutional layer and a causal convolutional layer, which can capture the multi-scale motion patterns of workers in continuous time frames. In the warehouse scenario, the expanded convolutional layer can expand the receptive field, thereby capturing the motion trends of workers in a longer time span, such as the long-term movement trajectory of workers from one area to another in the warehouse; the causal convolutional layer ensures the causal relationship of the time series, that is, the features of the current time frame only depend on the features of the past time frame, which conforms to the time sequence logic of the workers' movement in the warehouse. The dynamic behavior sequence features generated can fully reflect the movement trajectory, posture changes, and thermal distribution characteristics of the workers in the warehouse.

[0058] Step S123 includes: Step S1231 : Mapping the visible light behavior feature vector and the thermal imaging behavior feature vector to a high-dimensional feature space respectively to generate a first high-dimensional feature vector and a second high-dimensional feature vector.

[0059] Step S1232: Calculate a cross-modal weight matrix between the first high-dimensional feature vector and the second high-dimensional feature vector through an attention mechanism.

[0060] Step S1233: Perform weighted summation on the first high-dimensional feature vector and the second high-dimensional feature vector based on the cross-modal weight matrix to generate an intermediate fused feature vector.

[0061] Step S1234: input the intermediate fused feature vector into a fully connected layer for dimensionality reduction processing to generate the initial fused feature matrix.

[0062] The attention mechanism determines the contribution of each feature dimension in the cross-modal weight matrix by calculating the cosine similarity between the visible light behavior feature vector and the thermal imaging behavior feature vector.

[0063] In detail, in the warehouse scenario, the visible light behavior feature vector and the thermal imaging behavior feature vector are respectively mapped to the high-dimensional feature space, so that the features of the staff under visible light and the features under infrared thermal imaging have a new representation in the high-dimensional space. Then, the cross-modal weight matrix between the first high-dimensional feature vector and the second high-dimensional feature vector is calculated by the attention mechanism. This attention mechanism determines the contribution of each feature dimension in the cross-modal weight matrix by calculating the cosine similarity between the visible light behavior feature vector and the thermal imaging behavior feature vector. For example, if the body temperature of the staff rises while carrying heavy objects, the posture of the action under visible light also changes significantly, then the contribution of the feature dimensions related to thermal imaging and visible light will be adjusted accordingly. Based on the cross-modal weight matrix, the first high-dimensional feature vector and the second high-dimensional feature vector are weighted and summed to generate an intermediate fused feature vector, and then the intermediate fused feature vector is input into the fully connected layer for dimensionality reduction processing, and finally an initial fused feature matrix is ​​generated. The initial fused feature matrix integrates the behavioral feature information of the staff under the two modalities of visible light and infrared thermal imaging.

[0064] In a possible implementation, step S130 includes: Step S131 : dividing the dynamic behavior sequence feature into a plurality of time segment features, and performing local abnormality calculation on each time segment feature to obtain a local abnormality calculation result.

[0065] In a warehouse environment, time segment features can be divided according to the time phases when workers perform different tasks in the warehouse. For example, the process of workers moving goods from one shelf to another can be used as a time segment feature, or the time period when workers take inventory of goods in a certain area can also be used as a time segment feature. Then, the local anomaly degree is calculated for each time segment feature.

[0066] Step S132: performing cumulative analysis on the local abnormality calculation results based on a sliding window mechanism to generate a global abnormality score.

[0067] In this embodiment, the sliding window mechanism can determine the window size according to the work cycle or task flow of the staff in the warehouse scenario. For example, a sliding window cycle is defined as a round of cargo handling and inventory tasks completed by the staff. In each window, the local abnormality calculation results of each time segment feature are accumulated. As time goes by, the local abnormality calculation results in multiple sliding windows are continuously accumulated to form a global abnormality score, which comprehensively reflects the degree of abnormal behavior of the staff over a longer period of time.

[0068] Step S133, comparing the global abnormality score with a preset abnormality threshold, if the global abnormality score exceeds the abnormality threshold, it is determined that the target object triggers the abnormal behavior feature.

[0069] Among them, the local abnormality calculation includes a comprehensive evaluation of the deviation distance between the movement trajectory in the dynamic behavior sequence characteristics and the preset normal trajectory template, the difference between the posture change frequency and the preset normal posture frequency, and the similarity between the thermal distribution characteristics and the preset normal thermal distribution.

[0070] In this embodiment, the preset abnormal threshold is set according to the warehouse's safety requirements, work specifications, and past experience data. For example, if the global abnormality score indicates that the staff member has deviated from the normal trajectory many times, has frequently abnormal postures, and has serious abnormal thermal distribution, exceeding the set abnormal threshold, then it can be determined that the staff member has triggered abnormal behavior characteristics in the warehouse, which may mean that the staff member has unauthorized activities, physical discomfort that affects work, or there may be safety hazards, thereby providing a basis for subsequent early warning and security measures.

[0071] In a possible implementation, step S131 includes: Step S1311 , extracting a moving trajectory point set of the target object from the dynamic behavior sequence features, and calculating a Hausdorff distance between the moving trajectory point set and a preset normal trajectory template.

[0072] For example, the set of mobile trajectory points reflects the location of the staff at different points in time. Calculate the Hausdorff distance between this mobile trajectory point set and the preset normal trajectory template. The preset normal trajectory template is set based on the normal working process and safety regulations of the warehouse. For example, according to the prescribed cargo handling route, the staff should go from shelf A along a specific aisle to shelf B for cargo handover. If the staff's mobile trajectory point set deviates from this normal trajectory template, the Hausdorff distance will increase. For example, the staff should go to the cargo storage area along the main aisle, but go to the distribution room of the warehouse. This deviation will be quantified by the Hausdorff distance.

[0073] Step S1312: extracting the posture change frequency of the target object from the dynamic behavior sequence feature, and calculating the Euclidean distance between the posture change frequency and a preset normal posture frequency.

[0074] In this embodiment, the preset normal posture frequency is determined based on normal working operations in the warehouse. For example, the frequency of bending over, raising hands, and other actions of workers when carrying goods normally is relatively stable. If a worker suddenly changes his posture frequency abnormally, such as frequently bending over or raising hands in an area where no goods need to be carried, this will be different from the preset normal posture frequency, and the degree of this difference can be quantified by calculating the Euclidean distance.

[0075] Step S1313, extracting the thermal distribution histogram of the target object from the dynamic behavior sequence characteristics, and calculating the Bhattacharyya coefficient between the thermal distribution histogram and a preset normal thermal distribution.

[0076] In this embodiment, the preset normal thermal distribution is determined based on the body temperature and body heat distribution of the staff in the normal working state. For example, when the staff is working normally and in good health, the heat distribution of various parts of the body is relatively stable. When the staff is doing physical labor, the heat of the core part of the body and the muscle force part will increase slightly, but it is also within the normal range. If the staff has an abnormal situation, such as a sudden fever (maybe due to illness or carrying a heat source), then the thermal distribution histogram will be different from the preset normal thermal distribution, and the change in this similarity can be measured by calculating the Bhattacharyya coefficient.

[0077] Step S1314: input the Hausdorff distance, the Euclidean distance and the Bhattacharyya coefficient into a normalization function to generate a local abnormality calculation result of the time segment feature.

[0078] The normalization function is used to map abnormal indicators of different dimensions to a unified numerical range for weighted sum calculation.

[0079] In this embodiment, the numerical range of the Hausdorff distance may be different from the numerical ranges of the Euclidean distance and the Bhattacharyya coefficient. Through the normalization function, they can be converted into the same numerical range, so that the influence of these three factors on the local abnormality can be comprehensively considered.

[0080] In a possible implementation, step S140 includes: Step S141, determining the danger level of the abnormal behavior feature according to the duration and spatial coverage of the abnormal behavior feature.

[0081] In this embodiment, if the abnormal behavior of the staff is characterized by unauthorized entry into the valuables storage area, the duration of the abnormal behavior plays an important role in determining the danger level. For example, if the staff only enters the area by mistake for a short time, the danger level may be relatively low; but if the staff stays in the area for a long time, it indicates that there may be a greater security risk and the danger level increases accordingly. Spatial coverage is also critical. If the staff is only active in a small corner of the valuables storage area, compared with walking around the entire valuables storage area, the latter has a larger spatial coverage and a higher danger level.

[0082] Step S142, matching the corresponding warning level from a preset warning strategy library based on the danger level, and generating the graded warning signal.

[0083] In this embodiment, the early warning strategy library contains early warning trigger conditions, early warning information push objects and security equipment linkage rules corresponding to different danger levels. For abnormal behaviors with low danger levels, such as staff staying in non-working areas for a short time, the early warning trigger condition may be to send only a prompt message to the warehouse manager, and the early warning information push object is the warehouse manager. The security equipment linkage rule may simply mark the event in the monitoring system for subsequent viewing. For abnormal behaviors with high danger levels, such as suspicious activities in the valuables storage area for a long time, the early warning trigger condition may be to send detailed early warning notifications to the warehouse manager, security personnel and warehouse supervisor at the same time. The early warning information push objects include the above-mentioned personnel. The security equipment linkage rule may be to start the sound and light alarm device to warn other personnel, control the intelligent access control system to lock the entrance and exit of the valuables storage area in the warehouse to prevent further development of abnormal behaviors, and push the monitoring screen of the area to the terminal device of the relevant personnel in real time. Through such a matching process, the corresponding graded early warning signal is generated.

[0084] Step S143: Bind the graded warning signal with the target object identification information in the dynamic behavior sequence feature to generate a warning event record.

[0085] The warning strategy library contains warning trigger conditions, warning information push objects and security equipment linkage rules corresponding to different danger levels. The security response mechanism includes sending warning notifications to designated terminals, activating sound and light alarm devices and controlling the intelligent access control system to lock the entrances and exits of the monitoring area.

[0086] In a possible implementation, step S143 includes: Step S1431, extracting facial features, clothing color features and body contour features of the target object from the dynamic behavior sequence features to generate identification information of the target object.

[0087] For example, the facial features of staff members can be used to accurately identify individuals, and clothing color features (such as blue overalls) and body shape features (such as height and body shape, etc.) can be used as auxiliary identification information. The timestamp of the graded warning signal, the identification information of the target object, and the location code of the monitoring area (such as the specific shelf area number) are stored in the warning event database. For example, at a certain point in time, a staff member wearing a blue overalls and with a specific body shape is found to have abnormal behavior near shelf A in the valuables storage area. This timestamp, the identification information of the staff member, and the location code of shelf A are stored together in the warning event database.

[0088] Step S1432: storing the timestamp of the graded warning signal, the target object identification information and the location code of the monitoring area into a warning event database.

[0089] Step S1433: Generate a statistical report on abnormal behavior in the monitoring area based on the historical records in the warning event database.

[0090] The abnormal behavior statistical report includes the time distribution of abnormal events, high-frequency triggering areas and the behavior pattern classification of target objects.

[0091] In this embodiment, the abnormal behavior statistical report includes the time distribution law of abnormal events, high-frequency triggering areas and the behavior pattern classification of target objects. By analyzing the data in the early warning event database, the time distribution law of abnormal events is found. For example, it may be found that most abnormal events of unauthorized entry into specific areas occur at night or during shift changes of warehouse personnel, which reflects the time distribution law of abnormal events. The determination of high-frequency triggering areas helps to strengthen security precautions in a targeted manner. For example, if it is found that the storage area of ​​valuables and the vicinity of the distribution room are high-frequency triggering areas of abnormal behavior, monitoring equipment can be added in these areas or security patrols can be strengthened. The behavior pattern classification of the target object can help to better understand the type of abnormal behavior, such as classifying abnormal behavior into different behavior pattern classifications such as unauthorized entry into specific areas and abnormal operations in specific areas (such as damage to goods or equipment). The above abnormal behavior statistical report provides comprehensive data support for the safety management of the warehouse, which helps to optimize the security strategy and improve the overall safety of the warehouse.

[0092] In a possible implementation, step S150 includes: Step S151: If the spatial distribution of the abnormal behavior characteristics is concentrated in the first sub-area of ​​the monitoring area, control the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area.

[0093] Step S152, if the duration of the abnormal behavior feature exceeds a preset threshold, the visual acquisition device is controlled to expand the shooting angle range of the monitoring area, and the backup camera is enabled to cover the blind spot.

[0094] If the duration of abnormal behavior characteristics exceeds the preset threshold, for example, if the abnormal behavior of staff in a suspicious area in the warehouse lasts for a long time (exceeding the set duration threshold), it is necessary to control the visual acquisition device to expand the shooting angle range of the monitored area (the entire warehouse) and enable the backup camera to cover the blind area. In a warehouse environment, the original shooting angle of the visual acquisition device may only cover part of the area. When the abnormal behavior lasts too long, in order to prevent the abnormal behavior from occurring or expanding in the unmonitored area, it is necessary to expand the shooting angle range. This may involve adjusting the rotation range of the pan / tilt or using a visual acquisition device with an adjustable viewing angle so that more warehouse areas can be seen. At the same time, there may be some blind spots in the warehouse, such as behind large shelves or in the corners of the warehouse, which may not be fully monitored under normal circumstances. Enabling the backup camera can cover these blind spots to ensure that the entire warehouse is under effective monitoring.

[0095] Step S153: reacquire the video stream of the monitoring area based on the adjusted acquisition parameters, and update the analysis result of the dynamic behavior sequence feature.

[0096] In this embodiment, the adjustment of acquisition parameters includes real-time optimization of exposure time, focal length and white balance parameters to adapt to the needs of abnormal detection under different lighting conditions. In a warehouse, the lighting conditions may vary depending on the density of goods stacking, the layout of warehouse lighting equipment, and time (day or night). The adjustment of exposure time can ensure that clear images are obtained under different light intensities. For example, in areas with darker lighting in the warehouse, the exposure time can be appropriately increased; while in areas with stronger light, the exposure time is reduced. The adjustment of focal length helps to accurately focus on active objects. The focal length is adjusted according to the distance between the active object and the visual acquisition device and the area to be monitored. The optimization of white balance parameters can make the color of the image more accurate and avoid image color distortion caused by light color deviation (such as the color of certain special lights in the warehouse). By re-acquiring the video stream and updating the analysis results of the dynamic behavior sequence features, the abnormal behavior in the warehouse can be analyzed more accurately, and the accuracy and effectiveness of abnormal behavior detection can be improved, thereby better ensuring the safety management of the warehouse.

[0097] The adjustment of the acquisition parameters includes real-time optimization of exposure time, focal length and white balance parameters to adapt to the abnormality detection requirements under different lighting conditions.

[0098] Wherein, step S151 includes: Step S1511: identifying the bounding box coordinates of all active objects in the first sub-region by using a target detection algorithm, and calculating the center point position and coverage radius of the first sub-region according to the bounding box coordinates.

[0099] In this embodiment, when the spatial distribution of abnormal behavior characteristics is concentrated in the first sub-area of ​​the monitoring area (warehouse), for example, in the specific sub-area of ​​the valuables storage area of ​​the warehouse, in order to monitor the situation in the area more clearly and meticulously, it is necessary to control the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area. First, the bounding box coordinates of all active objects (such as staff or possible abnormal personnel) in the first sub-area (valuables storage area) are identified by the target detection algorithm. In a warehouse environment, the target detection algorithm determines the position range of each active object based on the object features in the image, and these position information is expressed in the form of bounding box coordinates. Then, the center point position and coverage radius of the first sub-area are calculated based on these bounding box coordinates. Assuming that in the valuables storage area, by analyzing and calculating the bounding box coordinates of multiple active objects, it is concluded that the center point position of the area is approximately located at the center position of a certain shelf combination, and the coverage radius is determined according to the distribution range of the active objects, for example, the distance from the center point to the most edge active object is the coverage radius.

[0100] Step S1512, based on the center point position and the coverage radius, a pan / tilt control instruction is generated, the pan / tilt rotation angle and zoom ratio of the visual acquisition device are adjusted, and under the adjusted pan / tilt rotation angle and zoom ratio, the visual acquisition device is controlled to increase the image acquisition frame rate and resolution of the first sub-area, and local high-definition video streams are acquired for the first sub-area, and spliced ​​with the video streams of other areas. The splicing process includes feature point matching and perspective transformation of the images in the overlapping areas to generate a seamlessly integrated panoramic monitoring screen.

[0101] In this embodiment, the adjustment of the pan-tilt rotation angle enables the visual acquisition device to accurately align with the center point position of the valuables storage area, and the adjustment of the zoom multiple determines the appropriate magnification according to the coverage radius so as to completely cover the sub-area. Under the adjusted pan-tilt rotation angle and zoom multiple, the visual acquisition device is controlled to increase the image acquisition frame rate and resolution of the first sub-area (valuables storage area). The increase in the image acquisition frame rate means that more image frames can be obtained per unit time, which helps to capture more subtle movement changes of the active object; the increase in resolution makes the image clearer and can more accurately identify the detailed features of the active object, such as facial features, held items, etc. After the local high-definition video stream of the first sub-area is collected, it is necessary to splice it with the video streams of other areas. During the splicing process, the images of the overlapping areas are matched with feature points and perspective transformed. Since the layout of the warehouse is fixed, the visual acquisition devices in different areas may have overlapping parts in terms of coverage. In the overlapping area, feature points in the image are identified, such as the corners of the shelf, specific identification of the goods, etc., and feature point matching is performed. Then, images from different perspectives are adjusted to images from the same perspective based on the principle of perspective transformation, thereby generating a seamlessly integrated panoramic monitoring picture. Such a panoramic monitoring picture allows monitoring personnel to observe the situation of the entire warehouse comprehensively and clearly, while focusing on the valuable storage area where abnormal behavior is concentrated.

[0102] In a possible implementation, the method further includes: Step S210: performing background modeling on the multimodal video stream to separate static background features and dynamic foreground features of the monitoring area.

[0103] In this embodiment, the multimodal video stream includes a visible light image sequence and an infrared thermal imaging image sequence. By performing background modeling on these multimodal video streams, the relatively fixed static background features in the warehouse can be accurately separated from the constantly changing dynamic foreground features. For the visible light image sequence, the static background features include the building structure of the warehouse, such as walls, ceilings, fixed shelf layouts, fixed lighting equipment, etc. These features will not change or change very slowly under normal circumstances. The dynamic foreground features are the target objects (staff, handling equipment, etc.) that are active in the warehouse and temporary situations, such as staff walking in the aisle, goods being transported to different locations, etc. For the infrared thermal imaging image sequence, the static background features may be the relatively stable part of the thermal conductivity characteristics of the warehouse building structure, and the thermal radiation characteristics of some equipment that is at room temperature and fixed. The dynamic foreground features are the body temperature thermal radiation of the staff, the heat emitted by the running equipment (such as electric transport trucks), etc.

[0104] Step S220: construct a three-dimensional spatial topological map of the monitoring area based on the static background features, and mark the location information of key security nodes.

[0105] During the construction process, various measurement and positioning technologies are used based on the actual layout of the warehouse. For example, an accurate three-dimensional space model is constructed through the known shelf size, aisle width, and length, width, and height information of the warehouse. Then the location information of key security nodes is marked on this three-dimensional space topology map. The key security nodes in the warehouse include the safe location, which is an important place to store valuables; the power control box, which is related to the power supply and equipment operation of the entire warehouse; and the emergency exit sign, which is a key location for personnel evacuation in an emergency. The locations of these key security nodes are accurately marked on the three-dimensional space topology map, so that there is a clear reference basis in the subsequent monitoring analysis.

[0106] Step S220, superimposing and analyzing the dynamic foreground feature and the three-dimensional space topology map to determine the real-time distance and movement trend of the target object relative to the key security node.

[0107] Take the activities of staff in the warehouse as an example. When staff walk in the warehouse, it is used as a dynamic foreground feature. By overlaying and analyzing the three-dimensional space topology map, the real-time distance and movement trend of the staff relative to the key security node can be determined. For example, when a staff member walks from a shelf to the safe, the real-time distance of the staff member relative to the safe can be obtained by locating the coordinates in the three-dimensional space topology map and calculating the distance to the safe position coordinates. At the same time, according to the position change of the staff member in the continuous time frame, its movement trend can be analyzed. If the movement trajectory of the staff member is gradually approaching the safe, then its movement trend is pointing to the key security node of the safe.

[0108] Step S230: if the real-time distance is less than a preset safety distance threshold and the movement trend points to the key security node, the detection sensitivity of the abnormal behavior feature is increased. The key security node includes the safe location, the power control box and the emergency exit sign.

[0109] The preset safety distance threshold is determined according to the warehouse's safety management regulations and risk assessment. For example, for the key security node of the safe, the set safety distance threshold may be 5 meters. If the real-time distance of the staff from the safe is less than 5 meters and is approaching the safe, there may be potential risks, such as unauthorized approach to the safe. At this time, improving the detection sensitivity of abnormal behavior characteristics can more accurately monitor the details of the staff's behavior. This may include more detailed analysis of abnormal behavior characteristics in terms of posture changes, residence time, and operating actions of the staff. For example, the normal operation of the staff near the safe may be just to sort out the surrounding goods, but if the residence time is too long or there are abnormal posture changes (such as frequently observing the environment around the safe, trying to touch the safe, etc.), these behaviors are more likely to be judged as abnormal behavior characteristics when the detection sensitivity is improved, thereby triggering corresponding early warnings and security measures in time to ensure the safety of the warehouse. The same is true for key security nodes such as power control boxes and emergency exit signs. In this way, potential security threats to key security nodes can be effectively prevented to ensure the normal operation of the warehouse and the safety of personnel and property.

[0110] Figure 2 The exemplary hardware components of the server 100 provided by some embodiments of the present application that can implement the concept of the present application are shown. For example, the processor 120 can be used on the server 100 and used to execute the functions in the present application.

[0111] The server 100 may be a general-purpose server or a special-purpose server, both of which may be used to implement the intelligent vision-based early warning analysis method of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application may be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0112] For example, the server 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the server 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The server 100 also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0113] For ease of explanation, only one processor is described in the server 100. However, it should be noted that the server 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the server 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually in one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor perform steps A and B together.

[0114] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer executable instructions are preset. When a processor executes the computer executable instructions, the above-mentioned early warning analysis method based on intelligent vision is implemented.

[0115] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. A warning analysis method based on intelligent vision, characterized in that: The method comprises: Acquire a multimodal video stream of a target scene in real time through a visual acquisition device deployed in a monitoring area, wherein the multimodal video stream includes a visible light image sequence and an infrared thermal imaging image sequence; Extracting spatiotemporal features from the multimodal video stream to generate dynamic behavior sequence features of the target object in the monitoring area, wherein the dynamic behavior sequence features include movement trajectory, posture change, and thermal distribution features of the target object; Perform multi-dimensional matching analysis on the dynamic behavior sequence characteristics based on a preset abnormal behavior determination model to determine whether the target object triggers abnormal behavior characteristics in the monitoring area; If it is detected that the target object triggers the abnormal behavior feature in the monitoring area, a graded warning signal is generated by combining the dynamic behavior sequence feature with the association weight of the abnormal behavior feature, and a corresponding security response mechanism is triggered based on the graded warning signal; According to the spatial distribution and time duration of the abnormal behavior characteristics, the acquisition parameters of the visual acquisition device and the coverage of the monitoring area are dynamically adjusted to optimize the anomaly detection accuracy of subsequent video streams.

2. The early warning analysis method based on intelligent vision according to claim 1 is characterized in that: The extracting spatiotemporal features of the multimodal video stream to generate dynamic behavior sequence features of the target object in the monitoring area includes: Inputting the visible light image sequence into a first convolutional neural network, extracting contour shape features and surface texture features of the target object, and generating a visible light behavior feature vector; Inputting the infrared thermal imaging image sequence into a second convolutional neural network, extracting the thermal radiation intensity distribution characteristics and temperature change gradient characteristics of the target object, and generating a thermal imaging behavior feature vector; Performing cross-modal fusion processing on the visible light behavior feature vector and the thermal imaging behavior feature vector to generate an initial fusion feature matrix; Based on the spatial position change of the target object in the continuous time frame, the initial fusion feature matrix is ​​modeled in time series to generate the dynamic behavior sequence feature; The time series modeling includes a continuity analysis of the movement speed, direction offset and posture angle of the target object in adjacent time frames to construct the spatiotemporal correlation between the movement trajectory and posture changes.

3. The early warning analysis method based on intelligent vision according to claim 2 is characterized in that: The step of performing time series modeling on the initial fusion feature matrix based on the spatial position change of the target object in the continuous time frame to generate the dynamic behavior sequence feature includes: Extracting the spatial position coordinates of the target object in continuous time frames from the initial fusion feature matrix to generate a spatial position coordinate sequence, and generating an instantaneous velocity vector sequence of the target object based on the position coordinate difference of adjacent time frames in the spatial position coordinate sequence, and generating a direction offset angle sequence based on the velocity vector direction angle difference of two consecutive time frames in the instantaneous velocity vector sequence; Extracting the posture angle parameters of the target object in continuous time frames from the initial fusion feature matrix to generate a posture angle sequence, and performing a differential operation on the posture angle parameters of adjacent time frames in the posture angle sequence to generate a posture change rate sequence; The instantaneous velocity vector sequence, the direction offset angle sequence and the attitude change rate sequence are aligned along the time axis to generate a joint motion feature matrix, and the joint motion feature matrix is ​​subjected to mean statistics and variance statistics using a sliding time window to generate a time window statistical feature matrix, and the time window statistical feature matrix is ​​spliced ​​with the initial fusion feature matrix in the channel dimension to generate an enhanced spatiotemporal feature matrix; Modeling the time dependency of the enhanced spatiotemporal feature matrix through a temporal convolutional network to generate dynamic behavior sequence features containing long-term and short-term motion correlations; The temporal convolutional network comprises a dilated convolutional layer and a causal convolutional layer, which are used to capture the multi-scale motion pattern of the target object in a continuous time frame; The size of the sliding time window is dynamically adjusted according to the historical movement frequency of the target object, so that a smaller window size is used during high-frequency movement and a larger window size is used during low-frequency movement; The channel dimension stitching includes performing feature channel expansion on the statistical indicators of the time window statistical feature matrix and the cross-modal features of the initial fusion feature matrix to form fusion data including motion statistical characteristics and original perceptual characteristics; The cross-modal fusion processing of the visible light behavior feature vector and the thermal imaging behavior feature vector to generate an initial fusion feature matrix includes: Mapping the visible light behavior feature vector and the thermal imaging behavior feature vector to a high-dimensional feature space respectively to generate a first high-dimensional feature vector and a second high-dimensional feature vector; Calculating a cross-modal weight matrix between the first high-dimensional feature vector and the second high-dimensional feature vector through an attention mechanism; Performing a weighted summation of the first high-dimensional feature vector and the second high-dimensional feature vector based on the cross-modal weight matrix to generate an intermediate fused feature vector; Inputting the intermediate fused feature vector into the fully connected layer for dimensionality reduction processing to generate the initial fused feature matrix; The attention mechanism determines the contribution of each feature dimension in the cross-modal weight matrix by calculating the cosine similarity between the visible light behavior feature vector and the thermal imaging behavior feature vector.

4. The early warning analysis method based on intelligent vision according to claim 1 is characterized in that: The performing of multi-dimensional matching analysis on the dynamic behavior sequence features based on a preset abnormal behavior determination model to determine whether the target object triggers abnormal behavior features in the monitoring area includes: Dividing the dynamic behavior sequence feature into a plurality of time segment features, and performing local abnormality calculation on each time segment feature to obtain a local abnormality calculation result; Performing cumulative analysis on the local abnormality calculation results based on a sliding window mechanism to generate a global abnormality score; Comparing the global abnormality score with a preset abnormality threshold, and if the global abnormality score exceeds the abnormality threshold, determining that the target object triggers the abnormal behavior feature; Among them, the local abnormality calculation includes a comprehensive evaluation of the deviation distance between the movement trajectory in the dynamic behavior sequence characteristics and the preset normal trajectory template, the difference between the posture change frequency and the preset normal posture frequency, and the similarity between the thermal distribution characteristics and the preset normal thermal distribution.

5. The early warning analysis method based on intelligent vision according to claim 4 is characterized in that: The local anomaly degree calculation is performed on each time segment feature to obtain the local anomaly degree calculation result, including: Extracting a moving trajectory point set of the target object from the dynamic behavior sequence features, and calculating the Hausdorff distance between the moving trajectory point set and a preset normal trajectory template; Extracting the posture change frequency of the target object from the dynamic behavior sequence features, and calculating the Euclidean distance between the posture change frequency and a preset normal posture frequency; Extracting a thermal distribution histogram of the target object from the dynamic behavior sequence characteristics, and calculating a Bhattacharyya coefficient between the thermal distribution histogram and a preset normal thermal distribution; Inputting the Hausdorff distance, the Euclidean distance and the Bhattacharyya coefficient into a normalization function to generate a local abnormality calculation result of the time segment feature; The normalization function is used to map abnormal indicators of different dimensions to a unified numerical range for weighted sum calculation.

6. The early warning analysis method based on intelligent vision according to claim 1 is characterized in that: The step of generating a graded warning signal by combining the correlation weights of the dynamic behavior sequence features and the abnormal behavior features includes: Determining the danger level of the abnormal behavior feature according to the duration and spatial coverage of the abnormal behavior feature; Matching the corresponding warning level from a preset warning strategy library based on the danger level, and generating the graded warning signal; Binding the graded warning signal with the target object identification information in the dynamic behavior sequence feature to generate a warning event record; The warning strategy library contains warning trigger conditions, warning information push objects and security equipment linkage rules corresponding to different danger levels; The security response mechanism includes sending an early warning notification to a designated terminal, activating an audible and visual alarm device, and controlling an intelligent access control system to lock the entrance and exit of the monitored area.

7. The early warning analysis method based on intelligent vision according to claim 6 is characterized in that: The step of binding the graded warning signal with the target object identification information in the dynamic behavior sequence feature to generate a warning event record includes: Extracting facial features, clothing color features and body contour features of the target object from the dynamic behavior sequence features to generate identification information of the target object; The timestamp of the graded warning signal, the target object identification information and the location code of the monitoring area are stored in a warning event database; Generate a statistical report on abnormal behavior in the monitoring area based on the historical records in the warning event database; The abnormal behavior statistical report includes the time distribution of abnormal events, high-frequency triggering areas and the behavior pattern classification of target objects.

8. The early warning analysis method based on intelligent vision according to claim 1 is characterized in that: The dynamically adjusting the acquisition parameters of the visual acquisition device and the coverage of the monitoring area according to the spatial distribution and time duration of the abnormal behavior characteristics includes: If the spatial distribution of the abnormal behavior characteristics is concentrated in the first sub-area of ​​the monitoring area, controlling the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area; If the duration of the abnormal behavior feature exceeds a preset threshold, the visual acquisition device is controlled to expand the shooting angle range of the monitoring area, and the backup camera is enabled to cover the blind spot; Reacquire the video stream of the monitoring area based on the adjusted acquisition parameters, and update the analysis result of the dynamic behavior sequence characteristics; The adjustment of the acquisition parameters includes real-time optimization of exposure time, focal length and white balance parameters to adapt to the abnormality detection requirements under different lighting conditions; The step of controlling the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area includes: Identify the bounding box coordinates of all active objects in the first sub-area by using a target detection algorithm, and calculate the center point position and coverage radius of the first sub-area according to the bounding box coordinates; Generate a pan / tilt control instruction based on the center point position and the coverage radius, adjust the pan / tilt rotation angle and zoom ratio of the visual acquisition device, and control the visual acquisition device to increase the image acquisition frame rate and resolution of the first sub-area under the adjusted pan / tilt rotation angle and zoom ratio, perform local high-definition video stream acquisition on the first sub-area, and splice it with the video streams of other areas; The stitching process includes performing feature point matching and perspective transformation on the images in the overlapping areas to generate a seamlessly integrated panoramic monitoring picture.

9. The early warning analysis method based on intelligent vision according to claim 1 is characterized in that: The method further comprises: Performing background modeling on the multimodal video stream to separate static background features and dynamic foreground features of the monitoring area; Constructing a three-dimensional spatial topological map of the monitoring area based on the static background features, and marking the location information of key security nodes; Overlaying and analyzing the dynamic foreground features with the three-dimensional space topology map to determine the real-time distance and movement trend of the target object relative to the key security node; If the real-time distance is less than the preset safety distance threshold and the movement trend points to the key security node, the detection sensitivity of the abnormal behavior feature is improved; wherein the key security node includes the safe location, power control box and emergency exit sign.

10. A server, characterized in that: The server includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the intelligent vision-based early warning analysis method described in any one of claims 1 to 9 above.

Images