Infrared thermal imaging abnormal scene monitoring method and related device based on multi-modal large model

Abstract

The present invention provides an infrared thermal imaging abnormal scene monitoring method and related devices based on a multimodal large model, and relates to the field of security information technology. This application solves the problem that traditional machine learning technology is difficult to obtain high accuracy in infrared thermal imaging abnormal scene monitoring by introducing the state probability transfer thinking chain technology based on the neural regulation conduction mechanism in the large model prompt tuning training. Compared with the existing technology, this solution can more accurately capture and predict the abnormal behavior of the crowd, and improve the accuracy and real-time performance of infrared thermal imaging abnormal scene monitoring.

Description

Technical Field

[0001] The present application relates to the field of security information technology, and in particular to an infrared thermal imaging abnormal scene monitoring method and related devices based on a multi-modal large model. Background Art

[0002] Abnormal scene monitoring in crowd activity places is an important development direction in the field of security information technology. Among many monitoring technologies, infrared thermal imaging technology is favored due to its unique advantages. This technology has the advantages of all-weather non-sensing monitoring, no electromagnetic interference, strong detection ability, long working distance and wide coverage, and can perform real-time monitoring in complex environments such as night and poor visibility. More importantly, infrared thermal imaging can not only detect abnormal behavior of the crowd, but also reflect the emotional changes of the crowd by capturing changes in body surface temperature, which has irreplaceable advantages in security monitoring.

[0003] However, there are still many challenges in the recognition of abnormal scenes in current infrared thermal imaging security monitoring. Traditional manual recognition methods are affected by subjective factors such as the monitoring personnel's attention, fatigue and recognition ability, making it difficult to achieve continuous, real-time and accurate recognition. Although with the development of artificial intelligence technology, machine learning technologies such as support vector machines, BP neural networks, and deep learning are increasingly being used in infrared thermal imaging security monitoring, these technologies still face many limitations.

[0004] First, the different characteristics of various places and the complexity of background environmental factors such as temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration pose a huge challenge to the accurate identification of abnormal scenes. Secondly, the complexity of the number, distribution, and behavioral and psychological changes of people in abnormal scenes in infrared thermal imaging further increases the difficulty of identification. These factors make it difficult for traditional machine learning technology to achieve high-accuracy recognition results in infrared thermal imaging abnormal scene monitoring.

[0005] In recent years, the emergence of multimodal large model technology has provided new possibilities for solving the above problems. This technology has the advantage of aligning various types of information, text semantics, and monitoring videos to achieve fusion recognition, which can well solve the difficulties in infrared thermal imaging abnormal scene monitoring. However, the application of multimodal large model technology to abnormal scene monitoring of infrared thermal imaging still faces some inherent challenges:

[0006] First, the effectiveness of large multimodal models depends heavily on the quantity and quality of training samples. However, in practical applications, it is often difficult to obtain a large number of high-quality abnormal scenario samples.

[0007] Secondly, large multimodal models have the "hallucination" problem, that is, they may produce outputs that do not match the actual situation, which is unacceptable in high-risk areas such as security monitoring.

[0008] Thirdly, large multimodal models have poor interpretability, and it is difficult to clearly explain their decision-making process to users, which is a significant flaw in the field of security monitoring that requires a high degree of transparency and traceability.

[0009] Finally, the computational overhead of large multimodal models is huge, which may affect the efficiency and response speed of real-time monitoring.

[0010] In view of the above problems, the existing technology needs to be improved urgently. Summary of the invention

[0011] The purpose of this application is to provide an infrared thermal imaging abnormal scene monitoring method and device based on a multimodal large model, which has the advantages of improving the accuracy of abnormal scene recognition, reducing the "hallucination" problem, enhancing interpretability and improving real-time monitoring efficiency.

[0012] In the first aspect, the present application provides an infrared thermal imaging abnormal scene monitoring method based on a multi-modal large model, which adopts the following technical solutions:

[0013] A method for monitoring abnormal infrared thermal imaging scenes based on a multi-modal large model, comprising:

[0014] Step 1: Monitoring data collection; prepare a site information table for the monitoring sites within the infrared thermal imaging field of view, including six items of information: site number, site purpose, site area, site length, site width, and site height; set up environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection devices at the monitoring sites, collect six environmental background real-time data streams, and collect video data streams through infrared thermal imaging equipment, and temporarily store them in the monitoring computer;

[0015] Step 2: Optimize sample collection; design a real crowd directing plan that can reflect the characteristics of various abnormal scenarios based on abnormal scenario cases and possible abnormal scenarios that have occurred in different places and similar places; use a real crowd to practice and present abnormal scenarios based on the above directing plan, and temporarily store the above crowd directing plan information and the monitoring data collected in step 1 during the practice and presentation in the monitoring computer;

[0016] Step 3: Generate a hybrid data set: Align the monitoring data collected in step 1 and the tuning sample data collected in step 2 in time to generate a hybrid data set for large model tuning training and actual scene monitoring;

[0017] Step 4: Tuning the large model prompts; using automatic prompt engineering technology, automatically generate prompt tuning instructions under the guidance of the thinking chain, and perform tuning training on the large model based on the tuning sample data generated in step 3; randomly select three quarters of the sample data as the training data set, and use the remaining one quarter of the sample data as the test data; when the abnormal scenario monitoring and identification index reaches the expected index, proceed to step 5, otherwise modify and improve the thinking chain prompts first. If the expected index still cannot be achieved, proceed to step 2, improve the crowd directing plan and collect new tuning samples until the expected index requirements are met;

[0018] Step 5: Abnormal scenario identification: The multimodal large model that has been tuned and trained to achieve the expected indicators is applied to actual scenario monitoring. When an abnormal scenario occurs, the large model automatically outputs the venue number, abnormal scenario category, abnormal scenario description, and crowd size information.

[0019] Optionally, in step 2, the designed real crowd directing plan includes the venue number, abnormal scenario category, abnormal scenario description, crowd number, crowd distribution and flow plan, crowd psychological and behavioral characteristics performance plan, and is practiced and presented according to the number of people such as single, double, 3-5, 20 or more people and the clothing that matches the venue temperature, humidity, wind speed, and light intensity.

[0020] Optionally, in step 4, the state probability transfer thinking chain technology based on the neural regulation conduction mechanism is introduced into the large model prompt tuning training to provide a thinking chain prompt for the dynamic correlation characteristics between the psychological conditions of the crowd and their behavioral performance.

[0021] Optionally, in step 4, the thought chain prompt is based on a state probability transition parameter model, specifically including the following state transition probabilities:

[0022] The probability of two people approaching (A1) to facing (A2) is 0.1631;

[0023] The probability of face orientation (A2) to body surface temperature rise (A3) is 0.1303;

[0024] The probability of body surface temperature rise (A3) to body posture change (A4) is 0.3388;

[0025] The probability of body posture change (A4) to increase in the amplitude and speed of interactive movements (A5) is 0.3677;

[0026] The probability of the amplitude and speed of the interactive action increasing from (A5) to a brawl (A6) is 0.7607.

[0027] Optionally, in step 5, the abnormal scenario recognition module can monitor and identify abnormal scenarios such as fire, building collapse, fights, illegal crossing of railings and walls in real time, and identify students' abnormal tendencies in combination with specific personal historical information.

[0028] Optionally, the infrared thermal imaging device has a sensitivity of less than 50mK, a frame rate of not less than 30Hz, and a resolution of not less than 640×512.

[0029] In a second aspect, the present application provides an infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model, comprising:

[0030] Monitoring data collection module, used to collect site information table, environmental background real-time data stream and infrared thermal imaging video data stream;

[0031] The tuning sample collection module is used to assist the real crowd director in designing the plan, monitor and manage the presentation of the real crowd rehearsal scene, and collect the large model prompt tuning sample data;

[0032] A mixed data set generation module is used to generate a mixed data set by aligning the monitoring data and the tuning sample data in time;

[0033] The large model prompt tuning module is used to automatically generate prompt tuning instructions under the guidance of the thinking chain and train the tuning sample data;

[0034] The abnormal scenario recognition module is used to monitor abnormal scenarios occurring in the venue in real time and identify their types based on the trained tuning model.

[0035] Optionally, the device also includes environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection equipment for collecting real-time data streams of environmental background.

[0036] Optionally, the device also includes an infrared thermal imaging device with a sensitivity of less than 50mK, a frame rate of not less than 30Hz, and a resolution of not less than 640×512.

[0037] Optionally, the device also includes a monitoring computer for temporarily storing the collected site information table, environmental background real-time data stream, infrared thermal imaging video data stream and tuning sample data.

[0038] This application solves the problem that traditional machine learning technology is difficult to obtain high accuracy in infrared thermal imaging abnormal scene monitoring by introducing the state probability transfer thinking chain technology based on the neural regulation conduction mechanism in the large model prompt tuning training. Compared with the existing technology, this solution can more accurately capture and predict the abnormal behavior of the crowd, and improve the accuracy and real-time performance of infrared thermal imaging abnormal scene monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the first embodiment of the infrared thermal imaging abnormal scene monitoring method based on a multi-modal large model of the present application;

[0040] Figure 2 It is a structural block diagram of the first embodiment of the infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The present application embodiment provides a method for monitoring abnormal scenes of infrared thermal imaging based on a multi-modal large model, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the infrared thermal imaging abnormal scene monitoring method based on a multi-modal large model of the present application.

[0043] In this embodiment, the infrared thermal imaging abnormal scene monitoring method based on the multi-modal large model includes the following steps:

[0044] Step 1: Monitoring data collection; prepare a site information table for the monitoring sites within the field of view of infrared thermal imaging, including six items of information: site number, site purpose, site area, site length, site width, and site height; set up environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection devices at the monitoring sites to collect six items of environmental background real-time data streams, and collect video data streams through infrared thermal imaging equipment and temporarily store them in the monitoring computer.

[0045] It should be noted that, first of all, monitoring data collection. A site information table is compiled for the monitoring sites within the infrared thermal imaging field of view, including site number, site purpose, site area, site length, site width, site height and other information. Ambient temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection devices are set up at the monitoring sites to collect real-time data streams of the environmental background, and video data streams are collected through infrared thermal imaging equipment and temporarily stored in the monitoring computer.

[0046] Next, fine-tune sample collection. Based on abnormal scenario cases and possible abnormal scenarios that have occurred in different places and similar places, design a real crowd directing plan that can reflect the characteristics of various abnormal scenarios. Based on the above directing plan, use a real crowd to practice and present abnormal scenarios, and temporarily store the above crowd directing plan information and the monitoring data collected through monitoring data during the practice presentation in the monitoring computer.

[0047] Then, a hybrid dataset is generated. The monitoring data and the tuning sample data are aligned in time to generate a hybrid dataset for large model tuning training and actual scene monitoring.

[0048] The next step is to optimize the large model prompts. Automatic prompt engineering technology is used to automatically generate prompt tuning instructions under the guidance of the thinking chain, and the large model is trained based on the tuning sample data. Three quarters of the sample data are randomly selected as the training data set, and the remaining quarter is used as the test data. When the abnormal scenario monitoring and identification indicators reach the expected indicators, proceed to the next step. Otherwise, modify and improve the thinking chain prompts first. If the expected indicators are still not met, proceed to the tuning sample collection step, improve the crowd directing plan and collect new tuning samples until the expected indicators are met.

[0049] Finally, abnormal scenarios are identified. The multimodal large model that has been tuned and trained to achieve the expected indicators is applied to actual scenario monitoring. When an abnormal scenario occurs, the large model automatically outputs the venue number, abnormal scenario category, abnormal scenario description, and crowd quantity information.

[0050] This method processes infrared thermal imaging data through multimodal large model technology, which can effectively solve the limitations of existing technologies in abnormal scenario monitoring. Specifically, comprehensive data support is obtained through monitoring data collection and tuning sample collection; the training effect of the model is improved through hybrid data set generation and large model prompt tuning; and real-time monitoring and identification of abnormal scenarios in actual scenarios is achieved through abnormal scenario identification. As a result, abnormal scenarios can be accurately monitored and identified in complex and changing environments, improving the accuracy and reliability of security monitoring.

[0051] Compared with the existing technology, the proposed method has significant advantages in data collection, model training and abnormal scenario recognition. By introducing multimodal large model technology, it can better integrate information from multiple data sources and improve the accuracy of abnormal scenario recognition. At the same time, the automatic prompt engineering technology and thinking chain guidance are used to improve the efficiency and effect of model tuning. Combined with specific parameters and examples, the effectiveness and feasibility of this method in practical applications are further verified.

[0052] In summary, this method can effectively solve the technical problems in abnormal scene monitoring of infrared thermal imaging and improve the accuracy and real-time performance of monitoring.

[0053] Step 2: Optimize sample collection; design a real crowd directing plan that can reflect the characteristics of various abnormal scenarios based on abnormal scenario cases and possible abnormal scenarios that have occurred in different places and similar places; use a real crowd to rehearse and present abnormal scenarios based on the above directing plan, and temporarily store the above crowd directing plan information and the monitoring data collected in step 1 during the rehearsal and presentation in the monitoring computer.

[0054] It should be noted that in step 2, the real crowd directing plan designed includes the venue number, abnormal scenario category, abnormal scenario description, crowd number, crowd distribution and flow plan, crowd psychological and behavioral characteristics performance plan, and is practiced and presented according to the number of people such as single, double, 3-5, 20 or more, and clothing that matches the temperature, humidity, wind speed, and light intensity of the venue.

[0055] The technical solution of this embodiment solves the limitation of abnormal scene monitoring in the prior art that relies on manual recognition, overcomes the influence of monitoring personnel's attention, fatigue and recognition ability, and realizes continuous, real-time and accurate abnormal scene recognition. By designing a real crowd directing scheme, it is possible to simulate various abnormal scenes and generate high-quality tuning sample data, providing a reliable data basis for the tuning training of large models.

[0056] Specifically, the design of the real crowd directing plan includes the following aspects: First, determine the number of monitoring sites and the category of abnormal scenarios, and describe the specific circumstances of the abnormal scenarios in detail. Secondly, according to different abnormal scenarios, design the number of people and their distribution and flow plans. For example, the performance of single, double, 3-5, 20 and more people in different scenes. Thirdly, design the psychological and behavioral characteristics of the crowd to simulate the behavioral characteristics under different psychological states. Finally, according to the environmental factors such as the temperature, humidity, wind speed, and light intensity of the venue, design matching clothing to ensure the authenticity of the drill.

[0057] Through the implementation of the above solution, a rich amount of tuning sample data can be generated, covering the behavioral characteristics of people in different abnormal scenarios. After time alignment, these data form a mixed data set for tuning and training of the large model. As a result, the trained multimodal large model can accurately identify abnormal scenarios in actual scenarios, improving the accuracy and reliability of monitoring.

[0058] Furthermore, by designing a real crowd directing scheme, it is possible to simulate various complex abnormal scenarios and generate high-quality tuning sample data, providing a reliable data basis for tuning training of large models. Compared with the prior art, the scheme of this application has significant advantages in the accuracy and real-time performance of abnormal scenario recognition, solves the shortcomings of the prior art that relies on manual recognition, and improves the overall performance of the infrared thermal imaging monitoring system.

[0059] Step 3: Generate a hybrid data set: The monitoring data collected in step 1 and the tuning sample data collected in step 2 are aligned in time to generate a hybrid data set for large model tuning training and actual scene monitoring.

[0060] It should be noted that the state probability transfer thinking chain technology based on the neural regulation conduction mechanism is introduced in the large model prompt tuning training, which gives a thinking chain prompt for the dynamic correlation characteristics between the psychological state of the crowd and its behavioral performance. Specifically, this technology describes the dynamic correlation characteristics between the psychological state and behavioral performance of the crowd through a state transition probability model. For example, the probability of two people approaching to face orientation is 0.1631, the probability of face orientation to body surface temperature rise is 0.1303, the probability of body surface temperature rise to body posture change is 0.3388, the probability of body posture change to increase in the amplitude and speed of interactive movements is 0.3677, and the probability of increase in the amplitude and speed of interactive movements to fighting is 0.7607. Therefore, through these state transition probabilities, the abnormal behavior of the crowd can be captured and predicted more accurately in the large model prompt tuning training.

[0061] It should be noted that in the large-model prompt tuning training, the introduction of the state probability transfer thinking chain technology is mainly based on the neural regulation conduction mechanism. The implementation methods of this technical feature include: 1. Establishment of the state probability transfer model: By analyzing the monitoring data of a large number of abnormal scenarios, a state transfer probability model between psychological conditions and behavioral performance is established. 2. Capture of dynamic correlation features: Using the state transfer probability model, in the large-model prompt tuning training, the correlation features between the psychological conditions and behavioral performance of the crowd are dynamically captured. 3. Generation of prompt tuning instructions: Based on the state probability transfer model, under the guidance of the thinking chain, prompt tuning instructions are automatically generated to train the tuning sample data.

[0062] This embodiment introduces the state probability transfer thinking chain technology based on the neural regulation conduction mechanism into the large model prompt tuning training, solving the problem that traditional machine learning technology is difficult to obtain high accuracy in infrared thermal imaging abnormal scene monitoring. Compared with the existing technology, this solution can more accurately capture and predict the abnormal behavior of the crowd, and improve the accuracy and real-time performance of infrared thermal imaging abnormal scene monitoring.

[0063] Step 4: Tuning the large model prompts; using automatic prompt engineering technology, automatically generate prompt tuning instructions under the guidance of the thinking chain, and perform tuning training on the large model based on the tuning sample data generated in step 3; randomly select three quarters of the sample data as the training data set, and use the remaining one quarter of the sample data as the test data; when the abnormal scenario monitoring and identification index reaches the expected index, proceed to step 5, otherwise modify and improve the thinking chain prompt first. If the expected index still cannot be achieved, go to step 2, improve the crowd directing plan and collect new tuning samples until the expected index requirements are met.

[0064] It should be noted that in step 4, the thought chain prompt is based on the state probability transfer parameter model, specifically including the following state transfer probabilities: the probability of two-person approach (A1) to face orientation (A2) is 0.1631; the probability of face orientation (A2) to body surface temperature rise (A3) is 0.1303; the probability of body surface temperature rise (A3) to body posture change (A4) is 0.3388; the probability of body posture change (A4) to increased amplitude and speed of interactive action (A5) is 0.3677; the probability of increased amplitude and speed of interactive action (A5) to fighting (A6) is 0.7607.

[0065] In specific implementation, the thought chain prompt in this embodiment is based on a state probability transition parameter model, specifically including the following state transition probabilities:

[0066] The probability of two people approaching each other to the point of facing each other is 0.1631; the probability of facing each other to the point of body surface temperature rising is 0.1303; the probability of body surface temperature rising to the point of body posture changing is 0.3388; the probability of body posture changing to the point of increased amplitude and speed of interactive movements is 0.3677; the probability of increased amplitude and speed of interactive movements to the point of fighting is 0.7607.

[0067] The technical solution of this embodiment aims to solve the problem of low accuracy in abnormal scene recognition due to the complex dynamic correlation characteristics of crowd psychology and behavior in infrared thermal imaging abnormal scene monitoring. By introducing the state probability transfer thinking chain technology based on the neural regulation conduction mechanism, it is possible to effectively capture the dynamic correlation characteristics between the psychological status and behavioral performance of the crowd, thereby improving the accuracy and real-time performance of abnormal scene recognition.

[0068] Specifically, the state probability transition parameter model simulates the behavioral changes of people in abnormal situations by setting the transition probabilities between different behavioral states. For example, when two people approach, they may turn to each other, and the probability of their faces facing each other is 0.1631; when their faces are facing each other, the probability of the body surface temperature rising is 0.1303; after the body surface temperature rises, the probability of the body posture changing is 0.3388; after the body posture changes, the probability of the amplitude and speed of the interactive action increasing is 0.3677; after the amplitude and speed of the interactive action increase, the probability of a fight is 0.7607. Through these probability parameters, a dynamic association model can be constructed to guide the large model to more accurately identify abnormal behaviors of people during tuning training.

[0069] Therefore, this embodiment enhances the ability of the large model to process complex behavior dynamic correlation features by introducing the state probability transfer thinking chain technology, and improves the accuracy and real-time performance of infrared thermal imaging abnormal scene monitoring. Compared with traditional methods, this embodiment can better adapt to changes in different places and background environments, and provide a more efficient and reliable abnormal scene monitoring solution.

[0070] Step 5: Abnormal scenario identification: The multimodal large model that has been tuned and trained to achieve the expected indicators is applied to actual scenario monitoring. When an abnormal scenario occurs, the large model automatically outputs the venue number, abnormal scenario category, abnormal scenario description, and crowd size information.

[0071] In the specific implementation, in step 5, the abnormal scenario recognition module can monitor and identify abnormal scenarios such as fire, building collapse, fights, illegal crossing of railings and walls in real time, and identify students' abnormal tendencies in combination with specific personal historical information.

[0072] It should be noted that the sensitivity of the infrared thermal imaging device is less than 50mK, the frame rate is not less than 30Hz, and the resolution is not less than 640×512.

[0073] It should be noted that the abnormal scenario recognition module can monitor and identify abnormal scenarios such as fire, building collapse, fights, illegal crossing of railings and walls in real time, and identify students' suicidal tendencies in combination with specific personal historical information.

[0074] In practical applications, the abnormal scenario recognition module can accurately identify and monitor various abnormal scenarios in real time by tuning and training the multimodal large model. By identifying scenes such as fire, building collapse, fighting, and illegal crossing of railings and walls, the sensitivity and accuracy of the monitoring device are effectively improved. The implementation of the abnormal scenario recognition module includes the following possible variants. First, the recognition accuracy of abnormal scenarios can be improved by introducing multi-sensor fusion technology and combining data from infrared thermal imaging equipment and other environmental detection equipment. Secondly, deep learning algorithms can be used to extract and classify features for different types of abnormal scenarios to improve the generalization ability of the model. Furthermore, by introducing an adaptive learning mechanism, the model parameters can be dynamically adjusted according to the actual application scenario to improve the adaptability and robustness of the system.

[0075] Through the above technical solution, this embodiment has obvious advantages in recognition accuracy, real-time and adaptability, and effectively solves the shortcomings of traditional monitoring methods.

[0076] This embodiment introduces the state probability transfer thinking chain technology based on the neural regulation conduction mechanism into the large model prompt tuning training, solving the problem that traditional machine learning technology is difficult to obtain high accuracy in infrared thermal imaging abnormal scene monitoring. Compared with the existing technology, this solution can more accurately capture and predict the abnormal behavior of the crowd, and improve the accuracy and real-time performance of infrared thermal imaging abnormal scene monitoring.

[0077] Reference Figure 2 , Figure 2 This is a structural block diagram of the first embodiment of the infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model of the present application.

[0078] like Figure 2 As shown, the infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model proposed in the embodiment of the present application includes:

[0079] Monitoring data collection module 10, used to collect site information table, environmental background real-time data stream and infrared thermal imaging video data stream;

[0080] The tuning sample collection module 20 is used to assist the real crowd director in designing the plan, monitor and manage the presentation of the real crowd rehearsal scene, and collect the large model prompt tuning sample data;

[0081] A hybrid data set generation module 30, used to generate a hybrid data set after aligning the monitoring data and the tuning sample data in time;

[0082] A large model prompt tuning module 40 is used to automatically generate prompt tuning instructions under the guidance of the thought chain and train the tuning sample data;

[0083] The abnormal scenario recognition module 50 is used to monitor abnormal scenarios occurring in the venue in real time and identify their types based on the trained tuning model.

[0084] It should be noted that the infrared thermal imaging abnormal scene monitoring device based on the multimodal large model includes environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection equipment, which is used to collect real-time data streams of environmental background.

[0085] The monitoring device can obtain the environmental background data of the monitoring site in real time through the detection equipment of environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration. These data include temperature, humidity, wind speed, light intensity, and carbon monoxide and carbon dioxide concentrations, ensuring comprehensive environmental monitoring of the monitoring site. These environmental data are combined with infrared thermal imaging video data streams to provide rich background information for abnormal scene recognition, which helps to improve the accuracy and reliability of recognition.

[0086] Specifically, the collection of real-time data streams of environmental background can reflect the immediate environmental conditions of the monitoring site, which is crucial for monitoring abnormal situations. For example, in fire monitoring, changes in ambient temperature can serve as an important early warning signal; when identifying fights, changes in wind speed and light intensity may also affect the effect of infrared thermal imaging. By acquiring these environmental data in real time, the monitoring device can more accurately analyze and identify abnormal situations.

[0087] Furthermore, the performance parameters of infrared thermal imaging equipment also play a key role in the identification of abnormal scenarios. The sensitivity of the equipment is less than 50mK, the frame rate is not less than 30Hz, and the resolution is not less than 640×512. These parameters ensure that the infrared thermal imaging equipment can efficiently capture subtle temperature changes and fast dynamic scenes, thereby improving the accuracy and real-time performance of monitoring.

[0088] As a result, the monitoring device can not only collect environmental background data in real time, but also obtain high-quality video data streams through high-performance infrared thermal imaging equipment. The combined use of these data enables the multimodal large model to more comprehensively understand and analyze the situation of the monitoring site, thereby more accurately identifying abnormal situations.

[0089] Compared with the existing technology, this embodiment significantly improves the accuracy and real-time performance of abnormal scene monitoring by introducing equipment for collecting real-time data streams of environmental background and high-performance infrared thermal imaging equipment. Traditional monitoring methods often rely on a single data source and are easily affected by environmental changes. This embodiment overcomes this limitation by integrating multimodal data, allowing the monitoring system to maintain efficient monitoring capabilities in complex and changing environments.

[0090] It is understandable that, in this embodiment, the apparatus further includes an infrared thermal imaging device, the sensitivity of which is less than 50 mK, the frame rate is not less than 30 Hz, and the resolution is not less than 640×512.

[0091] The infrared thermal imaging device is an important component of this embodiment. Its high sensitivity and high frame rate can ensure accurate capture of temperature changes and subtle movements in complex environments. A sensitivity of less than 50mK means that the device can detect extremely small temperature differences, which is critical in identifying abnormal scenarios. For example, when monitoring fires or abnormal human behavior, temperature changes are often the earliest warning signs. A frame rate of not less than 30Hz ensures the continuity and smoothness of the video stream, allowing the device to monitor changes in dynamic scenes in real time. A resolution of not less than 640×512 ensures image clarity, allowing the device to capture detailed information, thereby improving the accuracy of abnormal scenario identification.

[0092] Infrared thermal imaging equipment can use advanced detector materials and imaging algorithms to improve its sensitivity and resolution. For example, using vanadium oxide (VOx) microbolometers as detector materials can significantly improve the detection performance of the equipment. At the same time, combined with adaptive image enhancement algorithms, image quality can be further improved in complex environments. As a result, infrared thermal imaging equipment can provide high-quality data support in practical applications to ensure the accuracy and reliability of abnormal scenario monitoring.

[0093] This embodiment introduces high-sensitivity, high-frame-rate and high-resolution infrared thermal imaging equipment to solve the problem of insufficient sensitivity and unclear images of traditional infrared thermal imaging equipment in complex monitoring environments. As a result, the accuracy and real-time performance of abnormal scene monitoring are improved, and potential safety hazards can be more effectively identified and warned.

[0094] In a specific implementation, this embodiment further proposes that the device also includes a monitoring computer for temporarily storing the collected location information table, environmental background real-time data stream, infrared thermal imaging video data stream and tuning sample data.

[0095] In this embodiment, the monitoring computer plays a key role in the entire infrared thermal imaging abnormal scene monitoring device. It can not only temporarily store the collected site information table, environmental background real-time data stream, infrared thermal imaging video data stream, but also temporarily store the tuning sample data. In this way, the monitoring computer can effectively manage and process a large amount of monitoring data, and provide data support for subsequent mixed data set generation and large model prompt tuning.

[0096] There are many ways to implement the monitoring computer. For example, it can be implemented using a high-performance server or a customized embedded computing device. Specifically, the monitoring computer should have sufficient storage capacity and computing power to cope with the storage and processing needs of large-scale data. In addition, the monitoring computer must be equipped with an efficient data transmission interface to ensure real-time transmission and processing of data. As a preferred embodiment, the monitoring computer can be connected to other modules through a high-speed network to achieve rapid transmission and processing of data.

[0097] This embodiment effectively solves the problem of data management and processing in infrared thermal imaging abnormal scene monitoring by introducing a monitoring computer. Compared with the prior art, the device of this embodiment can more efficiently process and store a large amount of monitoring data, thereby improving the accuracy and real-time performance of abnormal scene identification. Therefore, this embodiment has significant technical advantages in the field of infrared thermal imaging abnormal scene monitoring.

[0098] The infrared thermal imaging abnormal scene monitoring device in this embodiment also includes a monitoring computer for temporarily storing the collected location information table, environmental background real-time data stream, infrared thermal imaging video data stream and tuning sample data.

[0099] The device includes a monitoring data acquisition module, which is used to collect site information tables, real-time data streams of environmental backgrounds, and infrared thermal imaging video data streams; a tuning sample acquisition module, which is used to assist in the design of real-people director programs, monitor and manage the presentation of real-people rehearsal scenes, and collect large-model prompt tuning sample data; a mixed data set generation module, which is used to generate a mixed data set after aligning monitoring data and tuning sample data in time; a large-model prompt tuning module, which is used to automatically generate prompt tuning instructions under the guidance of a thinking chain and train the tuning sample data; an abnormal scenario recognition module, which is used to monitor abnormal scenarios occurring in the site in real time and identify their types based on a trained tuning model.

[0100] The newly added monitoring computer is used to temporarily store the collected site information table, environmental background real-time data stream, infrared thermal imaging video data stream, and tuning sample data. The monitoring computer can use a high-performance computer equipped with a large-capacity storage device and efficient data processing capabilities to ensure fast storage and reading of data. Furthermore, the monitoring computer can be equipped with dedicated data management software to classify, organize and analyze the collected data for subsequent data processing and model training.

[0101] By introducing a monitoring computer, the device achieves efficient management and temporary storage of multiple data, ensures the integrity and consistency of the data, and improves the efficiency and accuracy of data processing. Compared with the prior art, the infrared thermal imaging abnormal scene monitoring device proposed in this embodiment can better adapt to complex monitoring environments and provide more accurate abnormal scene recognition results.

[0102] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of this embodiment. In specific applications, technicians in this field can make settings as needed, and this embodiment does not limit this.

[0103] This embodiment introduces the state probability transfer thinking chain technology based on the neural regulation conduction mechanism into the large model prompt tuning training, solving the problem that traditional machine learning technology is difficult to obtain high accuracy in infrared thermal imaging abnormal scene monitoring. Compared with the existing technology, this solution can more accurately capture and predict the abnormal behavior of the crowd, and improve the accuracy and real-time performance of infrared thermal imaging abnormal scene monitoring.

[0104] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this embodiment. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the scheme of this embodiment, and no limitation is made here.

[0105] In addition, for technical details not fully described in this embodiment, please refer to the infrared thermal imaging abnormal scene monitoring method provided in any embodiment of this embodiment, and will not be repeated here.

[0106] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0107] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application. The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structures or equivalent process changes made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are similarly included in the patent protection scope of the present application.

Claims

1. A method for monitoring abnormal situations using infrared thermal imaging based on a multi-modal large model, characterized in that: include: Step 1: Monitoring data collection; Prepare a site information table for the monitoring sites within the infrared thermal imaging field of view, including six items of information: site number, site purpose, site area, site length, site width, and site height; set up environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection devices at the monitoring sites to collect six environmental background real-time data streams, and collect video data streams through infrared thermal imaging equipment and temporarily store them in the monitoring computer; Step 2: Optimize sample collection; design a real crowd directing plan that can reflect the characteristics of various abnormal scenarios based on abnormal scenario cases and possible abnormal scenarios that have occurred in different places and similar places; use a real crowd to practice and present abnormal scenarios based on the above directing plan, and temporarily store the above crowd directing plan information and the monitoring data collected in step 1 during the practice and presentation in the monitoring computer; Step 3: Generate a hybrid data set: Align the monitoring data collected in step 1 and the tuning sample data collected in step 2 in time to generate a hybrid data set for large model tuning training and actual scene monitoring; Step 4: Tuning the large model prompts; using automatic prompt engineering technology, automatically generate prompt tuning instructions under the guidance of the thinking chain, and perform tuning training on the large model based on the tuning sample data generated in step 3; randomly select three quarters of the sample data as the training data set, and use the remaining one quarter of the sample data as the test data; when the abnormal scenario monitoring and identification index reaches the expected index, proceed to step 5, otherwise modify and improve the thinking chain prompts first. If the expected index still cannot be achieved, proceed to step 2, improve the crowd directing plan and collect new tuning samples until the expected index requirements are met; Step 5: Abnormal scenario identification: The multimodal large model that has been tuned and trained to achieve the expected indicators is applied to actual scenario monitoring. When an abnormal scenario occurs, the large model automatically outputs the venue number, abnormal scenario category, abnormal scenario description, and crowd size information.

2. The infrared thermal imaging abnormal scene monitoring method based on a multi-modal large model according to claim 1 is characterized in that: In step 2, the designed real crowd directing plan includes the venue number, abnormal scenario category, abnormal scenario description, crowd number, crowd distribution and flow plan, crowd psychological and behavioral characteristics performance plan, and is practiced and presented according to the number of people such as single, double, 3-5, 20 and above, and the clothing that matches the temperature, humidity, wind speed, and light intensity of the venue.

3. The infrared thermal imaging abnormal scene monitoring method based on a multi-modal large model according to claim 1 is characterized in that: In step 4, the state probability transfer thinking chain technology based on the neural regulation conduction mechanism is introduced into the large model prompt tuning training, which provides thinking chain prompts for the dynamic correlation characteristics between people's psychological conditions and their behavioral performance.

4. The infrared thermal imaging abnormal scene monitoring method based on a multi-modal large model according to claim 1 is characterized in that: In step 4, the thought chain prompt is based on a state probability transition parameter model, specifically including the following state transition probabilities: The probability of two people approaching (A1) to facing (A2) is 0.1631; The probability of face orientation (A2) to body surface temperature rise (A3) is 0.1303; The probability of body surface temperature rise (A3) to body posture change (A4) is 0.3388; The probability of body posture change (A4) to increase in the amplitude and speed of interactive movements (A5) is 0.3677; The probability of the amplitude and speed of the interactive action increasing from (A5) to a brawl (A6) is 0.7607.

5. The infrared thermal imaging abnormal scene monitoring method based on multi-modal large model according to claim 1 is characterized in that: In step 5, the abnormal scenario recognition module can monitor and identify abnormal scenarios such as fire, building collapse, fighting, illegal crossing of railings and walls in real time, and identify students' abnormal tendencies in combination with specific personal historical information.

6. The infrared thermal imaging abnormal scene monitoring method based on multi-modal large model according to claim 1 is characterized in that: The infrared thermal imaging device has a sensitivity of less than 50mK, a frame rate of not less than 30Hz, and a resolution of not less than 640×512.

7. An infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model, characterized in that: include: Monitoring data collection module, used to collect site information table, environmental background real-time data stream and infrared thermal imaging video data stream; The tuning sample collection module is used to assist the real crowd director in designing the plan, monitor and manage the presentation of the real crowd rehearsal scene, and collect the large model prompt tuning sample data; A mixed data set generation module is used to generate a mixed data set by aligning the monitoring data and the tuning sample data in time; The large model prompt tuning module is used to automatically generate prompt tuning instructions under the guidance of the thinking chain, and perform tuning training on the large model based on the generated tuning sample data; The abnormal scenario recognition module is used to monitor abnormal scenarios occurring in the venue in real time and identify their types based on the trained tuning model.

8. The infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model according to claim 7 is characterized in that: The device also includes environmental temperature, humidity, wind speed, light intensity, carbon monoxide concentration, and carbon dioxide concentration detection equipment for collecting real-time data streams of environmental background.

9. The infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model according to claim 7 is characterized in that: The device also includes an infrared thermal imaging device with a sensitivity of less than 50mK, a frame rate of not less than 30Hz, and a resolution of not less than 640×512.

10. The infrared thermal imaging abnormal scene monitoring device based on a multi-modal large model according to claim 7 is characterized in that: The device also includes a monitoring computer for temporarily storing the collected site information table, the environmental background real-time data stream, the infrared thermal imaging video data stream and the tuning sample data.

Images