Alarm cloud Internet of Things equipment three-dimensional scene building method and system based on digital twinning

Abstract

The invention discloses a digital twinning-based police cloud Internet of Things equipment three-dimensional scene building method and system, and belongs to the technical field of digital twinning, and the method comprises the steps: laying sensing equipment in a target region; establishing a three-dimensional entity model of the target area in the cloud server, laying a corresponding virtual model in the entity model based on the installation information of the sensing equipment, and accessing data from the sensing equipment into the virtual model to generate a digital twin model; the first troubleshooting unit analyzes the scene video and the environment data collected by the sensing equipment in real time, and if an environment risk is detected, a disposal scheme is generated; the second troubleshooting unit processes the scene video into a multi-section video, analyzes the multi-section video in combination with environmental data, and determines suspicious personnel and risk degree of the target area; and screening suspicious persons whose risk degrees are greater than a first threshold value as high-risk risks. According to the invention, safety management personnel can pay attention to and deal with high risks in time, and the emergency response speed and accuracy are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of digital twin technology, and specifically relates to a method and system for building a three-dimensional scene of police cloud Internet of Things equipment based on digital twins. Background Art

[0002] In the field of police cloud IoT equipment, digital twin technology can be used to associate physical entities such as police equipment and IoT sensors with digital models, which can achieve remote monitoring and danger warning. In addition, digital twin technology also has the characteristics of high precision, scalability and visualization, and can support large-scale and complex scene 3D modeling and simulation, providing strong technical support for the management and decision-making of police cloud IoT equipment.

[0003] Regarding the application of digital twins, in the prior art, for example, a Chinese patent document with publication number CN116187105A discloses a fire evacuation planning method and system based on digital twin technology. This method obtains the current burning range according to the fire starting position and camera position in the monitoring screen information, and inputs the current burning range into the building digital twin model. The burning range expansion speed is predicted according to the burning range change per unit time, and the burning spread trend is simulated in the building digital twin model. Then, according to the current position information, escape speed information and protection level information of the personnel, the building digital twin model is used for simulation. Evacuation planning and guidance are carried out according to the simulation results, so that the escape route of each person is more efficient and reasonable.

[0004] However, the above-mentioned digital twin technology is only used in the field of fire escape and has relatively simple functions. For densely populated public places, in addition to considering environmental hazards such as fire, the human risks of suspicious persons must also be considered. Therefore, it is necessary to further expand the application scope of digital twin technology in the field of police cloud IoT equipment, and include the monitoring, early warning and response to the human risks of suspicious persons. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a method and system for building a three-dimensional scene of police cloud IoT devices based on digital twins to solve the problems existing in the prior art.

[0006] In order to achieve the above-mentioned purpose of the invention, the present invention proposes a method for building a three-dimensional scene of police cloud IoT equipment based on digital twins, comprising: Deploy sensing equipment in the target area, wherein the sensing equipment includes a monitoring camera and an environmental sensor, wherein the monitoring camera is used to record scene video, and the environmental sensor is used to detect environmental data; Establishing a three-dimensional physical model of the target area in a cloud server, arranging a corresponding virtual model in the physical model based on the installation information of the sensing device, connecting the data from the sensing device to the corresponding virtual model, and generating a digital twin model; The first troubleshooting unit analyzes the scene video and the environmental data in real time, and if an environmental risk is detected, generates a disposal plan and displays it in the digital twin model; After the second screening unit processes the scene video into a multi-segment video, the multi-segment video is analyzed in combination with the environmental data to determine the suspicious persons and risk level in the target area; The suspicious persons whose risk level is greater than a first threshold are screened as high-risk persons, and a suspicious risk alarm pointing to the high-risk person is generated in the digital twin model.

[0007] Furthermore, real-time analysis of the scene video and the environmental data includes the following steps: The scene video to be analyzed is defined as the video to be analyzed, the video to be analyzed is extracted into multiple frame images based on a standard interval, the frame image at the current time in the video to be analyzed is located, if a dangerous marker is detected in the frame image, the corresponding environmental risk is determined according to the dangerous marker, and the disposal plan is generated; The environmental sensors include temperature sensors, smoke sensors and current sensors, and the environmental data include temperature data, smoke data and current data. If the environmental data exceeds the corresponding set threshold when the hazardous markers are not detected, it is determined that the environmental risk exists and the disposal plan is generated.

[0008] Furthermore, processing the scene video into the multi-segment video comprises the following steps: The multi-segment video includes a first segment, a second segment and a third segment, the frame images are arranged in chronological order, the change region between adjacent frame images is extracted based on the inter-frame difference method, and the number of changes in pixels included in the change region is counted; If the number of changes is greater than or equal to a dynamic threshold, the two frame images that generate the number of changes are marked as the first dynamic image. If the number of changes is less than the dynamic threshold, the size of the dynamic threshold is obtained. If the dynamic threshold is greater than or equal to a second threshold, the two frame images are marked as the second dynamic image. Otherwise, the frame images are marked as static images, and the first dynamic image is integrated into the first segment, the second dynamic image is integrated into the second segment, and the static image is integrated into the third segment.

[0009] Further, determining the dynamic threshold comprises the following steps: Acquire historical videos, process the historical videos into dynamic time series about the changing quantity, process the dynamic time series based on a sliding window method with a preset window size as a parameter, obtain historical window values ​​of multiple time windows, divide the recording time of the historical videos into multiple first time periods, and statistically analyze the historical window values ​​to determine an adjustment coefficient for each of the first time periods; Extract the mean of the number of changes in the second time period before the two frame images, determine the corresponding adjustment coefficient based on the first time period where the two frame images are located, correct the mean based on the adjustment coefficient, obtain a first value, set a basic value, correct the basic value based on the first value, and obtain the dynamic threshold.

[0010] Further, analyzing the multi-segment video includes the following steps: The first segment and the second segment in the multi-segment video are defined as a first analysis segment and a second analysis segment, respectively, and target tracking is performed on the first analysis segment based on a target detection algorithm. If a person walking is detected, the risk level is calculated based on the occurrence frequency, the activity trajectory entropy value, and the number of face matching failures; Obtain a second time period in which the second analysis section appears, intercept a video image of the scene video shot by other surveillance cameras in the second time period, define it as an auxiliary section, split the second analysis section and the auxiliary section into the frame images, and combine the frame images with the same shooting time into an image group, perform mapping analysis on the image group, and obtain the coordinate position of the same target in different frame images; The environmental sensor includes a beacon sensor, which obtains the return signal strength based on the employee's work card, combines the return signal strength into a signal group corresponding to the image group with time as a reference, and establishes an analysis model. The analysis model is based on the coordinate position of the image group and the return signal strength output corresponding to the signal group to output the risk of a suspicious person.

[0011] Furthermore, establishing the analysis model comprises the following steps: Collect historical data, the historical data includes the image group, the signal group and output labels, the output labels include no risk and risky, make numerical adjustments to the coordinate position in the image group and the return signal strength in the signal group to expand the historical data to obtain expanded data, establish the analysis model based on a neural network, and use the expanded data to train the analysis model; when the output result of the analysis model is that there is risk, use the probability value of the risk as the risk degree.

[0012] Furthermore, generating the treatment plan includes the following steps: The disposal plan includes a plurality of predetermined escape routes. If there are a plurality of said escape routes, the fire location is determined, a fire spreading direction is simulated and generated based on the fire location, and the best route is selected from the predetermined evacuation routes based on the spreading direction.

[0013] Furthermore, the danger signs include flames, smoke and accumulated water.

[0014] The present invention also provides a three-dimensional scene building system for police cloud IoT devices based on digital twins, which is used to implement the above-mentioned three-dimensional scene building method for police cloud IoT devices based on digital twins, and the system includes: A sensing device, wherein the sensing device includes a monitoring camera and an environmental sensor, wherein the monitoring camera is used to record scene video, and the environmental sensor is used to detect environmental data; A cloud server is used to establish a three-dimensional physical model of the target area, arrange a corresponding virtual model in the physical model based on the installation information of the sensing device, connect the data from the sensing device to the corresponding virtual model, and generate a digital twin model; A first troubleshooting unit is used to analyze the scene video and the environmental data in real time, and if an environmental risk is detected, generate a disposal plan and display it in the digital twin model; The second screening unit is used to process the scene video into a multi-segment video, analyze the multi-segment video in combination with the environmental data, determine the suspicious persons and risk levels in the target area, screen the suspicious persons whose risk levels are greater than a first threshold as high-risk risks, and generate a suspicious risk alarm pointing to the high-risk risks in the digital twin model.

[0015] Beneficial effects:

[0016] The present invention uses digital twin technology to establish a three-dimensional entity model of the target area in the cloud server, and corresponds to the actually deployed surveillance cameras and environmental sensors, forming a digital twin model that can reflect the actual scene status in real time. This model not only enhances the visualization and interactivity of the scene, but also enables users to view various data on the scene at any time. At the same time, the cloud server analyzes the scene video and environmental data in real time, and can promptly detect environmental risks including fires and generate corresponding disposal plans. By displaying the above content in the digital twin model, the emergency response speed and accuracy are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of the steps of a method for building a three-dimensional scene of police cloud IoT equipment based on digital twins of the present invention; Figure 2This is a structural schematic diagram of a three-dimensional scene building system for police cloud IoT devices based on digital twins in the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0020] like Figure 1 As shown, a method for building a three-dimensional scene of police cloud IoT equipment based on digital twins includes: S1: Deploy perception equipment in the target area. The perception equipment includes surveillance cameras and environmental sensors. The surveillance cameras are used to record scene videos, and the environmental sensors are used to detect environmental data.

[0021] S2: Establish a three-dimensional physical model of the target area in the cloud server, arrange the corresponding virtual model in the physical model based on the installation information of the sensing device, connect the data from the sensing device to the corresponding virtual model, and generate a digital twin model.

[0022] The target area includes various crowded areas, such as office buildings. The sensing devices include surveillance cameras and various types of environmental sensors. Environmental sensors include flame sensors, smoke sensors, temperature sensors, humidity sensors, current sensors, etc. Different types of sensors collect corresponding environmental data and upload them to the cloud server. A three-dimensional entity model corresponding to the actual scene is established in the cloud server, and a virtual model corresponding to the surveillance camera and environmental sensor is set in the three-dimensional entity model. After the cloud server receives the scene video and environmental data, it connects them to the corresponding virtual model to obtain a digital twin model corresponding to the actual scene. Users can click on the virtual model in the digital twin model to view various data on the scene in real time.

[0023] S3: The first investigation unit analyzes the scene video and environmental data in real time. If an environmental risk is detected, a disposal plan is generated and displayed in the digital twin model.

[0024] After the cloud server receives the scene video and environmental data, it captures the latest frame image in the scene video for analysis, and obtains the latest data in the environmental data for analysis to determine whether there are environmental risks. Environmental risks include fire, thick smoke, accumulated water, and overloaded electrical lines. For example, when it is determined that there is a flame in the current frame image, it is determined that there is a fire risk. Or when the smoke sensor detects that the concentration of particles larger than 2.5μm exceeds a certain threshold, it is determined that there is a thick smoke risk. If an environmental risk is detected, it is displayed in the digital twin model. At the same time, corresponding disposal plans are also generated, such as generating escape routes for fire risks, and generating corresponding solutions for accumulated water or overload short circuits to avoid the expansion of disasters.

[0025] S4: After the second screening unit processes the scene video into multi-segment videos, it analyzes the multi-segment videos in combination with the environmental data to determine suspicious persons and risk levels in the target area.

[0026] For areas such as office buildings, in addition to environmental risks, the human risks of outsiders also need to be considered. However, due to mask occlusion and other reasons, relying solely on face recognition technology and using the current few frames of images for judgment may not achieve the expected results. It is necessary to combine historical multi-frame images for comprehensive recognition of behavior trajectories. For this purpose, in order to increase the efficiency of recognition, the scene video is processed into multi-segment video before recognition. Specifically, the multi-segment video includes dynamic segments and static segments. After division, only the dynamic segments are analyzed and recognized, which can speed up the recognition efficiency. The specific identification of suspicious persons and the calculation method of risk degree will be introduced later.

[0027] S5: Screen suspicious persons whose risk level is greater than the first threshold as high-risk persons, and generate suspicious risk alerts pointing to high-risk persons in the digital twin model.

[0028] If it is determined that there is a suspicious person in the scene video and his risk level is greater than a first threshold, such as 0.8, the suspicious person is treated as a high risk and a risk alert is generated in the digital twin model for the specified area.

[0029] The present invention uses digital twin technology to establish a three-dimensional entity model of the target area in the cloud server, and corresponds to the actually deployed surveillance cameras and environmental sensors, forming a digital twin model that can reflect the actual scene status in real time. This model not only enhances the visualization and interactivity of the scene, but also enables users to view various data on the scene at any time. At the same time, the cloud server analyzes the scene video and environmental data in real time, and can promptly detect environmental risks including fires and generate corresponding disposal plans. By displaying the above content in the digital twin model, the emergency response speed and accuracy are effectively improved.

[0030] In order to improve the efficiency of identifying suspicious persons, the system of the present invention processes the scene video into multi-segment video, and only analyzes and identifies the dynamic segment, while combining environmental data and historical multi-frame images for comprehensive judgment. This method can more accurately identify suspicious persons and their risk level, reducing false positives and missed positives. For suspicious persons whose risk level is greater than the set threshold, the system can automatically screen them as high-risk risks and generate suspicious risk alarms pointing to the area in the digital twin model, which helps security managers to pay attention to and deal with high-risk risks in a timely manner, improving the level and efficiency of security management.

[0031] In this embodiment, real-time analysis of scene video and environmental data includes the following steps: The scene video to be analyzed is defined as the video to be analyzed, and the video to be analyzed is extracted into multiple frame images based on the standard interval. The frame image at the current time in the video to be analyzed is located. If a dangerous marker is detected in the frame image, the corresponding environmental risk is determined according to the dangerous marker, and a disposal plan is generated.

[0032] Hazard signs include flames, smoke, and standing water.

[0033] Specifically, the hazard markers in the current frame image can be identified based on convolutional neural network algorithms such as the Yolo algorithm and the U-net semantic segmentation algorithm. The hazard markers include flames, smoke, and accumulated water. Each marker has a preset corresponding disposal plan, such as an escape route for fire and smoke, and a corresponding water cleaning plan for accumulated water.

[0034] Environmental sensors include temperature sensors, smoke sensors and current sensors. Environmental data include temperature data, smoke data and current data. If the environmental data exceeds the corresponding set threshold when no hazardous markers are detected, it is determined that there is an environmental risk and a disposal plan is generated.

[0035] If no dangerous signs are identified in the scene video, but the temperature sensor detects that the temperature is too high, such as exceeding 80°C, it is determined that there is a fire. When the smoke sensor detects that the concentration of particles larger than 2.5μm exceeds 200μm / ㎡, it is determined that there is smoke. When the current sensor detects that the line current exceeds the specified value, such as 16A, it is determined that there is an overload risk. The area with overloaded current is cut off from power, and professional electricians are arranged to inspect and repair the lines to eliminate safety hazards and avoid serious accidents such as electrical fires caused by current overload.

[0036] In this embodiment, processing a scene video into a multi-segment video includes the following steps: The multi-segment video includes a first segment, a second segment and a third segment. The frame images are arranged in chronological order. The change area between adjacent frame images is extracted based on the inter-frame difference method, and the number of changes in pixels in the change area is counted.

[0037] The standard interval is set to 1s. Whenever the recorded scene video reaches 1min, the scene video is split into 60 frame images, and the 60 frame images are arranged in the order of recording time. Then, two frame images that are adjacent in time are compared in sequence based on the inter-frame difference method to obtain the change area between the frame images. The inter-frame difference method obtains the pixel difference by subtracting the pixel values ​​of the corresponding pixel points in the two frame images. If the pixel difference is greater than a predetermined value, the corresponding pixel point is determined as the changed pixel point, and the number of changes is determined by counting the number of changed pixel points determined after comparing the two frame images.

[0038] If the number of changes is greater than or equal to the dynamic threshold, the two frame images that generate the number of changes are marked as the first dynamic image. If the number of changes is less than the dynamic threshold, the size of the dynamic threshold is obtained. If the dynamic threshold is greater than or equal to the second threshold, the two frame images are marked as the second dynamic image. Otherwise, the frame images are marked as static images, and the first dynamic image is integrated into the first segment, the second dynamic image is integrated into the second segment, and the static image is integrated into the third segment.

[0039] The dynamic threshold is a value that is continuously adjusted according to the changes in the historical images. When the value of the dynamic threshold is small, it indicates that the historical images have changed little. For example, everyone is working in the office and there will be no major changes in the images for a long time. When the value of the dynamic threshold is large, it indicates that the historical images have changed greatly. For example, there are a large number of people moving around in the images. The specific method for determining the dynamic threshold will be introduced later.

[0040] If the number of changes is greater than the dynamic threshold, such as the resolution of the frame image is 1920*1080, the number of changes between the two frame images is determined to be 500000, the current dynamic threshold is 200000, and the number of changes is greater than the dynamic threshold, then the label representing the first dynamic image is set in both of the compared frame images, indicating that a large scene change has occurred in the two frame images, such as there is no person A in the first frame image, and person A walks into the second frame image. If it is less than the dynamic threshold, then continue to obtain the size of the dynamic threshold at this time. For the case of a resolution of 1920*1080, set the second threshold to 400000. When the dynamic threshold is less than the second threshold, it is determined that the current dynamic threshold is small, which means that the change between the two frame images is small, and then the label representing the static image is set in both of the compared frame images. On the contrary, when the dynamic threshold is greater than or equal to the second threshold, it is determined that the currently determined dynamic threshold is larger, indicating that even if there is a person moving in the image, it may be judged to be smaller than the dynamic threshold, that is, there may still be people moving in the second dynamic image. At this time, a label representing the second dynamic image is set in both compared frame images.

[0041] Finally, the frame image labeled as the first dynamic image is used as the first segment, the frame image labeled as the second dynamic image is used as the second segment, and the frame image labeled as the static image is used as the third segment.

[0042] In this embodiment, determining the dynamic threshold includes the following steps: Acquire historical videos, process the historical videos into dynamic time series about changing quantities, use a preset window size as a parameter, process the dynamic time series based on a sliding window method, obtain historical window values ​​of multiple time windows, split the recording time of the historical videos into multiple first time periods, and statistically analyze the historical window values ​​to determine the adjustment coefficient of each first time period.

[0043] Specifically, the historical video includes the scene video recorded within the past month. According to the above method, the scene video within one month is split into multiple frame images, and then two adjacent frame images are compared to obtain the number of changes between the two frame images. The number of changes is arranged in chronological order, and the dynamic time series corresponding to the historical video can be obtained.

[0044] For the sake of simplicity, let's take the dynamic time series as (11, 15, 30, 14, 16) and set the window size to 3. Then the first historical window value is (11+15+30) / 3=18.7, the second historical window value is (15+30+14) / 3=19.7, and so on, until the window slides to the end of the dynamic time series.

[0045] This embodiment divides the recording time into a first time period with a duration of one hour, and divides it by week, such as the first first time period is 00:00-01:00 on Monday, and the second first time period is 01:00-02:00 on Monday. Then the changes in the historical window values ​​in each first time period are counted. In actual processing, when the window size is set to 10, there are 3591 historical window values ​​in a first time period. This embodiment uses the following method to determine the adjustment coefficient. First, multiple adjustment coefficient values ​​are set, such as 0.1, 0.2, 0.3, etc. Each adjustment coefficient corresponds to a critical threshold, such as adjustment coefficients 0.1, 0.2, 0.3 correspond to critical threshold 1, critical threshold 2 and critical threshold 3, respectively, and an occupation ratio standard is also set, such as 80%.

[0046] For example, after statistics, there are a total of 3591*4=14364 historical window values ​​from 00:00 to 01:00 on four Mondays within a month, of which more than 98% of the historical window values ​​are less than the critical threshold 1, then the adjustment coefficient is set to 0.1; if more than 98% of the historical window values ​​are between the critical threshold 1 and the critical threshold 2, then the adjustment coefficient is set to 0.2, and so on.

[0047] Extract the mean of the number of changes in the second time period before the two frame images, determine the corresponding adjustment coefficient based on the first time period where the two frame images are located, correct the mean based on the adjustment coefficient, obtain the first value, set the basic value, correct the basic value based on the first value, and obtain the dynamic threshold.

[0048] For example, through statistics, it is determined that the historical window values ​​between 12:00-13:00 from Monday to Friday are all large, and the corresponding adjustment coefficient is set to 0.9. Here, the second duration is set to 20s. For example, the time points corresponding to the two frame images are 12:00:21 and 12:00:22 on Monday, respectively. Then, 20 frame images between 12:00:00 and 12:00:20 on Monday are obtained, and then the number of changes in the 20 frame images is calculated. Finally, 19 values ​​can be obtained. The average of the 19 values ​​is calculated, for example, 1000000. The basic value can be set to 50000. By setting the basic value, it can be avoided that when the picture changes slightly, the corresponding frame image is not classified as the first dynamic image. At this time, the dynamic threshold corresponding to the two frame images is 50000+1000000*0.9=950000.

[0049] The present embodiment analyzes a multi-segment video including the following steps: The first segment and the second segment in the multi-segment video are defined as the first analysis segment and the second analysis segment respectively. The target is tracked in the first analysis segment based on the target detection algorithm. If a person is detected moving, the risk is calculated based on the frequency of occurrence, the entropy value of the activity trajectory, and the number of face matching failures.

[0050] Based on the previous introduction, the first analysis section is a section where the picture changes suddenly appear in a static environment. Therefore, when the target detection algorithm is used to track the target in the first analysis section, a better tracking effect can be obtained. The target tracking algorithm can use the similarity matching algorithm of the Siamese network, the yolo algorithm, the correlation filtering algorithm, etc. In the process of tracking the target, the target face recognition is continuously performed to obtain the number of successful face matching and the number of failed face matching, and the target movement trajectory is recorded and generated at the same time. This embodiment is based on the following method to obtain the frequency of occurrence and the entropy value of the activity trajectory. First, multiple fixed position points are specified in the digital twin model. Whenever the tracked target passes through a fixed position point, the frequency of occurrence is accumulated once.

[0051] When calculating the entropy value of the activity trajectory, the target movement trajectory is first represented by multiple coordinate points, and then the first formula is used to calculate the entropy value of the activity trajectory. The probability of a coordinate point , the first formula is: ,in, is the number of coordinate points in the moving trajectory, Represents the moving trajectory through the The number of coordinate points. Then the activity trajectory entropy value is calculated by the second formula, which is: ,The calculation result of the second formula shows that the higher the entropy value, the more irregular the movement of the target. ,Under normal circumstances, the behavior of people working in office buildings has certain regularities.

[0052] Finally, the risk level is calculated by taking the weighted sum of the occurrence frequency, activity trajectory entropy, and number of face matching failures. The specific weights can be determined based on actual experience.

[0053] Obtain the second time period in which the second analysis segment appears, intercept the video image of the scene video shot by other surveillance cameras in the second time period, define it as an auxiliary segment, split the second analysis segment and the auxiliary segment into frame images, and combine the frame images with the same shooting time into an image group, perform mapping analysis on the image group, and obtain the coordinate position of the same target in different frame images.

[0054] The second analysis section is a section where the picture has always had a large change, in which there may be a situation where people need to move around. At this time, the use of the target tracking algorithm may obtain poor results due to the occlusion of the crowd's back and forth movement. For this reason, the present invention also proposes the following method. First, in an office building, in order to achieve full coverage of the same area, it is necessary to deploy multiple surveillance cameras, and there are overlapping monitoring areas between the cameras, so as to achieve no-dead-angle monitoring, such as being deployed at the four corners of the ceiling. On this hardware basis, it is assumed that there are three surveillance cameras in the target area, where the second analysis section is derived from the scene video A shot by surveillance camera 1, and the second time period where the second analysis section appears is determined, such as 12:00:00-12:00:10, then the video sections of 12:00:00-12:00:10 in the scene video B shot by surveillance camera 2 and the scene video C shot by surveillance camera 3 are obtained, which are defined as auxiliary sections 1 and auxiliary sections 2, respectively.

[0055] The second analysis section, auxiliary section 1 and auxiliary section 2 are split into frame images at intervals of 1s, and each frame image has a label of the shooting time. Afterwards, the frame images with the same shooting time are combined into image groups, and each image group contains three frame images, which come from the shooting angles of surveillance cameras 1, 2 and 3 respectively. Afterwards, the three frame images in the image group are mutually mapped and analyzed to determine the position of the same person in different frame images. The above similarity comparison algorithm or contour similarity comparison algorithm can be used. For the human body, the coordinate position can be selected from the head, limb endpoints or other positions, etc. The coordinate position can be one or more. The position of the head and limbs of the human body in the image can be determined using a bone analysis algorithm. The bone analysis algorithm is a prior art and will not be introduced here. The above scheme can be used to obtain the number of people in the target area to avoid a certain person appearing in multiple cameras and being repeatedly counted.

[0056] To simplify the determination of coordinate positions, this embodiment divides each frame image into a 10*10 grid, and then determines the coordinate position according to the grid where the user's head is located. For example, for employee A, his head is located in the grid area of ​​the third row and third column in the frame image, so his coordinate position is (3,3). For employee A, he appears in all three frame images in the image group, so his position coordinates are (3,3), (5,6), (6,4).

[0057] Environmental sensors include beacon sensors. The beacon sensors obtain the return signal strength based on the employee's work card, and combine the return signal strength into a signal group corresponding to the image group with time as a reference, and establish an analysis model. The analysis model outputs the risk of suspicious persons based on the coordinate position of the image group and the return signal strength in the corresponding signal group.

[0058] In this embodiment, the environmental sensor also includes a beacon sensor, and a beacon tag is embedded in the employee card. The beacon sensor can send a signal to the beacon tag and obtain the returned signal strength. In other embodiments, the environmental sensor can also be a Bluetooth sensor, or a wireless router. The Bluetooth sensor and the wireless router obtain the returned signal strength based on the employee's mobile terminal. The above devices are all devices that employees will carry with them, so they will have higher detection accuracy.

[0059] Based on the above introduction, through the 10 image groups in the period of 12:00:00-12:00:10, the coordinate changes of the people in them can be obtained, so as to construct a position matrix. The position matrix represents the flow of people in the target area within these 10 seconds. For example, in the position matrix, the first row and the first column represent the number of people in the target area in the first second, and the second row and the first column represent the horizontal coordinates of the coordinate position of the person in the first frame image in the image group. The other elements in the matrix are similar and will not be introduced one by one. Based on the 10 signal groups, a signal matrix is ​​constructed, in which the first row and the first column represent the return signal strength sequence of each beacon received by the beacon sensor in the first second. The above construction method for inputting the analysis model is not unique and can be determined according to the actual situation. Afterwards, the position matrix and the signal matrix are input into the analysis model, and the analysis model outputs the risk level of the suspicious person.

[0060] The core principle of the above method is that, for example, within 10 seconds, 8 people walking in the picture are detected through video analysis, but combined with the signal strength, it is determined that there are only seven people walking at the scene, which indicates that a suspicious person has appeared.

[0061] The present embodiment establishes the analysis model comprises the following steps: Collect historical data, which includes image groups, signal groups and output labels. The output labels include no risk and risky. Make numerical adjustments to the coordinate positions in the image group and the return signal strength in the signal group to expand the historical data and obtain expanded data. Establish an analysis model based on a neural network and use the expanded data to train the analysis model. When the output result of the analysis model is risky, the probability value of risk is used as the risk degree.

[0062] This embodiment establishes an analysis model based on the LSTM time series neural network. In other embodiments, a fully connected neural network can also be used to establish an analysis model. As a kind of supervised learning, the time series neural network needs to be pre-trained with data. For this purpose, historical data needs to be collected. The historical data includes input data and output data. The input data is consistent with the data form of the actual input, that is, a piece of historical data includes a position matrix and a signal matrix generated by an image group and a signal group, and the output data includes risk and no risk, and risk and no risk are marked in it by manual annotation. When the amount of data is reduced, the value of the elements in the position matrix or the image matrix can be fine-tuned, such as increasing the value of the coordinates, or appropriately reducing the return signal strength, so as to achieve the goal of expanding the data. Finally, the probability of risk or risk is output by adding a Softmax function to the neural network to obtain the corresponding risk degree.

[0063] In this embodiment, the fire generation and disposal plan includes the following steps: The disposal plan includes multiple predetermined escape routes. If there are multiple escape routes, the fire location is determined, the fire spread direction is simulated based on the fire location, and the best route is selected from the predetermined evacuation routes based on the spread direction.

[0064] Specifically, firstly, the fire source location data is collected in real time through smoke sensors, temperature detectors and video monitoring systems, and the coordinates of the fire point are accurately located in combination with the digital twin model, and the building structure parameters such as the ventilation system status and material combustion characteristics are simultaneously obtained; then, the fire smoke diffusion model based on Gaussian distribution is used to predict the spread path of smoke, high temperature and toxic gas, generate a thermal distribution map, and focus on marking key areas such as stairwells and corridors that may be covered by the fire; then, a topological analysis is performed on multiple pre-stored evacuation routes, and routes that pass through high temperature areas or downwind outlets are eliminated through a spatial superposition algorithm, and safe passages that are against the wind and away from thick smoke are given priority; finally, the Dijkstra algorithm is used to comprehensively evaluate the path length, coverage of dangerous areas and travel time, and the optimal route is pushed through the mobile terminal, and the data is dynamically updated every 30 seconds. When it is detected that the original route is blocked by the fire, the backup route is automatically enabled. The above-mentioned fire smoke diffusion model is a prior art and will not be introduced here.

[0065] like Figure 2 As shown, the present invention also provides a three-dimensional scene building system for police cloud IoT devices based on digital twins, which is used to implement the above-mentioned three-dimensional scene building method for police cloud IoT devices based on digital twins, and the system includes: Perception devices, which include surveillance cameras and environmental sensors. The surveillance cameras are used to record scene videos, and the environmental sensors are used to detect environmental data. The cloud server is used to establish a three-dimensional physical model of the target area, arrange the corresponding virtual model in the physical model based on the installation information of the sensing device, and connect the data from the sensing device to the corresponding virtual model to generate a digital twin model; The first investigation unit is used to analyze scene videos and environmental data in real time. If environmental risks are detected, a disposal plan is generated and displayed in the digital twin model; The second screening unit is used to process the scene video into multi-segment video, analyze the multi-segment video in combination with environmental data, determine the suspicious persons and risk levels in the target area, screen the suspicious persons whose risk levels are greater than the first threshold as high-risk risks, and generate suspicious risk alarms pointing to high-risk risks in the digital twin model.

[0066] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0068] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for building a three-dimensional scene of police cloud IoT equipment based on digital twins, characterized in that: include: Deploy sensing devices in the target area. The sensing devices include surveillance cameras and environmental sensors. The surveillance cameras are used to record scene videos, and the environmental sensors are used to detect environmental data. A three-dimensional physical model of the target area is established in the cloud server, a corresponding virtual model is arranged in the physical model based on the installation information of the sensing device, and the data from the sensing device is connected to the corresponding virtual model to generate a digital twin model; The first investigation unit analyzes scene videos and environmental data in real time. If an environmental risk is detected, a disposal plan is generated and displayed in the digital twin model. After the second investigation unit processes the scene video into multi-segment videos, it analyzes the multi-segment videos in combination with environmental data to determine suspicious persons and risk levels in the target area; Suspicious persons with a risk level greater than a first threshold are screened as high-risk persons, and suspicious risk alerts pointing to high-risk persons are generated in the digital twin model.

2. According to the method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 1, it is characterized in that: Real-time analysis of scene video and environmental data includes the following steps: Define the scene video to be analyzed as the video to be analyzed, extract the video to be analyzed into multiple frame images based on the standard interval, locate the frame image of the current time in the video to be analyzed, and if a dangerous marker is detected in the frame image, determine the corresponding environmental risk according to the dangerous marker and generate a disposal plan; Environmental sensors include temperature sensors, smoke sensors and current sensors. Environmental data include temperature data, smoke data and current data. If the environmental data exceeds the corresponding set threshold when no hazardous markers are detected, it is determined that there is an environmental risk and a disposal plan is generated.

3. The method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 2 is characterized in that: Processing a scene video into a multi-segment video includes the following steps: The multi-segment video includes a first segment, a second segment and a third segment, the frame images are arranged in chronological order, the change area between adjacent frame images is extracted based on the inter-frame difference method, and the number of changes in the pixel points in the change area is counted; If the number of changes is greater than or equal to the dynamic threshold, the two frame images that generate the number of changes are marked as the first dynamic image. If the number of changes is less than the dynamic threshold, the size of the dynamic threshold is obtained. If the dynamic threshold is greater than or equal to the second threshold, the two frame images are marked as the second dynamic image. Otherwise, the frame images are marked as static images, and the first dynamic image is integrated into the first segment, the second dynamic image is integrated into the second segment, and the static image is integrated into the third segment.

4. The method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 3 is characterized in that: Determining the dynamic threshold includes the following steps: Acquire historical videos, process the historical videos into dynamic time series about the changing quantity, process the dynamic time series based on the sliding window method with a preset window size as a parameter, obtain historical window values ​​of multiple time windows, split the recording time of the historical videos into multiple first time periods, and statistically analyze the historical window values ​​to determine the adjustment coefficient of each first time period; Extract the mean of the number of changes in the second time period before the two frame images, determine the corresponding adjustment coefficient based on the first time period where the two frame images are located, correct the mean based on the adjustment coefficient, obtain the first value, set the basic value, correct the basic value based on the first value, and obtain the dynamic threshold.

5. The method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 3 is characterized in that: Analyzing a multi-segment video involves the following steps: Define the first segment and the second segment in the multi-segment video as the first analysis segment and the second analysis segment respectively, and track the target in the first analysis segment based on the target detection algorithm. If a person is detected moving, calculate the risk based on the frequency of occurrence, the entropy value of the activity trajectory, and the number of face matching failures; Obtain a second time period in which the second analysis segment appears, intercept video images of the scene video captured by other surveillance cameras in the second time period, define the video images as auxiliary segments, split the second analysis segment and the auxiliary segments into frame images, and combine the frame images with the same shooting time into an image group, perform mapping analysis on the image group, and obtain the coordinate position of the same target in different frame images; Environmental sensors include beacon sensors. The beacon sensors obtain the return signal strength based on the employee's work card, and combine the return signal strength into a signal group corresponding to the image group with time as a reference, and establish an analysis model. The analysis model outputs the risk of suspicious persons based on the coordinate position of the image group and the return signal strength in the corresponding signal group.

6. The method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 5 is characterized in that: Building the analysis model includes the following steps: Collect historical data, which includes image groups, signal groups and output labels. The output labels include no risk and risky. Make numerical adjustments to the coordinate positions in the image group and the return signal strength in the signal group to expand the historical data and obtain expanded data. Establish an analysis model based on a neural network and use the expanded data to train the analysis model. When the output result of the analysis model is risky, the probability value of risk is used as the risk degree.

7. The method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 1 is characterized in that: Generating a disposal plan includes the following steps: The disposal plan includes multiple predetermined escape routes. If there are multiple escape routes, the fire location is determined, the fire spread direction is simulated based on the fire location, and the best route is selected from the predetermined evacuation routes based on the spread direction.

8. The method for building a three-dimensional scene of police cloud IoT equipment based on digital twins according to claim 2 is characterized in that: Hazard signs include flames, smoke, and standing water.

9. A three-dimensional scene construction system for police cloud IoT devices based on digital twins, used to implement the three-dimensional scene construction method for police cloud IoT devices based on digital twins as described in any one of claims 1 to 8, characterized in that: include, Perception devices, which include surveillance cameras and environmental sensors. The surveillance cameras are used to record scene videos, and the environmental sensors are used to detect environmental data. The cloud server is used to establish a three-dimensional physical model of the target area, arrange the corresponding virtual model in the physical model based on the installation information of the sensing device, and connect the data from the sensing device to the corresponding virtual model to generate a digital twin model; The first investigation unit is used to analyze scene videos and environmental data in real time. If environmental risks are detected, a disposal plan is generated and displayed in the digital twin model; The second screening unit is used to process the scene video into multi-segment video, analyze the multi-segment video in combination with environmental data, determine the suspicious persons and risk levels in the target area, screen the suspicious persons whose risk levels are greater than the first threshold as high-risk risks, and generate suspicious risk alarms pointing to high-risk risks in the digital twin model.

Images