Security monitoring method and system, terminal equipment and computer program product

Abstract

The invention belongs to the technical field of computer vision, and provides a safety monitoring method and system, terminal equipment and a computer program product, and the method comprises the steps: collecting intensity image data and three-dimensional point cloud data of a target region; performing target detection and segmentation on the intensity image data based on a target detection model to obtain regional information of a target person; performing three-dimensional mapping based on the three-dimensional point cloud data and the regional information of the target person to obtain three-dimensional space information of the target person; and performing safety warning operation under the condition that the three-dimensional space information is overlapped with a preset three-dimensional safety frame. According to the method provided by the invention, efficient, accurate and stable safety monitoring of the target area is realized, and the personnel safety of the target area is guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a security monitoring method, system, terminal device and computer program product. Background Art

[0002] Traditional personnel monitoring and security systems mainly rely on two-dimensional video surveillance, which makes it difficult to accurately determine the specific location and behavior of personnel in three-dimensional space. With the development of 3D perception technology, especially the application of 3D cameras, it is possible to simultaneously obtain intensity images and three-dimensional point cloud information of the scene, providing new possibilities for personnel detection and security monitoring in three-dimensional space.

[0003] However, how to effectively utilize these intensity images and three-dimensional data for stable and reliable security monitoring remains an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a security monitoring method, system, terminal device and computer program product to achieve accurate and stable security monitoring of personnel in complex environments and ensure the safety of personnel.

[0005] A first aspect of an embodiment of the present application provides a security monitoring method, including:

[0006] Collect intensity image data and three-dimensional point cloud data of the target area;

[0007] Performing target detection and segmentation on the intensity image data based on a target detection model to obtain region information of the target person;

[0008] Performing three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person to obtain the three-dimensional spatial information of the target person;

[0009] When the three-dimensional space information overlaps with a preset three-dimensional safety frame, a safety warning operation is performed.

[0010] In an implementation of the first aspect, performing target detection and segmentation on the intensity image data based on a target detection model to obtain region information of a target person includes:

[0011] Performing target detection on the intensity image based on a target detection model to obtain classification information of the target;

[0012] If the classification information of the target indicates that the target is a target person, then obtaining detection frame information of the target person;

[0013] Performing coordinate conversion according to the detection frame information of the target person and the preset detection frame information to obtain the target frame information of the detection frame of the target person on the intensity image;

[0014] Acquire segmentation information of the target person, where the segmentation information is used to indicate pixels corresponding to the target person;

[0015] According to the detection frame information and the target frame information, coordinate transformation is performed on the segmentation information to obtain the region information of the target person.

[0016] In an implementation of the first aspect, the classification information of the target includes a confidence level and a classification result;

[0017] If the classification information of the target indicates that the target is a target person, obtaining the detection frame information of the target person includes:

[0018] If the confidence level is higher than a preset threshold and the classification result indicates that the target is a target person, the detection frame information of the target person is obtained.

[0019] In an implementation of the first aspect, performing three-dimensional mapping based on the three-dimensional point cloud data and the region information of the target person to obtain the three-dimensional space information of the target person includes:

[0020] Obtaining the two-dimensional coordinates of each pixel in the region information of the target person;

[0021] Determine the three-dimensional coordinates of each pixel in the corresponding area information of the target person in the three-dimensional point cloud data according to the two-dimensional coordinates;

[0022] The three-dimensional coordinates are integrated to obtain the three-dimensional spatial information of the target person.

[0023] In an implementation of the first aspect, before performing the safety warning operation when the three-dimensional space information overlaps with the preset three-dimensional safety frame, the method further includes:

[0024] Detect safe frame configuration instructions;

[0025] If a safety frame configuration instruction is detected, the preset three-dimensional safety frame is updated according to the safety frame configuration instruction.

[0026] In an implementation of the first aspect, before performing target detection and segmentation on the intensity image data based on the target detection model to obtain region information of the target person, the method further includes:

[0027] An initial network model is trained based on an intensity image dataset to obtain a target detection model, wherein the output of the initial network model includes detection box information, classification information, and segmentation information of targets at multiple scales.

[0028] In an implementation of the first aspect, when the three-dimensional space information overlaps with a preset three-dimensional safety frame, performing a safety warning operation includes:

[0029] When the three-dimensional space information overlaps with a preset three-dimensional safety frame, triggering an acoustic signal, an optical signal, or an acoustic-optical signal warning;

[0030] The preset three-dimensional safety frame is highlighted on the monitoring display interface.

[0031] A second aspect of an embodiment of the present application provides a security monitoring system, including:

[0032] A data acquisition module, used to acquire intensity image data and three-dimensional point cloud data of the target area;

[0033] A target detection module, used to perform target detection and segmentation on the intensity image data based on a target detection model to obtain region information of a target person;

[0034] A three-dimensional mapping module, used to perform three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person, and obtain the three-dimensional space information of the target person;

[0035] The safety warning module is used to perform a safety warning operation when the three-dimensional space information overlaps with a preset three-dimensional safety frame.

[0036] The third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the security monitoring method described in the first aspect when executing the computer program.

[0037] A fourth aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables the security monitoring method described in the first aspect to be executed.

[0038] The beneficial effect of the first aspect of the embodiment of the present application is: by collecting intensity image data and three-dimensional point cloud data of the target area, and then performing target detection and segmentation on the intensity image data based on the target detection model, the regional information of the target person is obtained, and three-dimensional mapping is performed based on the three-dimensional point cloud data and the regional information of the target person to obtain the three-dimensional spatial information of the target person, and then when the three-dimensional spatial information overlaps with the preset three-dimensional safety frame, a safety warning operation is performed to achieve efficient, accurate and stable security monitoring of the target area, thereby ensuring the safety of personnel.

[0039] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a schematic diagram of an implementation flow of a security monitoring method provided in an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of an implementation flow of a security monitoring method provided in an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of a security monitoring system provided by an embodiment of the present application;

[0044] Figure 4 is a schematic diagram of a terminal device provided in an embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of a computer program product provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0047] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0048] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0050] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0051] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0052] The embodiment of the present application provides a security monitoring method for realizing accurate and stable security monitoring of a target area and ensuring the safety of personnel in the target area. The security monitoring method provided in the embodiment of the present application acquires intensity image data and three-dimensional point cloud data of the target area, and then performs target detection and segmentation on the intensity image data based on the target detection model to obtain the regional information of the target person, and performs three-dimensional mapping based on the three-dimensional point cloud data and the regional information of the target person to obtain the three-dimensional spatial information of the target person, and then performs a safety warning operation when the three-dimensional spatial information overlaps with a preset three-dimensional safety frame, thereby realizing efficient, accurate and stable security monitoring of the target area and ensuring the safety of personnel in the target area.

[0053] The security monitoring method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, vehicle-mounted equipment, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), desktop computers, notebooks, PDAs and cloud servers. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0054] like Figure 1 As shown, the embodiment of the present application provides a security monitoring method, including:

[0055] Step S11, collecting intensity image data and three-dimensional point cloud data of the target area.

[0056] In the application, taking the production workshop as the target area as an example, the production workshop includes: production line machinery and equipment, such as automated robotic arms, conveyor belts, etc.; dangerous areas, such as high-temperature equipment areas, hazardous chemical storage areas, etc.; prohibited areas, such as control rooms or equipment maintenance areas that are only accessible to specific personnel; dynamic objects, such as moving material transport vehicles (forklifts) and robots (AGVs); human flow activities, where employees may move in or out of the designated area during their daily work. In order to ensure safety during the production process, it is necessary to monitor the personnel in the production workshop (target area) in real time to prevent personnel from entering dangerous areas or colliding with automated robotic arms, material transport vehicles, etc., which may cause safety accidents.

[0057] In applications, intensity images are similar to photos taken by ordinary RGB cameras, recording the intensity value of light reflected or emitted by each pixel; a 3D point cloud is a collection of discrete points, each of which represents a position in space and carries x, y, z coordinate information. The intensity image data and 3D point cloud data of the target area can be obtained through a 3D camera equipped with one or more black and white or color imaging sensors such as a structured light camera (such as Kinect), a time-of-flight (ToF) camera, or a laser radar (LiDAR). It is understandable that in actual applications, the intensity image data and 3D point cloud data of the target area are obtained in real time through real-time video monitoring by a 3D camera.

[0058] In the application, when used for the first time or under different security monitoring requirements, the working parameters of the 3D camera for acquiring 3D point cloud data are configured, such as frame rate, camera exposure time, time synchronization information, photo mode, filter module, etc. The 3D camera continuously acquires intensity image data and 3D point cloud data of the target area based on the above working parameters, such as the preset frame rate.

[0059] Step S12, performing target detection and segmentation on the intensity image data based on the target detection model to obtain region information of the target person.

[0060] In the application, the target detection model is trained through deep learning methods such as convolutional neural networks (CNNs), and can automatically identify the target according to its features, and divide the target person's regional information in the intensity image. In the application, the target person's regional information is any one of a human-shaped area and a rectangular area frame at the pixel level. Through the above steps, the target person can be segmented earlier in the intensity image, which is conducive to reducing the calculation and processing content of subsequent steps and improving the overall monitoring efficiency.

[0061] Step S13, performing three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person to obtain the three-dimensional space information of the target person.

[0062] In the application, through three-dimensional mapping, combined with the regional information of the target person in the intensity image data and the three-dimensional space information in the three-dimensional point cloud data, the target person can be accurately mapped into the three-dimensional space, and his precise coordinates and posture in the space can be obtained, further improving the accuracy and reliability of subsequent security monitoring.

[0063] Step S14: When the three-dimensional space information overlaps with a preset three-dimensional safety frame, a safety warning operation is performed.

[0064] In the application, the three-dimensional safety frame can be pre-set at a fixed position according to actual needs, or it can be a relative position, and dynamically configured according to the area group signal. For example, around the material transport vehicle, the corresponding area group signal is triggered according to the movement of the material transport vehicle to configure the corresponding three-dimensional safety frame. One area group event signal corresponds to one three-dimensional safety frame or multiple three-dimensional safety frames, and different area group event signals independently correspond to one or more three-dimensional safety frames. The area group event signal may be a fixed signal or a dynamically changing signal. For example, for the safety monitoring of a factory production workshop, since the robotic arm is generally in a fixed area, its corresponding three-dimensional safety frame is also a fixed area. Therefore, the safety monitoring of the robotic arm in the target area is determined by the fixed area event signal to determine the fixed three-dimensional safety frame. For the safety monitoring of the material transport robot moving in the factory production workshop, since the robot is mobile, its movement modes include moving forward, moving backward, turning left, turning right, moving left and moving right, etc. It can be understood that the three-dimensional safety frames that need to be monitored for safety are different under different movement modes. For example, when turning left, the corresponding three-dimensional safety frames on the left, front, left front and left rear need to be monitored for safety; when moving forward, the corresponding three-dimensional safety frames in front need to be monitored for safety; when turning right, the corresponding three-dimensional safety frames on the right, front, right front and right rear need to be monitored for safety. In the application, for the moving robot, each of its movement modes can be assigned a unique area group event signal in advance and the corresponding three-dimensional safety frame group can be configured. Then, during the safety monitoring process, the target movement mode of the moving robot is obtained in real time, the area group event signal is determined according to the target movement mode, and then the corresponding three-dimensional safety frame group is determined, so as to realize the flexible configuration of the three-dimensional safety frame that needs to be analyzed for intrusion. In the application, the 3D information of the target person and the preset safety frame can be used to perform geometric calculations, such as distance, angle, etc., to determine whether the two overlap. The 3D information and the preset safety frame can also be projected onto three different 2D planes (such as OXY, OXZ, and OYZ planes), and then the overlap of these projections on the 2D planes can be compared. If two or more of the three projections overlap, the 3D information and the preset safety frame overlap.

[0065] like Figure 2 As shown, in one embodiment, the step S12, performing target detection and segmentation on the intensity image data based on the target detection model to obtain the region information of the target person, includes:

[0066] Step S121, performing target detection on the intensity image based on the target detection model to obtain classification information of the target.

[0067] In the application, the intensity image is input into the target detection model for target detection, and the classification information of each target in the intensity image is obtained. Before the intensity information is input into the target detection model, the intensity image is also preprocessed, such as adjusting the image size, normalizing, and denoising, to improve the accuracy of target detection. The classification information of the target includes the type of the target, such as mobile targets such as personnel and material transport vehicles.

[0068] Step S122: If the classification information of the target indicates that the target is a target person, then the detection frame information of the target person is obtained.

[0069] In applications, due to the difference in the distance between the target person and the 3D camera, the detection frame information of the target person includes detection frames of different sizes and resolutions. The initial coordinates of the detection frame are usually the normalized values ​​output by the network, expressed as (x, y, w, h), where x and y are the center coordinates of the detection frame (normalized to [0, 1]); w and h are the width and height of the detection frame (normalized to [0, 1]). The target detection frame output by the target detection model usually consists of the following parts: Anchor Boxes, which are predefined boxes whose size and shape are usually associated with the possible scale and aspect ratio of the target object. Each anchor box is aligned with a small area in the image. Assuming there are K anchor boxes, the coordinates of the anchor box can be expressed as (x_anchor, y_anchor, w_anchor, h_anchor), where x_anchor and y_anchor are the center coordinates of the anchor box, and w_anchor and h_anchor are the width and height of the anchor box. Offsets, each anchor box output by the network usually predicts 4 values, indicating the offset of the target relative to the anchor box, including: t_x, t_y represents the offset of the center of the detection box relative to the center of the anchor box; t_w, t_h represents the scale of the detection box relative to the width and height of the anchor box. Softmax output is the category probability of each anchor box output by the network. Softmax calculation is usually used to determine the category to which the detection box belongs, and select the category with the highest probability or a probability greater than the preset probability threshold.

[0070] In the application, if the classification information of the target indicates that the target is a target person, the identity information of the target person is collected. If the identity information of the target person is empty, jump to the step of performing a warning operation; if the identity information represents that the target person is a trusted employee, the detection frame information of the target person is obtained. In the application, as a trusted employee, a signal nameplate including identity information will be worn when entering the production workshop. The identity information of the target can be obtained by collecting the signal of the signal nameplate.

[0071] Step S123, performing coordinate transformation according to the detection frame information of the target person and the preset detection frame information to obtain the target frame information of the detection frame of the target person on the intensity image.

[0072] In the application, the preset detection frame information includes the anchor frame and the offset, and the detection frame information of the target person includes the coordinates of the detection frame of the target person. According to the center coordinates and size information of the anchor frame, the detection of the target person is restored from the offset form to the target frame information on the intensity image. The target frame information includes the center coordinates, width, and height of the target frame. The coordinate center of the target frame is decoded as follows: x_target = x_anchor + t_x × w_anchor, y_target = y_anchor + t_y × h_anchor, where t_x and t_y are the offsets predicted by the target detection model, and w_anchor and h_anchor are the width and height of the anchor frame. Width and height decoding: w_target = w_anchor × e_tw, h_target = h_anchor × e_th, where tw and th are the logarithmic scale offsets predicted by the target detection model. The exponential functions e_t and e_th are used to restore the scale of the target frame. The target frame coordinates obtained after decoding are: (x_target, y_target, w_target, h_target).

[0073] Step S124, obtaining segmentation information of the target person, where the segmentation information is used to indicate pixels corresponding to the target person.

[0074] In the application, the segmentation information is represented by the feature map output by the network segmentation branch of the target detection model, which indicates the probability of each pixel belonging to the target category. This feature map is usually of low resolution (such as 160×120) and needs to be upsampled and threshold segmented. Applying threshold segmentation, the pixels with probability values ​​≥ the segmentation threshold are marked as 1, and the other pixels are marked as 0, and a binary mask is obtained to indicate the pixel position of the target person in the intensity image. According to the coordinates of the detection box, the pixels in the segmentation mask belonging to the detection box area are intercepted to ensure that the segmentation information only contains the target person pixels within the detection box range.

[0075] In the application, more accurate detection of the target person is achieved through pixel-level segmentation information, which reduces the subsequent calculation amount and improves efficiency.

[0076] Step S125 , performing coordinate transformation on the segmentation information according to the detection frame information and the target frame information to obtain the region information of the target person.

[0077] In the application, coordinate transformation is performed on the pixel area representing the target person in the segmentation information to obtain the area information of the target person.

[0078] In applications, the category probability output by each anchor box can be normalized by Softmax to select the most likely target category. Then, the weighted expectation method is used to calculate the final detection box.

[0079] For an output layer containing C categories, the network provides a vector containing C category probabilities for each anchor box. Through Softmax, the normalized probability distribution of each category can be obtained, and then the category with the highest probability is selected as the category of the detection result. The probability distribution of the category is weighted to obtain the expectation of all anchor boxes. The final detection box combines the position and category information of all candidate anchor boxes in a weighted manner. For each anchor box, the category with the highest category probability is selected. For the selected category, the offset corresponding to its detection box and the anchor box are decoded to obtain the final target box.

[0080] In one embodiment, the classification information of the target includes a confidence level and a classification result;

[0081] In step S122, if the classification information of the target indicates that the target is a target person, obtaining the detection frame information of the target person includes:

[0082] Step S1121, if the confidence level is higher than a preset threshold, and the classification result indicates that the target is a target person, then the detection frame information of the target person is obtained.

[0083] In one embodiment, the step S13, performing three-dimensional mapping based on the three-dimensional point cloud data and the region information of the target person to obtain the three-dimensional space information of the target person, includes:

[0084] Step S131, obtaining the two-dimensional coordinates of each pixel in the area information of the target person.

[0085] Step S132, determining the three-dimensional coordinates of each pixel in the corresponding area information of the target person in the three-dimensional point cloud data according to the two-dimensional coordinates.

[0086] Step S133, integrating the three-dimensional coordinates to obtain the three-dimensional space information of the target person.

[0087] In one embodiment, the step S14, before performing a safety warning operation when the three-dimensional space information overlaps with a preset three-dimensional safety frame, further includes:

[0088] Step S141, detecting a safety frame configuration instruction.

[0089] Step S142: if a safety frame configuration instruction is detected, then the preset 3D safety frame is updated according to the safety frame configuration instruction.

[0090] In one embodiment, the step S12, before performing target detection and segmentation on the intensity image data based on the target detection model to obtain the region information of the target person, further includes:

[0091] Step S10, training an initial network model based on the intensity image data set to obtain a target detection model, wherein the output of the initial network model includes detection box information, classification information, and segmentation information of targets at multiple scales.

[0092] In one embodiment, the step S14, when the three-dimensional space information overlaps with a preset three-dimensional safety frame, performs a safety warning operation, including:

[0093] Step S141 , when the three-dimensional space information overlaps with a preset three-dimensional safety frame, triggering a sound signal, a light signal, or a sound and light signal warning.

[0094] Step S142: highlighting the preset three-dimensional safety frame on the monitoring display interface.

[0095] In one embodiment, when the three-dimensional space information overlaps with a preset three-dimensional safety frame, before performing the safety warning operation, the method further includes:

[0096] Get the target person's access attributes;

[0097] If the pass attribute of the target person is passable, a message notification is sent to the target person.

[0098] It is understandable that when the target person is passable, there is no need to trigger warning operations including sound and light signals.

[0099] In application, the method provided in the embodiment of the present application performs security monitoring through two independent security monitoring modules to obtain a first monitoring result and a second monitoring result;

[0100] If the three-dimensional space information represented by any one of the first monitoring result and the second monitoring result overlaps with the preset three-dimensional safety frame, a safety warning operation is performed.

[0101] It can be understood that by using two independent security monitoring modules and performing a logical OR operation on the two security monitoring results, the final security monitoring result is obtained, thereby ensuring the reliability of monitoring.

[0102] In one embodiment, the method further comprises:

[0103] When the reset signal is received, if the safety warning is currently being performed and there is no three-dimensional spatial information of the target person overlapping with the preset three-dimensional safety frame, the safety warning is terminated.

[0104] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0105] The embodiment of the present application also provides a security monitoring system for executing the steps in the above security monitoring method embodiment. The security monitoring system can be a virtual device (Virtual Appliance) in the terminal device, which is run by the processor of the terminal device, or it can be the terminal device itself.

[0106] like Figure 3 As shown, the security monitoring system 300 provided in the embodiment of the present application includes:

[0107] A data acquisition module 301 is used to acquire intensity image data and three-dimensional point cloud data of a target area;

[0108] The target detection module 302 is used to perform target detection and segmentation on the intensity image data based on the target detection model to obtain the region information of the target person;

[0109] A three-dimensional mapping module 303 is used to perform three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person to obtain the three-dimensional space information of the target person;

[0110] The safety warning module 304 is used to perform a safety warning operation when the three-dimensional space information overlaps with a preset three-dimensional safety frame.

[0111] In one embodiment, the target detection module 302 is used to:

[0112] Performing target detection on the intensity image based on a target detection model to obtain classification information of the target;

[0113] If the classification information of the target indicates that the target is a target person, then obtaining detection frame information of the target person;

[0114] Performing coordinate conversion according to the detection frame information of the target person and the preset detection frame information to obtain the target frame information of the detection frame of the target person on the intensity image;

[0115] Acquire segmentation information of the target person, where the segmentation information is used to indicate pixels corresponding to the target person;

[0116] According to the detection frame information and the target frame information, coordinate transformation is performed on the segmentation information to obtain the region information of the target person.

[0117] In one embodiment, the classification information of the target includes a confidence level and a classification result;

[0118] The target detection module 302 is configured to obtain detection frame information of the target person if the confidence level is higher than a preset threshold and the classification result indicates that the target is a target person.

[0119] In one embodiment, the three-dimensional mapping module 303 is used to:

[0120] Obtaining the two-dimensional coordinates of each pixel in the region information of the target person;

[0121] Determine the three-dimensional coordinates of each pixel in the corresponding area information of the target person in the three-dimensional point cloud data according to the two-dimensional coordinates;

[0122] The three-dimensional coordinates are integrated to obtain the three-dimensional spatial information of the target person.

[0123] In one embodiment, the security monitoring system 300 further includes: a security frame configuration module 304, which is used to:

[0124] Detect safe frame configuration instructions;

[0125] If a safety frame configuration instruction is detected, the preset three-dimensional safety frame is updated according to the safety frame configuration instruction.

[0126] In one embodiment, the security monitoring system 300 further includes: a model training module 305, which is used to:

[0127] An initial network model is trained based on an intensity image dataset to obtain a target detection model, wherein the output of the initial network model includes detection box information, classification information, and segmentation information of targets at multiple scales.

[0128] In one embodiment, the safety warning module 304 is used to:

[0129] When the three-dimensional space information overlaps with a preset three-dimensional safety frame, triggering an acoustic signal, an optical signal, or an acoustic-optical signal warning;

[0130] The preset three-dimensional safety frame is highlighted on the monitoring display interface.

[0131] In application, each module in the security monitoring system may be a software program module, or may be implemented through different logic circuits integrated in a processor, or may be implemented through multiple distributed processors.

[0132] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 4As shown, the terminal device 4 of this embodiment includes: at least one processor 40 ( Figure 4 Only one is shown in the figure) a processor, a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, wherein the processor 40 implements the steps in any of the above-mentioned security monitoring method embodiments when executing the computer program 42.

[0133] The terminal device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 4 and does not constitute a limitation on the terminal device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0134] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0135] In some embodiments, the memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. In other embodiments, the memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the terminal device 4. The memory 41 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0136] In the application, the terminal device 6 also includes: a communication module. The communication module can provide communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc., which are applied to network devices. The communication module can be one or more devices integrating at least one communication processing module. The communication module may include an antenna, which may have only one array element or an antenna array including multiple array elements. The communication module can receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module can also receive the signal to be sent from the processor, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0137] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0138] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0139] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0140] An embodiment of the present application provides a computer program product 5, which includes a computer program 50. When the computer program 50 is running, the steps in the above-mentioned security monitoring method embodiment are executed.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0144] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A security monitoring method, characterized in that: include: Collect intensity image data and three-dimensional point cloud data of the target area; Performing target detection and segmentation on the intensity image data based on a target detection model to obtain region information of the target person; Performing three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person to obtain the three-dimensional spatial information of the target person; When the three-dimensional space information overlaps with a preset three-dimensional safety frame, a safety warning operation is performed.

2. The security monitoring method according to claim 1, characterized in that: The target detection and segmentation of the intensity image data based on the target detection model to obtain the region information of the target person includes: Performing target detection on the intensity image based on a target detection model to obtain classification information of the target; If the classification information of the target indicates that the target is a target person, then obtaining detection frame information of the target person; Performing coordinate conversion according to the detection frame information of the target person and the preset detection frame information to obtain the target frame information of the detection frame of the target person on the intensity image; Acquire segmentation information of the target person, where the segmentation information is used to indicate pixels corresponding to the target person; According to the detection frame information and the target frame information, coordinate transformation is performed on the segmentation information to obtain the region information of the target person.

3. The security monitoring method according to claim 2, characterized in that: The classification information of the target includes confidence and classification result; If the classification information of the target indicates that the target is a target person, obtaining the detection frame information of the target person includes: If the confidence level is higher than a preset threshold and the classification result indicates that the target is a target person, the detection frame information of the target person is obtained.

4. The security monitoring method according to claim 2, characterized in that: The performing three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person to obtain the three-dimensional space information of the target person includes: Obtaining the two-dimensional coordinates of each pixel in the region information of the target person; Determine the three-dimensional coordinates of each pixel in the corresponding area information of the target person in the three-dimensional point cloud data according to the two-dimensional coordinates; The three-dimensional coordinates are integrated to obtain the three-dimensional spatial information of the target person.

5. The security monitoring method according to any one of claims 1 to 4, characterized in that: In the case where the three-dimensional space information overlaps with the preset three-dimensional safety frame, before performing the safety warning operation, the method further includes: Detect safe frame configuration instructions; If a safety frame configuration instruction is detected, the preset three-dimensional safety frame is updated according to the safety frame configuration instruction.

6. The security monitoring method according to claim 1, characterized in that: Before performing target detection and segmentation on the intensity image data based on the target detection model to obtain the region information of the target person, the method further includes: An initial network model is trained based on an intensity image dataset to obtain a target detection model, wherein the output of the initial network model includes detection box information, classification information, and segmentation information of targets at multiple scales.

7. The security monitoring method according to claim 1, characterized in that: When the three-dimensional space information overlaps with the preset three-dimensional safety frame, performing a safety warning operation includes: When the three-dimensional space information overlaps with a preset three-dimensional safety frame, triggering an acoustic signal, an optical signal, or an acoustic-optical signal warning; The preset three-dimensional safety frame is highlighted on the monitoring display interface.

8. A security monitoring system, characterized in that: include: A data acquisition module, used to acquire intensity image data and three-dimensional point cloud data of the target area; A target detection module, used to perform target detection and segmentation on the intensity image data based on a target detection model to obtain region information of a target person; A three-dimensional mapping module, used to perform three-dimensional mapping based on the three-dimensional point cloud data and the area information of the target person, and obtain the three-dimensional space information of the target person; The safety warning module is used to perform a safety warning operation when the three-dimensional space information overlaps with a preset three-dimensional safety frame.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which enables the method according to any one of claims 1 to 7 to be executed when the computer program is executed.

Images