Regional alarm method based on three-dimensional virtual fence and related device

Abstract

The invention discloses an area alarm method based on a three-dimensional virtual fence and a related device, and relates to the technical field of image analysis, and the method comprises the steps: determining an initial angle and an initial position of a camera based on the position information of a dangerous area, and collecting a first red, green and blue (RGB) image and a first depth image; performing image coverage analysis based on the first RGB image and the first depth image; adjusting an initial angle and an initial position based on comparison between the image coverage range data and a preset visual angle coverage standard; generating target point cloud data based on a second RGB image and a second depth image collected at the adjustment angle and the adjustment position in combination with a neural radiation field model; performing feature extraction based on the target point cloud data and the second RGB image to construct a three-dimensional virtual fence; and the dangerous area is monitored based on the three-dimensional virtual fence in combination with the alarm module. According to the invention, linkage of the three-dimensional virtual fence and the alarm module is formed, and complete closed-loop regional monitoring safety logic is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image analysis, in particular to a region alarm method based on a three-dimensional virtual fence and related devices. BACKGROUND

[0002] In the current human-robot collaborative scene of the industrial workshop, in order to protect the safety between the operating personnel and the robot device, the common protective measures include physical fences, safety grating and traditional camera monitoring systems.

[0003] However, the physical fence structure is rigid and has high modification cost, and is difficult to adapt to flexible production lines or variable operation units; the safety grating has non-contact triggering ability, but the deployment area is limited, and is easily affected by dust, water vapor and other factors, and has a high false triggering rate.

[0004] The traditional camera deployment usually relies on manual debugging, and is fixed by tripod, support or directly installed on the existing structure of the factory, and the adjustment mode is manual up and down movement or rotation. Due to the lack of real-time adjustment, it is easy to cause the dangerous area not to be covered or to have a blind area, and it is difficult to meet the current high-precision and high-efficiency deployment requirements, so that the region alarm of the workshop cannot achieve the expected effect. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, and the present application provides a region alarm method based on a three-dimensional virtual fence and related devices, which forms a linkage between the three-dimensional virtual fence and the alarm module, realizes a complete closed-loop region monitoring safety logic, and makes the region monitoring alarm of the workshop achieve a more ideal effect.

[0006] In order to solve the above technical problems, the present application provides a region alarm method based on a three-dimensional virtual fence, which comprises:

[0007] Determine the initial angle and initial position of the structured light camera based on the position information of the dangerous area, and the structured light camera collects a first red-green-blue (RGB) image and a first depth image at the initial angle and initial position;

[0008] Based on the first RGB image and the first depth image, the image coverage range is analyzed to obtain image coverage range data;

[0009] Compare the image coverage range data with the preset visual angle coverage standard, and adjust the initial angle and initial position of the structured light camera based on the comparison result to obtain the adjusted angle and adjusted position of the structured light camera;

[0010] The structured light camera head collects a second RGB image and a second depth image under an adjusted angle and an adjusted position, and generates target point cloud data based on the second RGB image and the second depth image in combination with a neural radiance field model;

[0011] Feature extraction is performed based on the target point cloud data and the second RGB image to obtain target feature data, and a three-dimensional virtual fence is constructed based on the target feature data;

[0012] The three-dimensional virtual fence is combined with an alarm module to monitor the dangerous area and determine whether an alarm information needs to be sent.

[0013] Optionally, the initial angle and the initial position of the structured light camera head are determined based on the dangerous area position information, comprising:

[0014] The dangerous area position and the dangerous area range are determined based on the dangerous area indication, and the initial angle and the initial position of the structured light camera head are matched based on the dangerous area position and the dangerous area range.

[0015] Optionally, the image coverage range data is obtained by performing image coverage range analysis based on the first RGB image and the first depth image, comprising:

[0016] The first RGB image and the first depth image are preprocessed to obtain preprocessed first RGB image and first depth image;

[0017] The target object recognition data is obtained by performing target object recognition based on the preprocessed first RGB image and the first depth image using a bilinear coding fusion network;

[0018] The object region position data is obtained by performing object region position analysis based on the target object recognition data using a graph-based image segmentation algorithm;

[0019] The image coverage range data is obtained by performing image coverage range analysis based on the target object recognition data and the object region position data.

[0020] Optionally, the initial angle and the initial position of the structured light camera head are adjusted based on a comparison between the image coverage range data and a preset visual angle coverage standard, and an adjusted angle and an adjusted position of the structured light camera head are obtained, comprising:

[0021] The image coverage range data is compared with the preset visual angle coverage standard, and if the image coverage range data does not meet the preset visual angle coverage standard, a deviation value of the image coverage range data from the preset visual angle coverage standard is calculated;

[0022] The first adjustment coefficient of the initial angle and the second adjustment coefficient of the initial position are matched based on the deviation value;

[0023] adjust the initial angle of the structured light camera based on the first adjustment coefficient to obtain an adjusted angle of the structured light camera, and adjust the initial position of the structured light camera based on the second adjustment coefficient to obtain an adjusted position of the structured light camera.

[0024] Optionally, the generating the target point cloud data based on the second RGB image and the second depth image in combination with the neural radiance field model comprises:

[0025] analyzing mapping data of two-dimensional image coordinates to three-dimensional space coordinates, and converting the second depth image into a depth map point cloud based on the mapping data;

[0026] extracting features of the second RGB image by using a segmentation model based on an attention mechanism to obtain RGB feature data, and generating a segmentation map based on the RGB feature data;

[0027] generating a neural point cloud based on the segmentation map by using the neural radiance field model;

[0028] determining target color information based on color information of the neural point cloud, and determining target position information based on position information of the neural point cloud and position information of the depth map point cloud;

[0029] performing point cloud fusion based on the target color information and the target position information to obtain the target point cloud data.

[0030] Optionally, the extracting features based on the target point cloud data and the second RGB image to obtain target feature data, and constructing a three-dimensional virtual fence based on the target feature data comprises:

[0031] extracting features of the second RGB image based on a two-dimensional convolutional neural network to obtain first feature data;

[0032] extracting features of the target point cloud data based on a three-dimensional convolutional neural network to obtain second feature data;

[0033] performing voxel filling and surface reconstruction based on the first feature data and the second feature data to obtain an initial three-dimensional dangerous area model;

[0034] optimizing the initial three-dimensional dangerous area model to obtain a target three-dimensional dangerous area model;

[0035] determining a first warning area and a second warning area based on the dangerous area drawing data, constructing a first virtual fence based on the first warning area in combination with the target three-dimensional dangerous area model, constructing a second virtual fence based on the second warning area in combination with the target three-dimensional dangerous area model, combining the first virtual fence and the second virtual fence to obtain the three-dimensional virtual fence.

[0036] Optionally, the monitoring of the dangerous area based on the three-dimensional virtual fence in combination with the alarm module judges whether an alarm information needs to be sent, comprising:

[0037] The three-dimensional virtual fence is connected to the alarm module, and when the alarm module monitors that a foreign matter or a human body enters the first virtual fence of the three-dimensional virtual fence, a first-level early warning information is sent;

[0038] When the alarm module monitors that a foreign matter or a human body enters the second virtual fence of the three-dimensional virtual fence, a second-level early warning information is sent.

[0039] In addition, the application also provides a three-dimensional virtual fence-based area alarm device, comprising:

[0040] The first image acquisition module is used to determine the initial angle and initial position of the structured light camera based on the dangerous area position information, and the structured light camera acquires a first red-green-blue (RGB) image and a first depth image at the initial angle and initial position;

[0041] The coverage analysis module is used to analyze the image coverage based on the first RGB image and the first depth image, and obtain image coverage data;

[0042] The camera adjustment module is used to compare the image coverage data with a preset visual angle coverage standard, and adjust the initial angle and initial position of the structured light camera based on the comparison result, and obtain an adjusted angle and an adjusted position of the structured light camera;

[0043] The point cloud generation module is used to acquire a second RGB image and a second depth image at the adjusted angle and adjusted position of the structured light camera, and generate target point cloud data based on the second RGB image and the second depth image in combination with a neural radiance field model;

[0044] The virtual fence construction module is used to extract features based on the target point cloud data and the second RGB image, obtain target feature data, and construct a three-dimensional virtual fence based on the target feature data;

[0045] The monitoring and alarm module is used to monitor the dangerous area based on the three-dimensional virtual fence in combination with the alarm module, and judge whether an alarm information needs to be sent.

[0046] In addition, the application also provides an electronic device, which comprises a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the above-mentioned three-dimensional virtual fence-based area alarm method.

[0047] In addition, the application further provides a computer readable storage medium storing computer instructions, which, when executed on an electronic device, causes the electronic device to perform the above-mentioned area alarm method based on the three-dimensional virtual fence.

[0048] In the embodiment of the application, image coverage range analysis is performed based on the first RGB image and the first depth image, comparison is performed between the image coverage range data and a preset visual angle coverage standard, and the initial angle and the initial position of the structured light camera are adjusted based on the comparison result, which improves the camera deployment efficiency and reduces the number of debugging trial and error. The target point cloud data is generated based on the second RGB image and the second depth image in combination with the neural radiance field model, which improves the accuracy of the generated point cloud data. Feature extraction is performed based on the target point cloud data and the second RGB image to obtain target feature data, and a three-dimensional virtual fence is constructed based on the target feature data, which improves the spatial integrity of the area coverage and the fence positioning accuracy. The dangerous area is monitored based on the three-dimensional virtual fence in combination with the alarm module, the linkage of the three-dimensional virtual fence and the alarm module is formed, the complete closed-loop area monitoring safety logic is realized, and the area monitoring and alarm of the workshop achieves a more ideal effect. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0050] Figure 1 is a flowchart of the area alarm method based on the three-dimensional virtual fence in the embodiment of the application;

[0051] Figure 2 is a flowchart of the area alarm method based on the three-dimensional virtual fence in another embodiment of the application;

[0052] Figure 3 is a structural composition schematic diagram of the area alarm device based on the three-dimensional virtual fence in the embodiment of the application;

[0053] Figure 4 is a structural composition schematic diagram of the electronic device in the embodiment of the application. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] Example 1

[0056] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for regional alarm based on a three-dimensional virtual fence according to an embodiment of the present invention, wherein the method includes:

[0057] S11: determining an initial angle and an initial position of a structured light camera based on the dangerous area position information, and collecting a first red, green, and blue (RGB) image and a first depth image at the initial angle and the initial position;

[0058] In the specific implementation process of the present invention, the initial angle and initial position of the structured light camera are determined based on the dangerous area position information, including: determining the dangerous area position and dangerous area range based on the dangerous area indication, and matching the initial angle and initial position of the structured light camera based on the dangerous area position and dangerous area range.

[0059] Specifically, the location and range of the dangerous area are determined based on the dangerous area indication, which is the dangerous area indicated by relevant personnel as requiring key monitoring. The initial angle and initial position of the structured light camera are matched based on the dangerous area location and dangerous area range. The structured light camera captures a first red-green-blue (RGB) image and a first depth image of the dangerous area requiring key monitoring at the initial angle and initial position.

[0060] S12: Performing image coverage analysis based on the first RGB image and the first depth image to obtain image coverage data;

[0061] In the implementation of the present application, the image coverage range analysis based on the first RGB image and the first depth image to obtain image coverage range data includes: pre-processing the first RGB image and the first depth image to obtain pre-processed first RGB image and first depth image; target object recognition based on the pre-processed first RGB image and the first depth image using a bilinear coding fusion network to obtain target object recognition data; object region position analysis based on the target object recognition data using a graph theory-based image segmentation algorithm to obtain object region position data; and image coverage range analysis based on the target object recognition data and the object region position data to obtain image coverage range data.

[0062] Specifically, the first RGB image and the first depth image are pre-processed, and the pre-processing includes noise reduction processing and image enhancement processing to obtain pre-processed first RGB image and first depth image. Target object recognition based on the pre-processed first RGB image and the first depth image using a bilinear coding fusion network, the first image features of the pre-processed first RGB image are extracted through the RGB image convolution layer of the bilinear coding fusion network, the second image features of the pre-processed first depth image are extracted through the depth image convolution layer, the first image features and the second image features are fused based on the bilinear fusion method to obtain fusion features, the sparse coefficients of the fusion features are determined based on sparse coding, the fusion features and the sparse coefficients are encoded into local aggregated descriptor vectors to obtain encoded local features, the local aggregated descriptor vector coding is a feature coding method used in image retrieval, video analysis and computer vision field, the corresponding feature vector is generated by aggregating the local feature descriptors, the encoded local features are aggregated and normalized to obtain global features, and target object recognition is performed based on the global features using the full connection layer to locate the target objects existing in the image, the target objects include robots, workbenches and their corresponding dangerous areas in the workshop, i.e. target object recognition data is obtained. The object region position data is obtained by dividing the target objects from the pre-processed first RGB image according to the target object recognition data to obtain a target object image, the image is segmented into regions with similar features by constructing a weighted graph model through the graph theory-based image segmentation algorithm, the contour information of the target objects is obtained by performing contour extraction of the target objects on the target object image through the graph theory-based image segmentation algorithm, and the area and position information of the occupied region are analyzed according to the contour point coordinates of the contour information of the target objects, i.e. the object region position data is obtained. The image coverage range data is obtained by inputting the target object recognition data and the object region position data into a pre-set neural network model for image coverage range analysis and comprehensively analyzing the positions and areas of all target objects in the image.

[0063] S13: comparing the image coverage range data with the preset visual angle coverage standard, and adjusting the initial angle and the initial position of the structured light camera based on the comparison result to obtain an adjusted angle and an adjusted position of the structured light camera;

[0064] In the specific implementation of the present application, the comparison of the image coverage range data with the preset visual angle coverage standard and the adjustment of the initial angle and the initial position of the structured light camera based on the comparison result to obtain the adjusted angle and the adjusted position of the structured light camera include: comparing the image coverage range data with the preset visual angle coverage standard, and if the image coverage range data does not meet the preset visual angle coverage standard, calculating the deviation value of the image coverage range data from the preset visual angle coverage standard; matching a first adjustment coefficient of the initial angle and a second adjustment coefficient of the initial position based on the deviation value; adjusting the initial angle of the structured light camera based on the first adjustment coefficient to obtain the adjusted angle of the structured light camera, and adjusting the initial position of the structured light camera based on the second adjustment coefficient to obtain the adjusted position of the structured light camera.

[0065] Specifically, the image coverage range data is compared with the preset visual angle coverage standard, which includes the data of the area standard coverage range, such as the area range of the dangerous area that should be displayed in the image, and if the image coverage range data meets the preset visual angle coverage standard, the initial angle and the initial position of the camera do not need to be adjusted, and the image collected by the initial angle and the initial position is directly used for subsequent analysis, and if the image coverage range data does not meet the preset visual angle coverage standard, the deviation value of the image coverage range data from the preset visual angle coverage standard is calculated. The first adjustment coefficient of the initial angle and the second adjustment coefficient of the initial position are matched based on the deviation value, and different adjustment coefficients can be matched according to different deviation values in the database, such as how much the camera is moved upward and how much the camera is rotated counterclockwise. The initial angle of the structured light camera is adjusted based on the first adjustment coefficient to obtain the adjusted angle of the structured light camera, such as rotating the camera clockwise by 15°, and the initial position of the structured light camera is adjusted based on the second adjustment coefficient, such as raising the camera by 10 cm, to obtain the adjusted position of the structured light camera. At the same time, when adjusting the angle and the position of the camera, historical deployment experience, spatial structure, and shielding information can also be referred to as parameters to assist in recommending deployment points for reference by relevant personnel.

[0066] S14: the structured light camera collects a second RGB image and a second depth image at the adjusted angle and the adjusted position, and generates target point cloud data based on the second RGB image and the second depth image combined with a neural radiance field model;

[0067] In the implementation of the present application, the generation of the target point cloud data based on the second RGB image and the second depth image in combination with the neural radiance field model comprises: analyzing mapping data of two-dimensional image coordinates to three-dimensional space coordinates, and converting the second depth image into a depth map point cloud based on the mapping data; performing feature extraction on the second RGB image by using a segmentation model based on an attention mechanism to obtain RGB feature data, and generating a segmentation map based on the RGB feature data; generating a neural point cloud based on the segmentation map by using the neural radiance field model; determining target color information based on color information of the neural point cloud, and determining target position information based on position information of the neural point cloud and position information of the depth map point cloud; and performing point cloud fusion based on the target color information and the target position information to obtain the target point cloud data.

[0068] Specifically, the structured light camera captures the second RGB image and the second depth image under the adjusted angle and the adjusted position, analyzes mapping data of two-dimensional image coordinates to three-dimensional space coordinates, and the expression of the mapping data is:

[0069]

[0070] Z = D (u, v) depth_scale,

[0071] Wherein, X is the X-axis coordinate in three-dimensional space, Y is the Y-axis coordinate in three-dimensional space, Z is the Z-axis coordinate in three-dimensional space, u is the horizontal axis coordinate in two-dimensional image, v is the vertical axis coordinate in two-dimensional image, cx and cy are the positions of the optical center of the camera in the image coordinate system, fx and fy are the focal length of the camera, depth_scale is the unit scaling factor, D(u,v) is the depth value of the pixel in the depth image, and the second depth image is converted into depth map point cloud data based on the mapping data, the depth image is projected into three-dimensional space according to the mapping data, forming point cloud data, which is the depth map point cloud. The segmentation model based on attention mechanism is used to extract features from the second RGB image, the second RGB image is input into the segmentation model based on attention mechanism for segmentation, and the segmented second RGB image is obtained. The mask image is extracted from the segmented second RGB image based on interactive segmentation. Interactive segmentation is to segment the target area in the image through user-computer interaction. In the interactive segmentation process, the computer generates a mask image according to the marked region of interest, extracts features according to the mask image, generates feature data, that is, obtains RGB feature data, and generates a segmentation map based on the RGB feature data. According to the point class prompt, the segmentation map is generated. The point class prompt represents the pixel position belonging to the point class region in the input data. According to the point class prompt, a segmentation line can be drawn on the image. For each pixel belonging to the point class, one or more line segments connected with the surrounding pixels are drawn to form a closed region, that is, the segmentation map is obtained. The segmentation map is input into the neural radiation field model to generate point cloud data, that is, the neural point cloud. The color information of the neural point cloud is determined as the color information of the fusion point cloud, and the target position information is determined based on the position information of the neural point cloud and the position information of the depth map point cloud. According to the position information of the neural point cloud and the position information of the depth map point cloud and the corresponding weight coefficient, the position information of the fusion point cloud is obtained, that is, the target position information. The point cloud fusion is carried out based on the target color information and the target position information. The neural point cloud and the depth map point cloud are fused through the target color information and the target position information, and the target point cloud data is obtained.

[0072] S15: Based on the target point cloud data and the second RGB image, feature extraction is performed to obtain target feature data, and a three-dimensional virtual fence is constructed based on the target feature data.

[0073] In the specific implementation process of the present invention, feature extraction is performed based on the target point cloud data and the second RGB image to obtain target feature data, and a three-dimensional virtual fence is constructed based on the target feature data, including: feature extraction of the second RGB image based on a two-dimensional convolutional neural network to obtain first feature data; feature extraction of the target point cloud data based on a three-dimensional convolutional neural network to obtain second feature data; voxel filling and surface reconstruction are performed based on the first feature data and the second feature data to obtain an initial three-dimensional dangerous area model; the initial three-dimensional dangerous area model is optimized to obtain a target three-dimensional dangerous area model; a first warning area and a second warning area are determined based on the dangerous area drawing data, and a first virtual fence is constructed based on the first warning area in combination with the target three-dimensional dangerous area model, and a second virtual fence is constructed based on the second warning area in combination with the target three-dimensional dangerous area model, and the first virtual fence and the second virtual fence are combined to obtain a three-dimensional virtual fence.

[0074] Specifically, the second RGB image is input into a two-dimensional convolutional neural network for feature extraction to obtain first feature data, which can be shape features, color features, corner features, etc. The target point cloud data is input into a three-dimensional convolutional neural network for feature extraction to obtain second feature data. The first feature data and the second feature data are input into a three-dimensional model software, the first feature data is mapped to a three-dimensional grid for surface reconstruction, and the second feature data is used for voxel filling and hole repair in the three-dimensional grid to obtain a three-dimensional model of the robot, the workbench and the warning area in the workshop working area, i.e., an initial three-dimensional dangerous area model. The initial three-dimensional dangerous area model is optimized by extracting texture features and color features from the second RGB image to obtain a target three-dimensional dangerous area model. The first warning area and the second warning area are determined based on the dangerous area drawing data, i.e., different warning areas are determined according to the region level information of the dangerous area. The first warning area is the area closest to the robot and the workbench, which is the area with the highest danger level. The second warning area is farther away from the robot and the workbench than the first warning area, which is adjacent to the first warning area. The first virtual fence is constructed based on the first warning area and the target three-dimensional dangerous area model. The target three-dimensional dangerous area model is input into the software, and the virtual fence is drawn in the model according to the first warning area, i.e., the first virtual fence. The second virtual fence is constructed based on the second warning area and the target three-dimensional dangerous area model. The construction method of the second virtual fence is the same as that of the first virtual fence, which will not be described here. The first virtual fence and the second virtual fence are combined to obtain a three-dimensional virtual fence.

[0075] S16: Monitoring the dangerous area based on the three-dimensional virtual fence and the alarm module to determine whether to send an alarm information.

[0076] In the specific implementation process of the present application, the monitoring of the dangerous area based on the three-dimensional virtual fence and the alarm module to determine whether to send an alarm information includes: connecting the three-dimensional virtual fence to the alarm module. If the alarm module detects that a foreign object or a human body enters the first virtual fence of the three-dimensional virtual fence, it sends a first-level warning information. If the alarm module detects that a foreign object or a human body enters the second virtual fence of the three-dimensional virtual fence, it sends a second-level warning information.

[0077] Specifically, the three-dimensional virtual fence is connected to the alarm module. If the alarm module monitors that a foreign object or a human body enters the first virtual fence of the three-dimensional virtual fence, a first-level warning information is sent. The first-level warning information is the highest level of warning information, which is an alarm information. After the first-level warning information is sent, the corresponding robot or workbench also stops working. If the alarm module monitors that a foreign object or a human body enters the second virtual fence of the three-dimensional virtual fence, a second-level warning information is sent. The second-level warning information is a warning prompt, such as a prompt that this area is a relatively dangerous area, please do not continue to advance.

[0078] In the embodiment of the present application, image coverage range analysis is performed based on the first RGB image and the first depth image, comparison is performed between the image coverage range data and a preset visual angle coverage standard, and the initial angle and the initial position of the structured light camera are adjusted based on the comparison result, which improves the camera deployment efficiency and reduces the number of debugging trial and error. Based on the second RGB image and the second depth image, the target point cloud data is generated by combining the neural radiance field model, which improves the accuracy of the generated point cloud data. Based on the target point cloud data and the second RGB image, feature extraction is performed to obtain target feature data, and a three-dimensional virtual fence is constructed based on the target feature data, which improves the spatial integrity of the area coverage and the fence positioning accuracy. Based on the three-dimensional virtual fence and the alarm module, the dangerous area is monitored, the linkage of the three-dimensional virtual fence and the alarm module is formed, the complete closed-loop area monitoring safety logic is realized, and the area monitoring alarm of the workshop achieves a more ideal effect.

[0079] Embodiment two

[0080] Please refer to Figure 2 , Figure 2 is a flowchart of a region alarm method based on a three-dimensional virtual fence in another embodiment of the present application. The method comprises:

[0081] S201: determining the initial angle and the initial position of the structured light camera based on the dangerous area position information, and the structured light camera collects a first red green blue (RGB) image and a first depth image at the initial angle and the initial position;

[0082] S202: preprocessing the first RGB image and the first depth image to obtain a preprocessed first RGB image and a preprocessed first depth image;

[0083] S203: performing target object recognition based on the preprocessed first RGB image and the preprocessed first depth image by using a bilinear coding fusion network to obtain target object recognition data;

[0084] S204: performing object region position analysis based on the target object recognition data by using a graph theory-based image segmentation algorithm to obtain object region position data;

[0085] S205: performing image coverage range analysis based on the target object recognition data and the object region position data to obtain image coverage range data;

[0086] S206: comparing the image coverage range data with a preset view angle coverage standard, and adjusting the initial angle and the initial position of the structured light camera based on the comparison result to obtain an adjusted angle and an adjusted position of the structured light camera;

[0087] S207: collecting a second RGB image and a second depth image under the adjusted angle and the adjusted position of the structured light camera, and generating target point cloud data based on the second RGB image and the second depth image in combination with a neural radiance field model;

[0088] S208: performing feature extraction based on the target point cloud data and the second RGB image to obtain target feature data, and constructing a three-dimensional virtual fence based on the target feature data;

[0089] S209: monitoring a dangerous region based on the three-dimensional virtual fence in combination with an alarm module to determine whether an alarm information needs to be sent.

[0090] In the embodiment of the present application, the image coverage range analysis is performed based on the first RGB image and the first depth image, the comparison is performed between the image coverage range data and the preset view angle coverage standard, and the initial angle and the initial position of the structured light camera are adjusted based on the comparison result, thereby improving the camera deployment efficiency and reducing the number of debugging trial and error. The target point cloud data is generated based on the second RGB image and the second depth image in combination with the neural radiance field model, thereby improving the accuracy of the point cloud data generation. The feature extraction is performed based on the target point cloud data and the second RGB image to obtain the target feature data, and the three-dimensional virtual fence is constructed based on the target feature data, thereby improving the spatial integrity of the region coverage and the fence positioning accuracy. The dangerous region is monitored based on the three-dimensional virtual fence in combination with the alarm module, thereby forming the linkage between the three-dimensional virtual fence and the alarm module, realizing the complete closed-loop region monitoring safety logic, and making the region monitoring alarm of the workshop achieve a more ideal effect.

[0091] Embodiment three

[0092] Please refer to Figure 3 , Figure 3 is a structural composition schematic diagram of a region alarm device based on a three-dimensional virtual fence in the embodiment of the present application, and the device comprises:

[0093] The first image acquisition module 31 is configured to determine the initial angle and the initial position of the structured light camera based on the dangerous region position information, and the structured light camera collects a first red green blue (RGB) image and a first depth image under the initial angle and the initial position;

[0094] The coverage analysis module 32 is configured to perform image coverage analysis based on the first RGB image and the first depth image to obtain image coverage data.

[0095] The camera adjustment module 33 is configured to compare the image coverage data with a preset view angle coverage standard, and adjust the initial angle and the initial position of the structured light camera based on the comparison result to obtain an adjusted angle and an adjusted position of the structured light camera.

[0096] The point cloud generation module 34 is configured to capture a second RGB image and a second depth image under the adjusted angle and the adjusted position of the structured light camera, and generate target point cloud data based on the second RGB image and the second depth image in combination with a neural radiance field model.

[0097] The virtual fence construction module 35 is configured to perform feature extraction based on the target point cloud data and the second RGB image to obtain target feature data, and construct a three-dimensional virtual fence based on the target feature data.

[0098] The monitoring and alarm module 36 is configured to perform monitoring of a dangerous area based on the three-dimensional virtual fence in combination with an alarm module, and determine whether to send an alarm information.

[0099] In the embodiment of the present application, the specific implementation of the device item can refer to the implementation of the above-mentioned method item, which will not be repeated here.

[0100] In the embodiment of the present application, the image coverage analysis is performed based on the first RGB image and the first depth image, the image coverage data is compared with the preset view angle coverage standard, and the initial angle and the initial position of the structured light camera are adjusted based on the comparison result, which improves the camera deployment efficiency and reduces the number of debugging and trial and error. The target point cloud data is generated based on the second RGB image and the second depth image in combination with the neural radiance field model, which improves the accuracy of the point cloud data generation. The target feature data is obtained by performing feature extraction based on the target point cloud data and the second RGB image, and the three-dimensional virtual fence is constructed based on the target feature data, which improves the spatial integrity of the area coverage and the fence positioning accuracy. The monitoring of the dangerous area is performed based on the three-dimensional virtual fence in combination with the alarm module, which forms the linkage of the three-dimensional virtual fence and the alarm module, realizes the complete closed-loop area monitoring safety logic, and makes the area monitoring and alarm of the workshop achieve a more ideal effect.

[0101] The computer readable storage medium provided by the embodiment of the present application stores a computer program, and the program is executed by a processor to realize the three-dimensional virtual fence based area alarm method in any one of the above embodiments. The computer readable storage medium includes, but is not limited to, any type of disk (including a floppy disk, a hard disk, an optical disk, a CD-ROM, and a magneto-optical disk), a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, a magnetic card, or an optical card. That is, the storage device includes any medium that stores or transmits information in a form capable of being read by an apparatus (for example, a computer, a mobile phone), and can be a read-only memory, a magnetic disk or an optical disk, etc.

[0102] Embodiment Four

[0103] Please refer to Figure 4 , Figure 4 is a structural composition schematic diagram of an electronic device in the embodiment of the present application.

[0104] The embodiment of the present application further provides an electronic device, as shown in Figure 4 , the electronic device includes a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and executable on the processor 43. Those skilled in the art can understand that Figure 3The electronic device shown does not constitute a limitation on all devices, and can include more or fewer components than shown, or combine some components. The memory 41 can be used to store computer programs 42 and various functional modules, and the processor 43 runs the computer programs 42 stored in the memory 41 to perform various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both the internal memory and the external memory. The internal memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random memory. The external memory can include a hard disk, a floppy disk, a ZIP disk, a U disk, a magnetic tape, etc. The processor 43 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip processor, or the processor 43 can also be any conventional processor, etc. The processor and the memory disclosed in the present application include but are not limited to these types of processors and memories. The processor and the memory disclosed in the present application are only examples and not limitations.

[0105] As an embodiment, the electronic device includes one or more processors 43, a memory 41, and one or more computer programs 42, wherein the one or more computer programs 42 are stored in the memory 41 and configured to be executed by the one or more processors 43, and the one or more computer programs 42 are configured to perform the three-dimensional virtual fence-based area alarm method in any one of the above embodiments. For specific implementation process, please refer to the above embodiments, which will not be repeated here.

[0106] In the embodiment of the present application, image coverage range analysis is performed based on the first RGB image and the first depth image, comparison is performed between the image coverage range data and a preset view angle coverage standard, and the initial angle and initial position of the structured light camera are adjusted based on the comparison result, thereby improving the camera deployment efficiency and reducing the number of debugging trial and error. The target point cloud data is generated based on the second RGB image and the second depth image in combination with the neural radiance field model, thereby improving the accuracy of the generated point cloud data. Feature extraction is performed based on the target point cloud data and the second RGB image to obtain target feature data, and a three-dimensional virtual fence is constructed based on the target feature data, thereby improving the spatial integrity of the area coverage and the fence positioning accuracy. The dangerous area is monitored based on the three-dimensional virtual fence in combination with the alarm module, thereby forming the linkage between the three-dimensional virtual fence and the alarm module, realizing the complete closed-loop area monitoring safety logic, and achieving a more ideal effect of the workshop area monitoring alarm.

[0107] In addition, the above describes in detail a kind of area alarm method and related device based on three-dimensional virtual fence provided by the embodiment of the present application, the principle and implementation mode of the present application are described in this paper using specific examples, the above embodiment is only for helping to understand the method of the present application and its core idea;Meanwhile, for the person skilled in the art, according to the idea of the present application, there will be changes in specific implementation mode and application range, and the above description should not be understood as the limitation of the present application.

Claims

1. A regional alarm method based on a three-dimensional virtual fence, characterized in that: The method comprises: Determine an initial angle and an initial position of the structured light camera based on the dangerous area position information, and collect a first red, green, and blue (RGB) image and a first depth image at the initial angle and initial position. Performing image coverage analysis based on the first RGB image and the first depth image to obtain image coverage data; Comparing the image coverage data with a preset viewing angle coverage standard, and adjusting an initial angle and an initial position of the structured light camera based on the comparison result to obtain an adjusted angle and an adjusted position of the structured light camera; The structured light camera captures a second RGB image and a second depth image while adjusting the angle and position, and generates target point cloud data based on the second RGB image and the second depth image in combination with the neural radiation field model; Performing feature extraction based on the target point cloud data and the second RGB image to obtain target feature data, and constructing a three-dimensional virtual fence based on the target feature data; Based on the three-dimensional virtual fence combined with the alarm module, dangerous areas are monitored to determine whether an alarm message needs to be issued.

2. The area alarm method based on three-dimensional virtual fence according to claim 1 is characterized in that: The determining of the initial angle and initial position of the structured light camera based on the dangerous area position information includes: The position and range of the dangerous area are determined based on the dangerous area indication, and the initial angle and initial position of the structured light camera are matched based on the position and range of the dangerous area.

3. The area alarm method based on three-dimensional virtual fence according to claim 1 is characterized in that: The performing image coverage analysis based on the first RGB image and the first depth image to obtain image coverage data includes: Preprocessing the first RGB image and the first depth image to obtain a preprocessed first RGB image and a first depth image; Based on the bilinear coding fusion network, the preprocessed first RGB image and the first depth image are used to perform target object recognition to obtain target object recognition data; The image segmentation algorithm based on graph theory uses the target object recognition data to perform object area position analysis and obtain object area position data; Image coverage analysis is performed based on the target object recognition data and the object area position data to obtain image coverage data.

4. The area alarm method based on three-dimensional virtual fence according to claim 1, characterized in that: The method of comparing the image coverage data with a preset viewing angle coverage standard and adjusting the initial angle and initial position of the structured light camera based on the comparison result to obtain the adjusted angle and adjusted position of the structured light camera includes: Comparing the image coverage data with a preset viewing angle coverage standard, and if the image coverage data does not meet the preset viewing angle coverage standard, calculating a deviation value between the image coverage data and the preset viewing angle coverage standard; Matching a first adjustment coefficient of the initial angle and a second adjustment coefficient of the initial position based on the deviation value; The initial angle of the structured light camera is adjusted based on the first adjustment coefficient to obtain the adjusted angle of the structured light camera, and the initial position of the structured light camera is adjusted based on the second adjustment coefficient to obtain the adjusted position of the structured light camera.

5. The area alarm method based on three-dimensional virtual fence according to claim 1, characterized in that: The generating target point cloud data based on the second RGB image and the second depth image in combination with the neural radiation field model includes: analyzing mapping data of two-dimensional image coordinates to three-dimensional space coordinates, and converting the second depth image into a depth map point cloud based on the mapping data; Using a segmentation model based on an attention mechanism to extract features from the second RGB image, obtaining RGB feature data, and generating a segmentation map based on the RGB feature data; Generate neural point cloud using segmentation map based on neural radiation field model; Determine target color information based on the color information of the neural point cloud, and determine target position information based on the position information of the neural point cloud and the position information of the depth map point cloud; Point cloud fusion is performed based on the target color information and target position information to obtain target point cloud data.

6. The area alarm method based on three-dimensional virtual fence according to claim 1, characterized in that: The method of extracting features based on the target point cloud data and the second RGB image to obtain target feature data, and constructing a three-dimensional virtual fence based on the target feature data, includes: Performing feature extraction on the second RGB image based on a two-dimensional convolutional neural network to obtain first feature data; Extract features of the target point cloud data based on a three-dimensional convolutional neural network to obtain second feature data; Perform voxel filling and surface reconstruction based on the first feature data and the second feature data to obtain an initial three-dimensional dangerous area model; Optimizing the initial three-dimensional hazardous area model to obtain the target three-dimensional hazardous area model; Based on the dangerous area drawing data, the first warning area and the second warning area are determined, and a first virtual fence is constructed based on the first warning area combined with the target three-dimensional dangerous area model. The second virtual fence is constructed based on the second warning area combined with the target three-dimensional dangerous area model. The first virtual fence and the second virtual fence are combined to obtain a three-dimensional virtual fence.

7. The area alarm method based on three-dimensional virtual fence according to claim 1, characterized in that: The monitoring of dangerous areas based on the three-dimensional virtual fence combined with the alarm module and determining whether an alarm message needs to be issued include: The three-dimensional virtual fence is connected to the alarm module. If the alarm module detects that a foreign object or a human body enters the first virtual fence of the three-dimensional virtual fence, it will issue a first-level warning message; If the alarm module detects that a foreign object or a human body has entered the second virtual fence of the three-dimensional virtual fence, a second-level warning message is issued.

8. A regional alarm device based on a three-dimensional virtual fence, characterized in that: The device comprises: A first image acquisition module is used to determine an initial angle and an initial position of the structured light camera based on the dangerous area position information, and the structured light camera acquires a first red, green, and blue (RGB) image and a first depth image at the initial angle and initial position; Coverage analysis module: used to perform image coverage analysis based on the first RGB image and the first depth image to obtain image coverage data; Camera adjustment module: used to compare the image coverage data with the preset view coverage standard, and adjust the initial angle and initial position of the structured light camera based on the comparison result to obtain the adjusted angle and adjusted position of the structured light camera; Point cloud generation module: used for the structured light camera to collect the second RGB image and the second depth image under adjusted angle and position, and generate target point cloud data based on the second RGB image and the second depth image combined with the neural radiation field model; A virtual fence construction module is used to extract features based on the target point cloud data and the second RGB image, obtain target feature data, and construct a three-dimensional virtual fence based on the target feature data; Monitoring and alarm module: used to monitor dangerous areas based on three-dimensional virtual fences combined with the alarm module to determine whether an alarm message needs to be issued.

9. An electronic device comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the area alarm method based on three-dimensional virtual fence according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the area alarm method based on a three-dimensional virtual fence according to any one of claims 1 to 7.

Images