Safety monitoring method and device based on region of interest, electronic equipment and storage medium

Abstract

The invention provides a safety monitoring method and device based on a region of interest, electronic equipment and a storage medium, and the method comprises the steps: obtaining the point cloud data of the region of interest in a depth image, the point cloud data comprising the depth value of each point cloud in the region of interest; based on the depth value of each point cloud, a depth discrete value of the region of interest is calculated, and the depth discrete value is used for representing the fluctuation amplitude of the depth change of each point cloud in the region of interest; and judging whether the depth dispersion rate is greater than or equal to a preset depth dispersion rate threshold, and if so, outputting a safety control signal. According to the invention, the point cloud data of the region of interest is obtained in the depth image, the depth discrete value of the region of interest is calculated by using the depth value of each point cloud, and when the depth discrete rate is greater than or equal to the preset depth discrete rate threshold, the safety control signal is output. The problems that an existing method is long in operation time and cannot accurately process point cloud on the rear side of a projection plane are solved.

Description

Technical Field

[0001] The present invention relates to the field of visual technology, and in particular to a safety monitoring method, device, electronic equipment and storage medium based on a focus area. Background Art

[0002] Stereo safety sensors, also known as Time-of-Flight (ToF) cameras or 3D TOF cameras, are mainly used for stereo protection in the industrial field. Their working principle is to calculate the distance information by measuring the time it takes for light to be emitted from the stereo safety sensor to the object in the detection area and then reflected back to the camera, thereby obtaining the three-dimensional spatial data of the scene. When a target object invades the detected area, it will output a safety control signal to the host computer or relay, thereby controlling the shutdown of the working equipment.

[0003] At present, when monitoring the area of ​​interest, the traditional method divides the area of ​​interest into a cube with the optical center O of the stereo safety sensor as the center and the vertices ABCDEFGH defined, and the six tetrahedrons (OABEH, OCDFG, OABCD, OEFGH, OADEF, OBCGH) in the cube as sub-areas, and then performs a two-dimensional plane projection on the point cloud in the area of ​​interest, and judges whether there is an intrusion of the target object based on the two-dimensional plane projection. Since it is necessary to process the multi-sided pyramid area connected to the optical center of the sensor, the computational complexity is high each time the two-dimensional projection surface is transformed, which leads to a long operation time and the problem of the point cloud behind the projection plane cannot be accurately processed. Summary of the invention

[0004] The embodiment of the present invention provides a safety monitoring method based on the area of ​​interest, which can accurately process the point cloud problem behind the projection plane. By obtaining the point cloud data of the area of ​​interest in the depth image, the depth value of each point cloud is used to calculate the depth discrete value of the area of ​​interest. When the depth discrete rate is greater than or equal to the preset depth discrete rate threshold, a safety control signal is output, which solves the problem that the existing method has a long operation time and cannot accurately process the point cloud problem behind the projection plane.

[0005] In a first aspect, an embodiment of the present invention provides a security monitoring method based on a region of interest, the method comprising the following steps:

[0006] Acquire point cloud data of a focus area in the depth image, the point cloud data including depth values ​​of each point cloud in the focus area, the focus area being a designated area in the depth image, and one point cloud corresponding to one pixel in the depth image;

[0007] Calculating a depth discrete rate of the region of interest based on the depth values ​​of each of the point clouds, wherein the depth discrete value is used to represent a fluctuation amplitude of a depth change of each of the point clouds in the region of interest;

[0008] It is determined whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, a safety control signal is output.

[0009] Optionally, the calculating the depth discrete rate of the area of ​​interest based on the depth value of each of the point clouds includes:

[0010] Based on the depth values ​​of each of the point clouds, calculating the average depth value of all point clouds in the area of ​​interest;

[0011] Based on the average depth value, calculating the depth standard deviation or depth variance corresponding to all point clouds in the region of interest;

[0012] A depth discreteness of the region of interest is determined based on the depth standard deviation or the depth variance.

[0013] Optionally, after outputting the safety control signal, the method further includes:

[0014] Calculating an absolute value of a difference between the depth discrete rate and the depth discrete rate threshold;

[0015] If the absolute value of the difference is greater than or equal to a preset absolute value threshold of the difference, the depth discrete rate threshold is updated.

[0016] Optionally, updating the depth discrete rate threshold includes:

[0017] Acquire a first historical depth image each time a safety control signal is output within a preset time period;

[0018] Calculating a first historical depth discrete rate corresponding to the area of ​​interest in each of the first historical depth images, each of the first historical depth images corresponding to one first historical depth discrete rate;

[0019] Based on the first historical depth image, obtaining a false trigger rate of the safety prompt;

[0020] The depth discrete rate threshold is updated based on the first historical depth discrete rate and the false trigger rate.

[0021] Optionally, the updating of the depth discrete rate threshold based on the first historical depth discrete rate and the false trigger rate includes:

[0022] Based on all the first historical depth dispersion rates, calculating an average historical depth dispersion rate;

[0023] Calculating a mean value between the average historical depth dispersion rate and the depth dispersion rate threshold;

[0024] Based on the mean value and the false trigger rate, a new depth discrete rate threshold is calculated.

[0025] Optionally, updating the depth discrete rate threshold includes:

[0026] Acquire a second historical depth image each time the safety prompt is falsely triggered and a third historical depth image each time the safety prompt is correctly triggered within a preset time period;

[0027] Calculate a second historical depth discrete rate corresponding to the focus area in each of the second historical depth images, each of the second historical depth images corresponding to one second historical depth discrete rate;

[0028] and calculating a third historical depth discrete rate corresponding to the focus area in each of the third historical depth images, each of the third historical depth images corresponding to one third historical depth discrete rate;

[0029] If the smallest third historical depth discrete rate is greater than the largest second historical depth discrete rate, updating the depth discrete rate threshold based on the average of the smallest third historical depth discrete rate and the largest second historical depth discrete rate;

[0030] If the smallest third historical depth discrete rate is smaller than the largest second historical depth discrete rate, the depth discrete rate threshold is updated based on the smallest third historical depth discrete rate.

[0031] Optionally, a shielding area is set in the area of ​​interest, and the method further includes:

[0032] If the target is only in the focus area, the point cloud data in the shielding area is eliminated;

[0033] If the target is in the concerned area and the target is in the shielding area, the point cloud data in the shielding area is protected.

[0034] In a second aspect, an embodiment of the present invention further provides a safety monitoring device based on a region of interest, the safety monitoring device based on a region of interest comprising:

[0035] An acquisition module, configured to acquire point cloud data of a focus area in a depth image, wherein the point cloud data includes a depth value of each point cloud in the focus area, wherein the focus area is a designated area in the depth image, and one point cloud corresponds to one pixel in the depth image;

[0036] A calculation module, used to calculate the depth discrete rate of the focus area based on the depth value of each point cloud, wherein the depth discrete value is used to represent the fluctuation amplitude of the depth change of each point cloud in the focus area;

[0037] The judging module is used to judge whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, output a safety control signal.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the security monitoring method based on the area of ​​interest provided in an embodiment of the present invention are implemented.

[0039] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the security monitoring method based on the area of ​​interest provided in the embodiment of the invention are implemented.

[0040] In an embodiment of the present invention, point cloud data of a region of interest is obtained in a depth image, the point cloud data includes depth values ​​of each point cloud in the region of interest, the region of interest is a designated region in the depth image, and one point cloud corresponds to one pixel in the depth image; based on the depth values ​​of each point cloud, a depth discrete value of the region of interest is calculated, the depth discrete value is used to represent the fluctuation amplitude of the depth change of each point cloud in the region of interest; it is determined whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, a safety control signal is output. The present invention obtains point cloud data of a region of interest in a depth image, and uses the depth values ​​of each point cloud to calculate the depth discrete value of the region of interest, and outputs a safety control signal when the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, thereby solving the problem that the existing method has a long operation time and cannot accurately process the point cloud behind the projection plane. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a security monitoring method based on a focus area provided by an embodiment of the present invention;

[0043] Figure 2 is a structural schematic diagram of a safety monitoring device based on a region of interest provided by an embodiment of the present invention;

[0044] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, Figure 1 : is a flow chart of a security monitoring method based on a region of interest provided by an embodiment of the present invention, the security monitoring method based on the region of interest comprises the steps of:

[0047] 101. Obtain point cloud data of the area of ​​interest in the depth image.

[0048] In an embodiment of the present invention, the security monitoring method can be applied to a security monitoring system, which includes a host computer and a point cloud data acquisition device, which can be a three-dimensional camera, a depth camera or a time-of-flight camera, such as a stereo security sensor. The point cloud data acquisition device is used to collect point cloud data of the corresponding area.

[0049] The above security monitoring method is mainly used for the host computer of the security monitoring system. The above point cloud data can be understood as the three-dimensional coordinate data (x, y, z) of each pixel point in the depth map collected by the stereo security sensor.

[0050] The above-mentioned point cloud data can also be called 3D point cloud data. The collection of point cloud data in this embodiment can be based on the iTOF ranging principle. Through CW modulation driven by VCSEL, VCSEL modulates light with a frequency of f. Because it is necessary to analyze the phase difference between the reflected light and the emitted light to calculate the distance, in order to avoid multiple solutions, only the distance of the target within 1 / 2 wavelength can be measured, that is, the maximum ranging is 1 / 2 wavelength of the modulated light wave. The two frequencies of 15Mhz and 120Mhz adopted in this example, the theoretical maximum detection distance at 15Mhz frequency is 10m, and the theoretical maximum detection distance at 120Mhz frequency is 1.25m. The theoretical maximum detection distance of dual-frequency rotation is the least common multiple of the theoretical maximum detection distances corresponding to the two frequencies, that is, the greatest common multiple of 1.5m and 10m, that is, the theoretical maximum detection distance under the conditions of 15Mhz and 120Mhz rotation is 10m.

[0051] The distortion problem of the depth map is solved one by one through the camera intrinsic calibration. In the intrinsic calibration module, the common "pinhole + distortion" model in computer vision can be used to characterize the imaging system parameters such as lens distortion and principal point offset. The parameters include:

[0052] 1. Main point c x 、c y , represents the pixel coordinates of the intersection of the lens optical axis and the imaging target surface.

[0053] 2. Focal length f x 、f y , which represents the ratio of the lens focal length to the pixel size.

[0054] 3. Distortion parameters k1, k2, k 3； , p1, p2, where k1, k2, k3 are parameters representing the radial distortion of the lens, and p1 and p2 are parameters representing the tangential distortion of the lens. In the internal parameter calibration module, the calibration method is used to solve the model parameters. The camera is controlled to collect the intensity image of the calibration plate. Given the coordinates of the marking points in the calibration plate, the features of the marking points in the image are extracted to obtain the image coordinates of the marking points, the objective function is established, and the LM method is used for optimization and solution. Finally, the principal point c is obtained. x 、c y , focal length f x 、f y , and distortion parameters k1, k2, k3, p1, p2, and write the calibration results into the camera.

[0055] During the depth calibration (phase calibration) process, the measurement accuracy of TOF is affected by many factors. The depth calibration module mainly considers three items:

[0056] 1. When measuring an object with a distance of zero, when a zero-phase signal is transmitted, the delay difference between the phase of the signal received by RX (RF receiver) and the zero-phase signal is called zero drift.

[0057] 2. Characterize the zero drift that indicates the difference in the initial exposure time of different pixels, i.e., the fixed phase template noise.

[0058] 3. The deviation between the light waveform emitted by the TX (radio frequency transmitter) and the cosine signal causes harmonic errors that vary with distance.

[0059] The absolute errors introduced by the above items are usually in the order of centimeters. To ensure the accuracy of the distance measurement results, the above errors of each stereo safety sensor must be calibrated one by one before leaving the factory.

[0060] In order to calibrate the above errors, it is necessary to know the real distance of the object to be measured, collect TOF images at the same time, and calculate the actual measured distance. An error model is established between the two, and the model parameters are solved. In specific implementation, the stereo safety sensor is placed on the track table, and 90% diffuse reflection planes are photographed at different distances to collect TOF data. The distance between the stereo safety sensor and the diffuse reflection plane given by the track table and the internal parameter calibration model are used to calculate the real distance of each pixel from the diffuse reflection plane. Call the depth calibration algorithm to process the collected TOF data, calculate the actual measurement distance, and optimize the model parameters.

[0061] The depth calculation algorithm completes the calculation from the raw image to the tap image, the tap image to the intensity image, the amplitude image, the phase image, the phase image to the distance image, the distance image to the depth image, and the depth image to the point cloud image. The depth calculation algorithm also has the following two functions:

[0062] 1. In the process of calculating the phase, the depth calculation algorithm needs to call the calibration results to compensate the calculated phase to eliminate the system error and obtain higher measurement accuracy.

[0063] 2. Because the stereo safety sensor works in dual-frequency mode, the depth calculation algorithm needs to fuse the dual-frequency data and output a higher quality measurement result after fusion.

[0064] The depth filtering algorithm is effective at different stages of the depth calculation algorithm, processing the intermediate quantities of the calculation, filtering out errors and low signal-to-noise ratio results, and ensuring the quality of the output results. The depth filtering algorithm consists of five modules:

[0065] 1. Temporal filtering module: averages multiple frames of the depth map to improve the signal-to-noise ratio. You can set the temporal filtering algorithm on and off, as well as the fusion weight of the current frame.

[0066] 2. Spatial filtering module: Perform spatial Gaussian filtering on the 1Q image to improve the signal-to-noise ratio. You can set the spatial filtering algorithm on and off, as well as the size of the Gaussian filtering window.

[0067] 3. Amplitude filter module: Set a certain threshold to filter out pixel depths with amplitudes below this threshold. You can set the amplitude filter algorithm on and off, as well as the amplitude filter threshold.

[0068] 4. Fusion error filter module: Calculate the dual-frequency fusion error. When the phase is greater than the set value, filter out the corresponding pixel depth. You can set the fusion error filter algorithm to open or close, as well as the size of the interval.

[0069] 5. Flying point filter module: convert the depth into point cloud, calculate the distance between the point and the surrounding points, and filter out the corresponding pixel depth when the minimum distance is greater than the set value. You can set the flying point filter to be on or off, as well as the size of the distance value.

[0070] The following is an example of point cloud generation:

[0071] Through CW modulation driven by VCSEL, VCSEL modulates two frequencies of light, 15MHz and 120MHz, respectively. The two frequencies of light are exposed under the illumination conditions to obtain energy integral images of four phases of 0°, 180°, 90°, and 270° at each frequency. Therefore, 8 energy integral images need to be collected together:

[0072] Rawdata120_0: 0° phase energy integration diagram under 120Mhz frequency modulated light.

[0073] Rawdata120_90: 90° phase energy integration diagram under 120Mhz frequency modulated light.

[0074] Rawdata120_180: 180° phase energy integration diagram under 120Mhz frequency modulated light.

[0075] Rawdata120_270: 270° phase energy integration diagram under 120Mhz frequency modulated light.

[0076] Rawdata15_0: 0° phase energy integration diagram under 15Mhz frequency modulated light.

[0077] Rawdata15_90: 90° phase energy integration diagram under 15Mhz frequency modulated light.

[0078] Rawdata15_180: 180° phase energy integration diagram under 15Mhz frequency modulated light.

[0079] Rawdata15_270: 270° phase energy integral diagram under 15Mhz frequency modulated light.

[0080] Dual frequency rotation:

[0081] 1. First, use 120MHz to take 4 pictures. Each picture has 2 phases (2 taps). The exposure time of each picture is the set exposure value (eg. the exposure is set to 1000us, 4 times of taking pictures is 4000us, and there is a waiting time between two pictures, the minimum is about 4ms).

[0082] 2. Then 15MHz, take 4 pictures, each picture has 2 phases (2 taps), and the exposure time of each picture is the set exposure value (eg. the exposure is set to 1000us, 4 photos are taken for 4 times, and there is a waiting time between two pictures, the minimum is about 4ms).

[0083] 3. At each modulation frequency, the tap phase sequence of the four figures is:

[0084] The tap sequence is (0°, 180°) → (90°, 270°) → (180°, 0°) → (270°, 90°).

[0085] The value of each pixel of Rawdata at a certain frequency and phase represents the integral of the target reflected energy at the direction corresponding to this pixel. For the same pixel, the integral energy Q0, Q 90 , Q 180 , Q 270 , the phase offset value of this point can be calculated

[0086]

[0087] Q0 represents the energy integral of the 0 phase of the pixel.

[0088] Q 90 Represents the energy integral of the 0 phase of the pixel.

[0089] Q 180 Represents the energy integral of the 0 phase of the pixel.

[0090] Q 270 Represents the energy integral of the 0 phase of the pixel.

[0091] According to the phase of the pixel The detection distance at this frequency can be calculated, and the corresponding distances d1 and d2 can be calculated for the two frequencies respectively.

[0092]

[0093] The final distance is obtained by dual-frequency solution.

[0094] (1) Calculate the dual-frequency integrated frequency.

[0095] f_max is the greatest common divisor of frequency f1 and frequency f2: when f1 = 15Mhz, f2 = 120Mhz, f_max = 15Mhz

[0096] (2) Calculate M_f1 and M_f2 as follows:

[0097] M_f1=f1 / f_max

[0098] M_f2=f2 / f_max

[0099] Calculate A_f1, A_f2 according to the following formula:

[0100]

[0101] Calculate ω according to the following formula:

[0102]

[0103] The final dual-frequency solution distance is:

[0104] d=d1·ω+d2·(1-ω)

[0105] Depth map generation:

[0106] ①According to the results of phase calibration, the relationship between the actual detected phase and the theoretical phase is obtained;

[0107] ② Based on the current actual detected phase, the current true phase can be calculated and converted into the calculated distance, which is the actual detection distance value.

[0108] Depth map converted into point cloud:

[0109] (1) Based on the calibration of the camera’s intrinsic parameters, the azimuth angle corresponding to each pixel can be obtained.

[0110] (2) Based on the depth d and azimuth, the three-dimensional coordinates x, y, and z can be calculated.

[0111] The above point cloud data includes the depth value of each point cloud in the area of ​​interest. The depth value refers to the distance from each point in the point cloud to the camera or sensor, and the depth value can directly reflect the geometric shape of the object surface. The above area of ​​interest is a specified area in the depth image, and one point cloud corresponds to a pixel in the depth image. Each pixel in the depth image corresponds to a point cloud. It can be understood that if there is a depth image of size M*N, MN point cloud data of the area of ​​interest are obtained from the M*N depth image, and each point corresponds to a pixel position in the depth image. Depth

[0112] The above-mentioned depth image is used to reflect the three-dimensional structure of the scene, and the depth image records the distance information from each pixel to the camera or sensor.

[0113] The above mentioned region of interest may be predefined according to the application scenario, and may be a specific structure or detail, etc. The above mentioned region of interest may also be referred to as a ROI region.

[0114] The above point cloud data can be understood as a collection of points in three-dimensional space, each of which has its corresponding position coordinates. In a depth image, each pixel can be mapped to a point in three-dimensional space, and the depth value of the point directly corresponds to the distance measured by the pixel.

[0115] It should be noted that obtaining point cloud data of the area of ​​interest in the depth image can reflect the three-dimensional structure of the scene.

[0116] In a possible embodiment, before executing the above step 101, the region of interest may be detected first to avoid abnormal detection of the region of interest. Specifically, the region of interest may be calibrated by the following steps:

[0117] Step 1: For each area of ​​interest, extract the coordinate data of all points in the area of ​​interest, and calculate the centroid coordinates and point repetition accuracy threshold.

[0118] Step 2: In the current frame, for each point in the area of ​​interest, calculate the centroid coordinates and repeatability of all points in the area of ​​interest.

[0119] Step 3. Determine whether the centroid coordinates of all areas of concern are within the point safety coordinate range. If the centroid coordinates of any area of ​​concern are within the point safety coordinate range of the area of ​​concern or the repetition accuracy of any area of ​​concern does not exceed the point repetition accuracy threshold of the area of ​​concern, it means that there are no unacceptable changes in all areas of concern, all areas of concern are normal, and OSSD outputs safety signals.

[0120] Step 4: If the centroid coordinates of any area of ​​concern exceed the point safety coordinate range of the area of ​​concern, or the repetition accuracy of any area of ​​concern exceeds the point repetition accuracy threshold of the area of ​​concern, it means that there is an unacceptable change and an abnormal OSSD output danger signal exists in the area of ​​concern.

[0121] Step 5: The OSSD front output is a danger signal. The reset signal is currently received, and all the focus areas in the current frame are normal. The OSSD resets and outputs a safety signal.

[0122] It should be noted that the above-mentioned OSSD can be understood as the abbreviation of output signal switching device (Output Signal Switching Device) or optical signal switching device (Optical Signal Switching Device). The above-mentioned safety control signal may include a safety signal, a danger signal and a safety signal after reset. By transmitting the safety control signal to the host computer or relay, the working equipment can be controlled to stop or start working accordingly.

[0123] 102. Based on the depth values ​​of each point cloud, a discrete depth value of the focus area is calculated.

[0124] In the embodiment of the present invention, the above-mentioned depth discrete value is used to indicate the fluctuation amplitude of the depth change of each point cloud in the focus area. The fluctuation amplitude is used to indicate the degree of change of the depth value in the focus area. If the depth discrete value is larger, the depth value in the focus area changes more drastically, that is, there is a large fluctuation; if the depth discrete value is smaller, it means that the depth value in the focus area changes more slowly, that is, the fluctuation amplitude is smaller.

[0125] Specifically, the depth discrete value of the focus area can be obtained by calculating the standard deviation of the depth values ​​of all point clouds in the focus area. The larger the depth discrete value, the more drastic the depth change in the focus area; the smaller the depth discrete value, the gentler the depth change in the focus area.

[0126] The depth discrete value of the focus area can be obtained by calculating the variance of the depth values ​​of all point clouds in the focus area. The larger the depth discrete value, the more drastic the depth change in the focus area; the smaller the depth discrete value, the gentler the depth change in the focus area.

[0127] 103. Determine whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold; if so, output a safety control signal.

[0128] In the embodiment of the present invention, the above-mentioned preset depth discreteness threshold is a pre-set depth discreteness threshold, which is used to compare with the depth discreteness of the region of interest. The depth discreteness is an indicator to measure the fluctuation amplitude of the depth change in the region of interest, which can be obtained by calculating the difference in depth values ​​between different points in the region.

[0129] If the depth discrete rate is greater than or equal to the preset depth discrete rate threshold, it means that there may be potential risks or unstable factors in the system. Therefore, it is necessary to output a safety control signal to alert the user or administrator and take corresponding measures.

[0130] The above safety control signals can be used as safety prompts in the form of voice, flashing warning lights, etc. to ensure the stability and safety of the system.

[0131] In an embodiment of the present invention, point cloud data of a region of interest is obtained in a depth image, the point cloud data includes depth values ​​of each point cloud in the region of interest, the region of interest is a designated region in the depth image, and one point cloud corresponds to one pixel in the depth image; based on the depth values ​​of each point cloud, a depth discrete value of the region of interest is calculated, the depth discrete value is used to represent the fluctuation amplitude of the depth change of each point cloud in the region of interest; it is determined whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, a safety control signal is output. The present invention obtains point cloud data of a region of interest in a depth image, and uses the depth values ​​of each point cloud to calculate the depth discrete value of the region of interest, and outputs a safety control signal when the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, thereby solving the problem that the existing method has a long operation time and cannot accurately process the point cloud behind the projection plane.

[0132] It is understandable that in the specific implementation of this application, image data, task data, equipment data and other related data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0133] Optionally, in the step of calculating the depth discrete rate of the area of ​​interest based on the depth values ​​of each point cloud, the average depth value of all point clouds in the area of ​​interest can be calculated based on the depth values ​​of each point cloud; based on the average depth value, the depth standard deviation or depth variance corresponding to all point clouds in the area of ​​interest can be calculated; based on the depth standard deviation or depth variance, the depth discrete rate of the area of ​​interest can be determined.

[0134] In the embodiment of the present invention, the above average depth value is obtained by summing the depth values ​​of all point clouds in the region of interest and then dividing it by the number of points in the region of interest to obtain an average value, which is the average depth value. The average depth value reflects the overall depth of the point cloud in the region of interest. The average depth value represents the central trend of the depth of the point cloud in the region.

[0135] The above depth standard deviation measures the dispersion of the depth values ​​of the point cloud, reflecting the distance of the data points from the mean. A larger depth standard deviation means that the depth values ​​are more dispersed in the area of ​​interest, while a smaller depth standard deviation means that the depth values ​​are more concentrated or similar.

[0136] The above-mentioned depth variance measures the dispersion of the depth values ​​of each point cloud in the focus area relative to the average depth value. The depth variance can reflect the fluctuation of the depth information.

[0137] For each point cloud, the difference between the depth value of the point cloud and the average value is calculated, and the standard deviation or variance of the difference between the depth value of the point cloud and the average value is calculated to obtain the depth standard deviation or depth variance corresponding to all point clouds in the focus area.

[0138] The above depth dispersion rate is used to quantify the inconsistency or dispersion of depth information in the region of interest. A larger depth dispersion rate means that the surface of the region of interest has complex textures, irregular shapes, or is in sharp contrast with other regions. A smaller depth dispersion rate means that the surface of the region of interest is relatively flat and the contrast with other regions is not obvious.

[0139] Optionally, after the step of outputting the safety control signal, the absolute value of the difference between the depth discrete rate and the depth discrete rate threshold may be calculated; if the absolute value of the difference is greater than or equal to a preset absolute value of the difference threshold, the depth discrete rate threshold is updated.

[0140] In the embodiment of the present invention, the depth discrete rate threshold is a depth discrete rate threshold preset by the system.

[0141] The degree of difference between the depth discreteness and the depth discreteness threshold can be obtained by the absolute value of the difference between the depth discreteness and the depth discreteness threshold. The absolute value of the difference is the absolute degree of difference between the two.

[0142] Specifically, if the calculated absolute value of the difference exceeds the preset depth discrete rate threshold, it means that the current depth discrete rate has exceeded the preset depth discrete rate threshold, and there may be a safety hazard. Therefore, the depth discrete rate threshold can be adjusted or updated according to this situation to make it more stringent or sensitive, so that similar safety issues can be detected earlier in the future, so that the system can adapt to changes in the environment and improve its safety and stability.

[0143] In one possible embodiment, for example, if the depth discrete rate is 0.5 and the preset threshold is 0.3, the difference is 0.2. If the calculated difference is 0.2 and the preset difference absolute value threshold is 0.1, the update condition is not met. However, if the difference increases to 0.4, it exceeds the threshold, and the system will update the depth discrete rate threshold to prevent similar situations in the future.

[0144] Optionally, in the step of updating the depth discrete rate threshold, a first historical depth image can be obtained each time a safety control signal is output within a preset time period; the first historical depth discrete rate corresponding to the focus area in each first historical depth image is calculated; based on the first historical depth image, the false trigger rate of the safety prompt is obtained; based on the first historical depth image, the depth discrete rate threshold is updated.

[0145] In the embodiment of the present invention, each first historical depth image corresponds to a first historical depth discrete rate.

[0146] The above-mentioned preset time period is a time period preset by the system, which may be 10ms, 15ms, 20ms, etc.

[0147] The above-mentioned first historical depth image can be understood as the first historical depth image recorded each time a safety control signal is output within a preset time period.

[0148] The first historical depth discrete rate may be understood as a degree of variation of depth measurement values ​​within the region of interest. The standard deviation or coefficient of variation of the depth values ​​within the region of interest may be calculated as a measure of the discrete rate.

[0149] The false trigger rate refers to the frequency at which the safety prompt is falsely triggered, and can be calculated by analyzing the safety prompt record of the first historical depth image.

[0150] It should be noted that if it is found that the depth variation in a certain area is large but the false trigger rate is high, the depth discrete rate threshold of the area may be adjusted accordingly to reduce unnecessary safety prompts and improve the overall performance of the system and user experience.

[0151] Optionally, in the step of updating the depth discrete rate threshold based on the first historical depth discrete rate and the false trigger rate, the average historical depth discrete rate can be calculated based on all first historical depth discrete rates; the mean between the average historical depth discrete rate and the depth discrete rate threshold is calculated; and based on the mean and the false trigger rate, a new depth discrete rate threshold is calculated.

[0152] In the embodiment of the present invention, the average historical depth dispersion rate is obtained by calculating the average value of all first historical depth dispersion rates, and reflects the overall dispersion degree of the first historical depth image.

[0153] The purpose of calculating the mean value between the average historical depth dispersion rate and the depth dispersion rate threshold is to evaluate the difference or similarity between the average historical depth dispersion rate and the depth dispersion rate threshold.

[0154] Specifically, first, the average historical depth discrete rate is calculated based on all the first historical depth discrete rates, then the mean between the average historical depth discrete rate and the depth discrete rate threshold is calculated, and finally, a new depth discrete rate threshold is calculated based on the mean between the average historical depth discrete rate and the depth discrete rate threshold and the false trigger rate.

[0155] The new depth discrete rate threshold can more accurately reflect the actual situation of the data and reduce the possibility of false triggering.

[0156] Optionally, in the step of updating the depth discrete rate threshold, a second historical depth image for each time the safety prompt is falsely triggered and a third historical depth image for each time the safety prompt is correctly triggered within a preset time period can be obtained; the second historical depth discrete rate corresponding to the focus area in each second historical depth image is calculated; and the third historical depth discrete rate corresponding to the focus area in each third historical depth image is calculated; if the minimum third historical depth discrete rate is greater than the maximum second historical depth discrete rate, the depth discrete rate threshold is updated based on the average between the minimum third historical depth discrete rate and the maximum second historical depth discrete rate; if the minimum third historical depth discrete rate is less than the maximum second historical depth discrete rate, the depth discrete rate threshold is updated based on the minimum third historical depth discrete rate.

[0157] In the embodiment of the present invention, each second historical depth image corresponds to a second historical depth discrete rate, and each third historical depth image corresponds to a third historical depth discrete rate.

[0158] For each false triggering of the safety prompt within a preset time period, the second historical depth image and the third historical depth image are obtained, and the second historical depth discrete rate of the focus area in the second historical depth image and the third historical depth discrete rate of the focus area in the third historical depth image are calculated. If the minimum third historical depth discrete rate is greater than the maximum second historical depth discrete rate, the depth discrete rate threshold is updated by the average of the minimum third historical depth discrete rate and the maximum second historical depth discrete rate; if the minimum third historical depth discrete rate is less than the maximum second historical depth discrete rate, the depth discrete rate threshold is updated according to the minimum third historical depth discrete rate.

[0159] It should be noted that by analyzing the second historical depth image and its corresponding discrete rate and the third historical depth image and its corresponding discrete rate, the depth discrete rate threshold is dynamically adjusted to more accurately determine when the safety prompt should be triggered.

[0160] Optionally, a shielding area is set in the area of ​​interest. If the target is only in the area of ​​interest, the point cloud data in the shielding area is eliminated; if the target is in the area of ​​interest and the target is in the shielding area, the point cloud data in the shielding area is protected.

[0161] In an embodiment of the present invention, taking a robot or a robotic arm as an example, the working area of ​​the robot or the robotic arm is a shielding area, and the area outside the shielding area is a protection area or an alarm area. When point cloud data appears in the protection area or the alarm area, it means that an object has invaded the protection area or the alarm area from the shielding area during the working process of the robot or the robotic arm, and a safety control signal can be output to the host computer or computer, thereby controlling the corresponding working equipment to stop working.

[0162] It should be noted that when the target is only in the focus area, we can remove the point cloud data in the shielded area to reduce the amount of calculation. When the target is in the focus area and the target is in the shielded area, the point cloud data in the shielded area may not be removed to protect the integrity of the point cloud data and improve the accuracy of target detection.

[0163] like Figure 2 As shown, an embodiment of the present invention provides a safety monitoring device based on a region of interest, and the safety monitoring device based on a region of interest includes:

[0164] An acquisition module 201 is used to acquire point cloud data of a focus area in a depth image, wherein the point cloud data includes a depth value of each point cloud in the focus area, wherein the focus area is a designated area in the depth image, and one point cloud corresponds to one pixel in the depth image;

[0165] A first calculation module 202 is used to calculate the depth discrete rate of the focus area based on the depth value of each point cloud, and the depth discrete value is used to represent the fluctuation amplitude of the depth change of each point cloud in the focus area;

[0166] The judging module 203 is used to judge whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, output a safety control signal.

[0167] Optionally, the first calculation module 202 is also used to calculate the average depth value of all point clouds in the focus area based on the depth value of each point cloud; based on the average depth value, calculate the depth standard deviation or depth variance corresponding to all point clouds in the focus area; based on the depth standard deviation or the depth variance, determine the depth discrete rate of the focus area.

[0168] Optionally, after the outputting of the safety control signal, the device further comprises:

[0169] A second calculation module, used to calculate the absolute value of the difference between the depth discrete rate and the depth discrete rate threshold;

[0170] An updating module is configured to update the depth discreteness threshold if the absolute value of the difference is greater than or equal to a preset absolute value threshold of the difference.

[0171] Optionally, the update module is also used to obtain a first historical depth image each time a safety control signal is output within a preset time period; calculate a first historical depth discrete rate corresponding to the focus area in each of the first historical depth images, and each first historical depth image corresponds to one first historical depth discrete rate; based on the first historical depth image, obtain a false trigger rate of the safety prompt; based on the first historical depth discrete rate and the false trigger rate, update the depth discrete rate threshold.

[0172] Optionally, the update module is also used to calculate an average historical depth discrete rate based on all the first historical depth discrete rates; calculate the mean between the average historical depth discrete rate and the depth discrete rate threshold; and calculate a new depth discrete rate threshold based on the mean and the false trigger rate.

[0173] Optionally, the update module is also used to obtain a second historical depth image each time the safety prompt is falsely triggered within a preset time period, and a third historical depth image each time the safety prompt is correctly triggered; calculate the second historical depth discrete rate corresponding to the focus area in each second historical depth image, and each second historical depth image corresponds to one second historical depth discrete rate; and calculate the third historical depth discrete rate corresponding to the focus area in each third historical depth image, and each third historical depth image corresponds to one third historical depth discrete rate; if the minimum third historical depth discrete rate is greater than the maximum second historical depth discrete rate, then based on the average between the minimum third historical depth discrete rate and the maximum second historical depth discrete rate, the depth discrete rate threshold is updated; if the minimum third historical depth discrete rate is less than the maximum second historical depth discrete rate, then based on the minimum third historical depth discrete rate, the depth discrete rate threshold is updated.

[0174] Optionally, a shielding area is provided in the area of ​​interest, and the device further comprises:

[0175] A rejection module, used for rejecting the point cloud data in the shielding area if the target is only in the focus area;

[0176] The processing module is used for protecting the point cloud data in the shielding area if the target is in the concerned area and the target is in the shielding area.

[0177] like Figure 3 As shown, an embodiment of the present invention further provides an electronic device, including a processor, and the processor can execute any of the above-mentioned security monitoring methods based on the area of ​​interest.

[0178] Specifically, it includes a processor 301 and a memory 302, and a computer program for executing a safety monitoring method based on a region of interest, which is stored in the memory 302 and can be run on the processor 301, wherein:

[0179] The processor 301 runs the computer program of the safety monitoring method based on the area of ​​interest stored in the memory 302, and performs the following steps:

[0180] Acquire point cloud data of a focus area in the depth image, the point cloud data including depth values ​​of each point cloud in the focus area, the focus area being a designated area in the depth image, and one point cloud corresponding to one pixel in the depth image;

[0181] Calculating a depth discrete rate of the region of interest based on the depth values ​​of each of the point clouds, wherein the depth discrete value is used to represent a fluctuation amplitude of a depth change of each of the point clouds in the region of interest;

[0182] It is determined whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, a safety control signal is output.

[0183] Optionally, the processor 301 calculates the depth discrete rate of the area of ​​interest based on the depth value of each point cloud, including:

[0184] Based on the depth values ​​of each of the point clouds, calculating the average depth value of all point clouds in the area of ​​interest;

[0185] Based on the average depth value, calculating the depth standard deviation or depth variance corresponding to all point clouds in the region of interest;

[0186] A depth discreteness of the region of interest is determined based on the depth standard deviation or the depth variance.

[0187] Optionally, after outputting the safety control signal, the method executed by the processor 301 further includes:

[0188] Calculating an absolute value of a difference between the depth discrete rate and the depth discrete rate threshold;

[0189] If the absolute value of the difference is greater than or equal to a preset absolute value threshold of the difference, the depth discrete rate threshold is updated.

[0190] Optionally, the updating of the depth discrete rate threshold performed by the processor 301 includes:

[0191] Acquire a first historical depth image each time a safety control signal is output within a preset time period;

[0192] Calculating a first historical depth discrete rate corresponding to the area of ​​interest in each of the first historical depth images, each of the first historical depth images corresponding to one first historical depth discrete rate;

[0193] Based on the first historical depth image, obtaining a false trigger rate of the safety prompt;

[0194] The depth discrete rate threshold is updated based on the first historical depth discrete rate and the false trigger rate.

[0195] Optionally, the updating of the depth discrete rate threshold based on the first historical depth discrete rate and the false trigger rate performed by the processor 301 includes:

[0196] Based on all the first historical depth dispersion rates, calculating an average historical depth dispersion rate;

[0197] Calculating a mean value between the average historical depth dispersion rate and the depth dispersion rate threshold;

[0198] Based on the mean value and the false trigger rate, a new depth discrete rate threshold is calculated.

[0199] Optionally, the updating of the depth discrete rate threshold performed by the processor 301 includes:

[0200] Acquire a second historical depth image each time the safety prompt is falsely triggered and a third historical depth image each time the safety prompt is correctly triggered within a preset time period;

[0201] Calculate a second historical depth discrete rate corresponding to the focus area in each of the second historical depth images, each of the second historical depth images corresponding to one second historical depth discrete rate;

[0202] and calculating a third historical depth discrete rate corresponding to the focus area in each of the third historical depth images, each of the third historical depth images corresponding to one third historical depth discrete rate;

[0203] If the smallest third historical depth discrete rate is greater than the largest second historical depth discrete rate, updating the depth discrete rate threshold based on the average of the smallest third historical depth discrete rate and the largest second historical depth discrete rate;

[0204] If the smallest third historical depth discrete rate is smaller than the largest second historical depth discrete rate, the depth discrete rate threshold is updated based on the smallest third historical depth discrete rate.

[0205] Optionally, a shielding area is set in the area of ​​interest, and the method executed by the processor 301 further includes:

[0206] If the target is only in the focus area, the point cloud data in the shielding area is eliminated;

[0207] If the target is in the concerned area and the target is in the shielding area, the point cloud data in the shielding area is protected.

[0208] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the security monitoring method based on the area of ​​interest provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0210] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A safety monitoring method based on a region of interest, characterized in that: The method comprises the following steps: Acquire point cloud data of a focus area in the depth image, the point cloud data including depth values ​​of each point cloud in the focus area, the focus area being a designated area in the depth image, and one point cloud corresponding to one pixel in the depth image; Calculating a depth discrete rate of the region of interest based on the depth values ​​of each of the point clouds, wherein the depth discrete value is used to represent a fluctuation amplitude of a depth change of each of the point clouds in the region of interest; It is determined whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, a safety control signal is output.

2. The security monitoring method based on the area of ​​interest according to claim 1, characterized in that: The calculating the depth discrete rate of the area of ​​interest based on the depth values ​​of each of the point clouds comprises: Based on the depth values ​​of each of the point clouds, calculating the average depth value of all point clouds in the area of ​​interest; Based on the average depth value, calculating the depth standard deviation or depth variance corresponding to all point clouds in the region of interest; A depth discreteness of the region of interest is determined based on the depth standard deviation or the depth variance.

3. The security monitoring method based on the area of ​​interest according to claim 1, characterized in that: After the outputting of the safety control signal, the method further comprises: Calculating an absolute value of a difference between the depth discrete rate and the depth discrete rate threshold; If the absolute value of the difference is greater than or equal to a preset absolute value threshold of the difference, the depth discrete rate threshold is updated.

4. The security monitoring method based on the area of ​​interest according to claim 1, characterized in that: The updating of the depth discrete rate threshold comprises: Acquire a first historical depth image each time a safety control signal is output within a preset time period; Calculating a first historical depth discrete rate corresponding to the area of ​​interest in each of the first historical depth images, each of the first historical depth images corresponding to one first historical depth discrete rate; Based on the first historical depth image, obtaining a false trigger rate of the safety prompt; The depth discrete rate threshold is updated based on the first historical depth discrete rate and the false trigger rate.

5. The security monitoring method based on the area of ​​interest according to claim 4, characterized in that: The updating of the depth discrete rate threshold based on the first historical depth discrete rate and the false trigger rate includes: Based on all the first historical depth dispersion rates, calculating an average historical depth dispersion rate; Calculating a mean value between the average historical depth dispersion rate and the depth dispersion rate threshold; Based on the mean value and the false trigger rate, a new depth discrete rate threshold is calculated.

6. The security monitoring method based on the area of ​​interest according to claim 1, characterized in that: The updating of the depth discrete rate threshold comprises: Acquire a second historical depth image each time the safety prompt is falsely triggered and a third historical depth image each time the safety prompt is correctly triggered within a preset time period; Calculate a second historical depth discrete rate corresponding to the focus area in each of the second historical depth images, each of the second historical depth images corresponding to one second historical depth discrete rate; and calculating a third historical depth discrete rate corresponding to the focus area in each of the third historical depth images, each of the third historical depth images corresponding to one third historical depth discrete rate; If the smallest third historical depth discrete rate is greater than the largest second historical depth discrete rate, updating the depth discrete rate threshold based on the average of the smallest third historical depth discrete rate and the largest second historical depth discrete rate; If the smallest third historical depth discrete rate is smaller than the largest second historical depth discrete rate, the depth discrete rate threshold is updated based on the smallest third historical depth discrete rate.

7. The method for safety monitoring based on the area of ​​interest according to claim 5, characterized in that: A shielding area is provided in the area of ​​interest, and the method further comprises: If the target is only in the focus area, the point cloud data in the shielding area is eliminated; If the target is in the concerned area and the target is in the shielding area, the point cloud data in the shielding area is protected.

8. A safety monitoring device based on a region of interest, characterized in that: The safety monitoring device based on the area of ​​interest includes: An acquisition module, configured to acquire point cloud data of a focus area in a depth image, wherein the point cloud data includes a depth value of each point cloud in the focus area, wherein the focus area is a designated area in the depth image, and one point cloud corresponds to one pixel in the depth image; A calculation module, used to calculate the depth discrete rate of the focus area based on the depth value of each point cloud, wherein the depth discrete value is used to represent the fluctuation amplitude of the depth change of each point cloud in the focus area; The judging module is used to judge whether the depth discrete rate is greater than or equal to a preset depth discrete rate threshold, and if so, output a safety control signal.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the security monitoring method based on the area of ​​interest as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the security monitoring method based on the area of ​​interest as described in any one of claims 1 to 7 are implemented.

Images