Regional monitoring detection method and system with multi-view and multi-modal fusion

Abstract

The present invention provides a multi-view multi-modal fusion plant monitoring detection method and system, which obtains point cloud information at the same time and image information at different acquisition positions; extracts features from the acquired point cloud information to obtain point cloud features; extracts features from the image information separately to obtain multiple groups of image features; rotates the point cloud features and each image feature using internal and external parameter matrices, introduces an attention mechanism to enhance the spatial correlation of each feature after rotation, rotates and aligns the voxel features corresponding to the same feature point, and cascades; uses a target detection network model to perform target detection on the cascaded features to obtain detection results. The present invention uses two sensors, cameras and laser radars with different view angles to work together, and performs target detection based on a multi-modal fusion point cloud target detection network, which optimizes the field stability of the field monitoring system and improves the robustness against large-area occlusion and environmental variability.

Description

Technical Field

[0001] The present invention belongs to the field of monitoring technology, and in particular relates to a method and system for regional monitoring detection with multi-view and multi-modal fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] At present, on-site roadside detection is mainly used to monitor the overall traffic conditions, pedestrian safety and vehicle flow in target areas (such as factories, warehouses, intersections, etc.). It is of great significance for regional management, urban management, traffic planning, safety monitoring and accident prevention.

[0004] According to the inventors, the current field-side roadside detection mainly relies on the use of fixed cameras, radars or other sensors to monitor specific areas. Therefore, in actual application, it may be limited by the installation position and angle, and there are blind spots in the field of vision. For example, when the vehicle is large, it is easy for the monitoring camera to fail to effectively capture the top or rear of the vehicle, which increases the blind spot of the monitoring system and greatly affects the coverage stability of the monitoring system over the monitored area. Alternatively, under severe weather conditions, camera monitoring is easily affected by rain and fog, resulting in blurred and blocked images. Insufficient light at night causes contrast reduction and light and shadow problems, which comprehensively affect the clarity and accuracy of the monitoring image.

[0005] If fixed cameras, radars or other sensors are added, or the monitoring range is increased to provide a panoramic view, or high-definition equipment is invested, the monitoring cost will inevitably increase, the monitoring data will increase, and a large amount of data will need to be processed and analyzed, resulting in a burden and redundancy in data processing. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a multi-view and multi-modal fusion factory monitoring and detection method and system. The present invention adopts two sensors, cameras with different view angles and lidar, to work together, and performs target detection based on a multi-modal fusion point cloud target detection network, which optimizes the field-side monitoring system's field of view stability and improves its robustness against large-area occlusion and changing environment problems.

[0007] According to some embodiments, the present invention adopts the following technical solutions:

[0008] A multi-view multi-modal fusion regional monitoring detection method comprises the following steps:

[0009] Obtain point cloud information at the same time and image information at different acquisition locations;

[0010] Perform feature extraction on the acquired point cloud information to obtain point cloud features;

[0011] Extract features from the image information respectively to obtain multiple groups of image features;

[0012] The point cloud features and each image feature are rotated by internal and external parameter matrices, and the attention mechanism is introduced to enhance the spatial correlation of each feature after rotation. The voxel features corresponding to the same feature point are rotated and aligned, and then cascaded.

[0013] The target detection network model is used to perform target detection on the cascaded features to obtain the detection results.

[0014] As an optional implementation, the picture information is collected by an image acquisition device, and the image acquisition device includes multiple devices installed at different locations of the target area;

[0015] The point cloud information is collected by a laser radar.

[0016] As a further step, the image acquisition device and the laser radar are jointly calibrated to perform spatial and temporal alignment, and obtain the relative position relationship and absolute coordinate information of each device in the world coordinate system, that is, the intrinsic parameter matrix and extrinsic parameter matrix group of the device.

[0017] As an optional implementation, multiple sets of image features are used to generate point cloud features through 2D perspective semantic information and camera intrinsic parameters.

[0018] As a further step, the H*W beam is determined according to the internal parameters of the image acquisition device, and each feature pixel is discretized into D discrete points along the camera beam. Each discrete point is associated with its corresponding depth probability to obtain a sparse point cloud of size H*W*D, where H*W is the number of pixels in the image.

[0019] As an optional implementation, the specific process of rotating the point cloud features and each image feature by the internal and external parameter matrix includes rotating and aligning multiple groups of point cloud features obtained from multiple groups of image features by initially calibrating external parameters.

[0020] As an alternative implementation, multiple feature cascading is performed from a BEV perspective.

[0021] A multi-view multi-modal fusion regional monitoring and detection system, comprising:

[0022] An information acquisition module is configured to acquire point cloud information at the same time and image information at different acquisition locations;

[0023] A point cloud feature extraction module is configured to perform feature extraction on the acquired point cloud information to obtain point cloud features;

[0024] The image feature module is configured to extract features from the image information to obtain multiple groups of image features;

[0025] The feature conversion module is configured to perform internal and external parameter matrix rotation on the point cloud features and each image feature, introduce an attention mechanism to enhance the spatial correlation of each feature after rotation, and rotate and align the voxel features corresponding to the same feature point, and perform cascading;

[0026] The target detection module is configured to use the target detection network model to perform target detection on the cascaded features to obtain a detection result.

[0027] As a further feature, the information acquisition module includes a laser radar and an image acquisition device, the picture information is acquired by the image acquisition device, and the image acquisition device includes a plurality of devices installed at different locations in the target area;

[0028] The point cloud information is collected by a laser radar.

[0029] As a further feature, the image acquisition device is a camera or a webcam, and is installed at least at one point in the monitoring area and on its opposite side, and is pre-calibrated.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention provides a target detection method with multi-view and multi-modal fusion. The bird's-eye view formed is easier to provide a comprehensive perspective of the entire area, which is convenient for observing and extracting more important and complete ground plane position information, and provides a basis for subsequent plant digital twins or anomaly detection.

[0032] The present invention supplements the sparse features of point clouds through the high-density semantic information of multi-view images, while reducing the impact of large obstacles on the field of view of single-sided radar and camera, making the monitoring area more complete and stable.

[0033] The present invention reduces the dependence of the system on monitoring RGB acquisition information and improves the robustness of the target detection system in harsh environments.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 This is a schematic diagram of the installation of multiple field cameras and laser radars in a factory area in an embodiment;

[0037] Figure 2 is a schematic diagram of a factory field target detection model in an embodiment;

[0038] Figure 3 is a schematic diagram of a method flow in an embodiment;

[0039] Figure 4 is a schematic diagram of generating point cloud features through 2D perspective semantic information and camera intrinsic parameters in an embodiment;

[0040] Figure 5 The diagram is a schematic diagram of aligning corresponding feature points by rotating the camera extrinsic parameters in an embodiment. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0044] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0045] Embodiment 1

[0046] This embodiment is described by taking the factory area as the monitoring target area. However, in other embodiments, it is not limited to the above application scenario.

[0047] A factory field monitoring detection method based on multi-view multi-modal fusion. The implementation of this method requires the installation of more than two (or n) cameras (C1, C2...C n ) and LiDAR (L1), such as Figure 1 shown.

[0048] This method, such as Figure 2 and Figure 3 As shown, the following steps are included:

[0049] After the assembly is completed, the laser radar and camera joint calibration method is used to achieve spatial and temporal alignment, and the relative position relationship and absolute coordinate information of each sensor in the world coordinate system are obtained, that is, the intrinsic parameter matrix and extrinsic parameter matrix group of multiple sensors. Camera Ci The internal reference group consists of c i Indicates that the laser radar L j The internal parameter matrix is ​​composed of l j Represented by. The relevant external parameter matrix between multiple sensors is represented by T ij express.

[0050] The image information and point cloud information collected at the same time are obtained as model input. The image features are extracted through Resnet-50, and the point cloud features are extracted through PointNet. The lidar point cloud is obtained by a simple coordinate system rotation to achieve a bird's eye view (due to the tilt of the point cloud caused by the angle when the sensor is installed).

[0051] Multiple sets of image features are used to generate point cloud features through 2D view semantic information and camera intrinsic parameters, such as Figure 4 As shown in FIG. 1 , the specific method is to determine the H*W bundle of camera rays (H*W is also the number of image pixels) through the camera intrinsic parameters. Each feature pixel is discretized into D discrete points along the camera rays, and each discrete point is associated with its corresponding depth probability. It is easy to obtain that the probabilities of the D discrete points on each ray are summed to 1, so that a tensor of size H*W*D is obtained, which can be understood as obtaining the corresponding H*W*D sparse point cloud through the camera 2D perspective semantic information.

[0052] N (H*W*D) tensors obtained by N cameras and 1 lidar point cloud have been obtained. Since the acquisition is obtained from multiple different perspectives, the N groups of point cloud features need to be rotated and aligned through the initial calibration external parameters, such as Figure 5 As shown (the green square indicates a feature, such as Figure 4 The feature is in different positions in different tensors at different views).

[0053] Since this transformation has the possibility of one-to-many, an attention mechanism is introduced to enhance its spatial correlation, and the voxel features corresponding to the same feature point are rotated and aligned, and then cascaded.

[0054] Feature concatenation is a method of combining different feature sets or data sources in an end-to-end manner. In tasks involving multiple types of data, features of different modalities can be combined together through concatenation. Suppose there are two feature vectors A and B, where the dimension of A is m and the dimension of B is n. The feature concatenation operation will create a new feature vector C with a dimension of m+n, which contains all the elements of A and B.

[0055] Finally, multiple features are cascaded from the BEV perspective and input into the point-level object detection network, such as CenterPoint.

[0056] In the above-mentioned embodiment, the RGB image under the field-side perspective is converted into point cloud features and fused with the original ones, and finally input into the target detection network, rather than directly input into the target detection network or architecture, to realize the detection of vehicles (especially large vehicles) and pedestrians, and obtain the final bird's-eye view point cloud visualization map. Compared with the direct use of the point-level target detection network detection method, it can achieve enhanced target detection performance in occluded areas (blind field perception) and improve the detection recall rate.

[0057] Embodiment 2

[0058] A multi-view multi-modal fusion regional monitoring and detection system, comprising:

[0059] An information acquisition module is configured to acquire point cloud information at the same time and image information at different acquisition locations;

[0060] A point cloud feature extraction module is configured to perform feature extraction on the acquired point cloud information to obtain point cloud features;

[0061] The image feature module is configured to extract features from the image information to obtain multiple groups of image features;

[0062] The feature conversion module is configured to perform internal and external parameter matrix rotation on the point cloud features and each image feature, introduce an attention mechanism to enhance the spatial correlation of each feature after rotation, and rotate and align the voxel features corresponding to the same feature point, and perform cascading;

[0063] The target detection module is configured to use the target detection network model to perform target detection on the cascaded features to obtain a detection result.

[0064] The information acquisition module includes a laser radar and an image acquisition device, and the image information is acquired by the image acquisition device. The image acquisition device includes multiple devices and is installed at different positions in the target area;

[0065] The point cloud information is collected by a laser radar.

[0066] The image acquisition device is a camera or a webcam, and is installed at least at one point in the monitoring area and on the opposite side thereof, and is pre-calibrated.

[0067] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A multi-view multi-modal fusion regional monitoring detection method, characterized in that: The following steps are involved: Obtain point cloud information at the same time and image information at different acquisition locations; Feature extraction is performed on the acquired point cloud information to obtain point cloud features. The laser radar point cloud is obtained by a simple coordinate system rotation to achieve a bird's-eye view. Feature extraction is performed on the image information to obtain multiple sets of image features. The multiple sets of image features are used to generate bird's-eye view (BEV) features through 2D perspective semantic information and camera intrinsic parameters. Specifically, the H*W beam is determined according to the intrinsic parameters of the image acquisition device. Each feature pixel is discretized into D discrete points along the camera beam. Each discrete point is associated with its corresponding depth probability to obtain a sparse point cloud of size H*W*D, where H*W is the number of image pixels. The point cloud features and each image feature are rotated by internal and external parameter matrices, and the attention mechanism is introduced to enhance the spatial correlation of each feature after rotation. The voxel features corresponding to the same feature point are rotated and aligned, and then cascaded. The target detection network model is used to perform target detection on the cascaded features to obtain the detection results.

2. The method for regional monitoring and detection with multi-view and multi-modal fusion as claimed in claim 1, characterized in that: The picture information is collected by an image collection device, and the image collection device includes a plurality of devices installed at different positions of the target area; The point cloud information is collected by a laser radar.

3. The method for regional monitoring and detection with multi-view and multi-modal fusion as claimed in claim 2, characterized in that: The image acquisition device and the laser radar are jointly calibrated to perform spatial and temporal alignment, and obtain the relative position relationship and absolute coordinate information of each device in the world coordinate system, that is, the intrinsic parameter matrix and extrinsic parameter matrix group of the device.

4. The method for regional monitoring and detection with multi-view and multi-modal fusion as claimed in claim 1, characterized in that: The specific process of rotating the point cloud features and each image feature by the internal and external parameter matrix includes rotating and aligning multiple groups of point cloud features obtained by multiple groups of image features through the initial calibration of external parameters.

5. The method for regional monitoring and detection with multi-view and multi-modal fusion as claimed in claim 1, characterized in that: Perform multi-feature cascading from a bird's-eye view.

6. A multi-view multi-modal fusion regional monitoring and detection system, characterized in that: include: An information acquisition module is configured to acquire point cloud information at the same time and image information at different acquisition locations; The point cloud feature extraction module is configured to extract features from the acquired point cloud information to obtain point cloud features. The laser radar point cloud is obtained by a simple coordinate system rotation to realize a bird's-eye view. The image feature module is configured to extract features from the image information respectively to obtain multiple sets of image features. The multiple sets of image features are used to generate bird's-eye view BEV features through 2D view semantic information and camera internal parameters. Specifically, the H*W beam is determined according to the internal parameters of the image acquisition device. Each feature pixel is discretized into D discrete points along the camera beam. Each discrete point is associated with its corresponding depth probability to obtain a sparse point cloud of size H*W*D, where H*W is the number of image pixels. The feature conversion module is configured to perform internal and external parameter matrix rotation on the point cloud features and each image feature, introduce an attention mechanism to enhance the spatial correlation of each feature after rotation, and rotate and align the voxel features corresponding to the same feature point, and perform cascading; The target detection module is configured to use the target detection network model to perform target detection on the cascaded features to obtain a detection result.

7. The multi-view multi-modal fusion regional monitoring and detection system as claimed in claim 6, characterized in that: The information acquisition module includes a laser radar and an image acquisition device, and the image information is acquired by the image acquisition device. The image acquisition device includes multiple devices and is installed at different positions in the target area; The point cloud information is collected by a laser radar.

8. The multi-view multi-modal fusion regional monitoring and detection system as claimed in claim 6, characterized in that: The image acquisition device is a camera or a camera head, and is installed at least at one point in the monitoring area and on the opposite side thereof, and is pre-calibrated.

Images