Cross-camera disease detection and positioning tracing method, system, equipment and medium based on multimodal fusion

Abstract

The present invention discloses a method, system, device and medium for cross-camera disease detection and positioning tracing based on multimodal fusion, including: realizing multimodal fusion target detection based on the fusion of two data sources, LiDAR point cloud data and visual image data; performing monocular camera tracking based on a tracking algorithm, including using the Reid algorithm to achieve target deduplication for a single camera; spatial positioning of the detection equipment based on GPS, combining the three-dimensional coordinate calculation of the target by the laser radar to obtain the target real space coordinate information; for the newly detected target and the target with the same spatial position as the existing target, the Reid algorithm is used to calculate the feature similarity, and it is determined whether to retain the newly detected target result, so as to achieve cross-camera target deduplication and tracing. The present invention fully integrates different data sources, and in addition to using more visual images, it also includes LiDAR point cloud data and GPS latitude and longitude information; the method of the present invention can be suitable for different scene requirements and different complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and multimodal fusion, and in particular to a cross-camera disease detection and positioning tracing method, system, equipment and medium based on multimodal fusion. Background Art

[0002] Road engineering is a series of processing processes with roads as the core. In addition to the corresponding road professional technology, the corresponding efficiency and other aspects need to be considered. With the development of society, people's demand for motor vehicles is constantly increasing, and the demand for large-load transportation is also increasing day by day. Therefore, roads need to bear the repeated pressure of these vehicles. At the same time, because roads are exposed to the natural environment, they also need to bear the complex natural environment. These problems may lead to the occurrence or further deterioration of diseases. The occurrence of pavement disease events will lead to the reduction of pavement life and the decline of pavement performance, which seriously affects travel safety and increases the pressure of pavement safety management and operating costs. The most direct impact of the pavement problem is the existence of obvious safety hazards and the decline of pavement use ability. The runway pavement is broken, including the whole board and part, and there is a misalignment or local looseness. It should be compensated or handled in time after discovery; the pavement management agency should also need to conduct a comprehensive evaluation of the pavement in time, including functional evaluation and performance evaluation. Therefore, advanced pavement detection, management and maintenance technologies are urgently needed to improve the pavement performance and extend its service life.

[0003] In modern road engineering management, intelligent inspection equipment has gradually replaced traditional manual inspections, effectively improving the efficiency of road maintenance. The current inspection equipment is equipped with a variety of inspection equipment, multi-source data acquisition equipment including vehicle-mounted cameras, and edge computing equipment. At the same time, a variety of pavement detection algorithms are also deployed, which can achieve rapid detection and upload of road surface diseases, etc., which is helpful for the maintenance and tracing of road diseases. It can effectively increase the service life of roads and increase driving safety and comfort.

[0004] The current inspection equipment is equipped with a variety of inspection equipment, including positioning equipment GNSS and GPS, multi-source data collection equipment including vehicle-mounted cameras, and edge computing equipment. At the same time, a variety of pavement detection algorithms are also deployed to achieve rapid detection and upload of road diseases, which is helpful for the maintenance and tracing of road diseases. By using intelligent inspection equipment, the inspection efficiency of roads can be effectively improved, making inspections more frequent. Through the collected data information, the intelligent inspection equipment can intelligently identify various diseases existing on the current road, which greatly improves the inspection and maintenance efficiency compared to traditional inspection methods.

[0005] Inspection equipment deploys a variety of data collection devices, and can carry out inspections based on multi-dimensional data such as visual information and geographic information. However, due to the complex and changeable geographical environment and diverse natural environment during the inspection process, as well as the interference of weather such as rainy and foggy days, higher requirements are placed on the detection effect of diseases. In addition, unlike conventional targets, the degree of similarity of disease features is high, and due to the instability of detection, the ability to track diseases also needs to be further improved. As intelligent inspections begin to replace manual inspections, road inspections have gradually become a more frequent task. It is difficult to adapt to more complex inspection scenes and changes in the natural environment by relying solely on on-board cameras. The camera's shooting quality and the interference of targets will greatly affect the detection effect. In addition, although multi-source data acquisition devices are installed in the inspection equipment, including visual information, location information, etc., the processing of different data source information is relatively fragmented during the entire inspection process, making it difficult to achieve joint identification of multi-source data. In terms of target deduplication, cross-camera target tracking is currently mainly used in vehicle and pedestrian detection. In comparison, the feature similarity of defects is high and the detection stability is low. Therefore, cross-camera tracking of defects has always been difficult to carry out. Multiple identifications of the same target will greatly increase data storage costs and ineffective maintenance costs. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-camera disease detection and positioning tracing method, system, equipment and medium based on multimodal fusion, combining two types of data sources, namely, vehicle-mounted cameras and vehicle-mounted lidars, and combining the corresponding GPS positioning system and lidar ranging system to carry out research on cross-camera disease detection and tracking based on multi-source data fusion to suit different scene requirements and different complex environments.

[0007] To achieve the above purpose, the technical solution provided by the present invention is:

[0008] The first aspect of the present application provides a cross-camera disease detection and positioning tracing method based on multimodal fusion, comprising:

[0009] The multi-modal fusion target detection is realized based on the fusion of two data sources: LiDAR point cloud data and visual image data;

[0010] Monocular camera tracking based on tracking algorithms, including the use of Reid algorithm to achieve target deduplication for a single camera;

[0011] Based on the spatial positioning of the detection equipment by GPS, combined with the three-dimensional coordinate calculation of the target by LiDAR, the real spatial coordinate information of the target is obtained; the Reid algorithm is used to calculate the feature similarity with the existing targets in the same spatial position to decide whether to retain the newly detected target results, thereby achieving deduplication and tracing of targets across cameras.

[0012] To optimize the above technical solutions, the specific measures taken also include:

[0013] The target detection based on the fusion of two data sources, LiDAR point cloud data and visual image data, to achieve multimodal fusion includes:

[0014] Distortion processing of visual images: Zhang calibration method is used to calibrate the distortion and obtain the calibrated visual image;

[0015] Generate a two-dimensional projection of LiDAR point cloud data: Since the dimension of the laser point cloud does not match the dimension of the visual image, the LiDAR point cloud data of the laser point cloud needs to be converted. The two-dimensional projection correspondence between the LiDAR point cloud data and the data of the corresponding camera coordinate system is:

[0016]

[0017] in, represents the transformation matrix from the LiDAR coordinate system to the image coordinate system, R represents the rotation and translation matrix of the camera, t represents the translation matrix of the camera, v represents the LiDAR point cloud data, y represents the data of the camera coordinate system, and P cam Represents the distortion calibration matrix of the camera; thereby obtaining the point cloud camera two-dimensional projection image of the laser point cloud projected in the camera coordinate system;

[0018] Construct a multimodal convolutional neural network: Based on the YOLO algorithm, the single-channel BackBone structure is optimized into a dual-input feature extraction network. The visual image collected from the on-board camera is distorted, the calibration image, and the two-dimensional projection image of the point cloud camera after data conversion are respectively input into their respective BackBone. In each BackBone, the respective feature layer is derived, and FPN is introduced later to achieve multimodal feature-level fusion. In the feature fusion process, the attention mechanism is introduced to optimize the feature fusion method and strengthen the weight of important features. The final detection output is achieved by adding a detection head Detect to detect targets of different scales. The output results of the multimodal convolutional neural network include: the detection center of the corresponding target (centre x , cent y ), the corresponding detection box size (w, h) and the confidence s of the detected target.

[0019] The monocular camera tracking based on the tracking algorithm includes the Reid algorithm to achieve target deduplication of the single camera, including: using the Deepocsort algorithm to track the target, which includes image calibration, external feature matching, detection frame prediction and target association of the detection output results to achieve tracking of the detected target; the Deepocsort tracking algorithm is based on OC-SORT and adds a target re-identification module containing deep visual information. The Deepocsort tracking algorithm combines visual information with the prediction and estimation information of OC-SORT, and uses the appearance characteristics of the target to correct the prediction of the result.

[0020] In the target deduplication of a single camera, the output results of the detection are formed into a target detection set (cent x -w / 2, cent y +h / 2, cent x +w / 2, cent y -h / 2,s), where cent x -w / 2, centy+h / 2 represent the coordinates of the upper left corner of the detection box, centx+w / 2, centy-h / 2 represent the coordinates of the lower right corner of the detection box; CMC technology is used to provide camera compensation between each frame; the Reid algorithm is used to recognize the appearance features of the detection target, and the detection targets with relatively high confidence s are selected among the detection targets. The appearance features of these detection targets are dynamically introduced into the deep visual track of the target re-identification module, and the weight ratio between the OC-SORT module and the target re-identification module is dynamically adjusted.

[0021] The deduplication of targets for a single camera also includes: adaptive weight allocation, adaptively changing the feature appearance weights by introducing the appearance feature matrix, and determining the appearance matrix composition based on the number of detection frames and the current number of tracks; detection frame prediction: OC-SORT combines OCM, OCR and OOS modules to optimize the error in the camera motion model, uses the Kalman filter to predict the results, and associates the prediction results through the Hungarian algorithm; target matching: calculates the feature similarity and target association results of the detected target and the predicted target, and uses adaptive weighting to associate the detected target with the predicted target to achieve the best match.

[0022] The spatial positioning of the detection equipment based on GPS is read with direction information, and the three-dimensional coordinates of the target are calculated by laser radar to obtain the real spatial coordinate information of the target; the target at the same spatial position is calculated with the Reid algorithm to determine whether to retain the newly detected target result, so as to achieve cross-camera target deduplication and tracing, specifically:

[0023] The obtained target real space coordinate information is used as the target coordinate label; the current date is obtained to form a date label; the target type is added to each target to form a type label; the target coordinate label, type label, and date label are formed into a complete target detection object label, which is compared with the target detection in the database: for targets with the same category label and similar coordinate labels, the Reid algorithm is used to calculate the feature similarity. If the similarity is within the error range, the target is considered to be the target stored in the current database; then, based on the difference in the acquisition date and the analysis requirements for the development of the disease with the date, it is determined whether the current target needs to be eliminated, and finally it is decided whether to retain the newly detected target result.

[0024] The method for spatial positioning of the detection equipment and reading the direction information is as follows:

[0025] First, target recognition is performed on the data collected by the current i-th detection vehicle, and the targets of the single camera are deduplicated to obtain the target detection set collected by the current vehicle; the GPS latitude and longitude information of the current detection vehicle where the detection equipment is located is obtained; the image coordinates of the four corner points of the detection frame of the detection target are calculated; the current detection frame corner points are mapped to the point cloud two-dimensional projection points of the nearest Euclidean distance in the two-dimensional projection image of the point cloud camera, and the real three-dimensional coordinates of the detection frame corner points are read through the real space point cloud corresponding to the point cloud two-dimensional projection points to obtain the three-dimensional coordinates of the detection target relative to the detection vehicle; the longitude and latitude coordinates of the detection target are obtained by combining the deflection angle and longitude and latitude information of the detection vehicle.

[0026] The second aspect of the present application provides a cross-camera disease detection and positioning tracing system based on multimodal fusion, including:

[0027] Multimodal data fusion target detection module, which is used to realize multimodal fusion target detection based on the fusion of two data sources: LiDAR point cloud data and visual image data;

[0028] The monocular camera tracking module is used to track the monocular camera based on the tracking algorithm, including the use of the Reid algorithm to achieve target deduplication of the single camera;

[0029] The cross-camera target deduplication and tracing module is used to obtain the target's real spatial coordinate information based on the spatial positioning of the detection equipment based on GPS and the three-dimensional coordinate calculation of the target by the laser radar. The target with the same spatial position as the existing one is calculated by the Reid algorithm for feature similarity, and it is decided whether to retain the newly detected target result, so as to achieve cross-camera target deduplication and tracing.

[0030] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the cross-camera disease detection and positioning tracing method based on multimodal fusion as described above is implemented.

[0031] A fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the cross-camera disease detection and positioning tracing method based on multimodal fusion as described above.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] At present, the recognition of diseases is still mainly based on visual images. Since the characteristics of diseases are similar and uncertain, the scale varies greatly, and there are many types, the recognition effect of diseases still needs to be further improved. When the computing power meets the conditions, the most commonly used LiDAR and vehicle-mounted cameras are combined to realize multi-modal data fusion recognition of diseases. On the basis of improving the detection accuracy, the algorithm's anti-interference ability is also increased, which effectively expands the application scenarios of the algorithm and further improves the robustness of the algorithm. At present, the cross-camera multi-target recognition and deduplication algorithm is the mainstream development direction of target detection. Due to the influence of factors such as the background, angle and scale of the target in different cameras, the detection effect of the algorithm still has certain limitations.

[0034] The current monocular target deduplication mainly relies on the spatial prediction of the target and the comparison of its appearance features to achieve target tracking. Considering that the position coordinates of road surface diseases in the geographical environment are fixed, in addition to the algorithm factors, the coordinate information of the target is additionally introduced in the present invention. Through high-precision GPS positioning and high-precision LiDAR ranging, the coordinate information of the target relative to the vehicle is converted into the longitude and latitude coordinate information of the target in the entire space. Combined with the type of target, the date of acquisition, and the similarity analysis of the target with the target in the database, cross-camera target detection deduplication and tracing can be achieved.

[0035] In summary, the present invention fully integrates different data sources. In addition to using more visual images, it also includes LiDAR point cloud data and GPS longitude and latitude information. By establishing a new model, the accuracy of target detection is first improved, and by adding additional physical labels to the detected targets, cross-camera multi-target deduplication, which is currently difficult to achieve with algorithms alone, is achieved. Specifically, the present invention makes full use of various data sources in the vehicle-mounted equipment, and based on target detection and monocular tracking, it realizes target design features and coordinate labels, and realizes cross-camera target deduplication by comparing multiple labels with database results. The present invention is based on the use of multi-source data formed in actual situations, and involves multiple algorithms such as the processing of point cloud data and the fusion model of multi-source data. The method of the present invention can adapt to different scene requirements and different complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 : Schematic diagram of the Muti-CNN model structure of the multimodal convolutional neural network of the present invention.

[0037] Figure 2 : Schematic diagram of the monocular tracking process of the monocular camera tracking module of the present invention.

[0038] Figure 3 : Schematic diagram of the cross-camera target deduplication process of the cross-camera target deduplication and tracing module of the present invention. DETAILED DESCRIPTION

[0039] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0040] Some of the terms involved in the solution of the present invention are explained as follows:

[0041] LiDAR (Light Detection And Ranging): It is a remote sensing (sensing from a distance using a sensor) method that uses near-infrared light, visible light, or ultraviolet light to illuminate an object and capture its reflected light with an optical sensor to measure the distance.

[0042] Reid (re-identification) is a technology that uses computer vision technology to determine whether a specific target exists in an image or video sequence.

[0043] YOLO (You Only Look Once, target detection algorithm): a typical single-stage target detection algorithm.

[0044] CMC (Camera Motion Compensation): estimates the global rotation and offset between adjacent frames captured by the camera to reduce image distortion.

[0045] OCM (Observation-Centric Momentum): The directional consistency of the trajectory is added to the cost matrix to achieve better matching.

[0046] OCR (Observation-Centric Recovery): Recover lost tracking due to occlusion.

[0047] OOS (Observation-centric Online Smoothing): Reduces the accumulation of squared errors in Kalman predictions with respect to time.

[0048] In one embodiment, the present invention proposes a cross-camera disease detection and positioning tracing method based on multimodal fusion, including:

[0049] The multi-modal fusion target detection is realized based on the fusion of two data sources: LiDAR point cloud data and visual image data;

[0050] Monocular camera tracking is performed based on the tracking algorithm, and the Reid algorithm is combined to achieve target deduplication of a single camera;

[0051] Based on the spatial positioning of the detection equipment by GPS, combined with the three-dimensional coordinate calculation of the target by LiDAR, the real spatial coordinate information of the target is obtained; the Reid algorithm is used to calculate the feature similarity with the existing targets in the same spatial position to decide whether to retain the newly detected target results, thereby achieving deduplication and tracing of targets across cameras.

[0052] The present invention can adapt to different scene requirements and different complex environments. The whole process can be further divided into the following three modules. The first part is multimodal data fusion target detection, which is mainly based on the fusion of two data sources, LiDAR and visual images, to realize multimodal fusion target detection; the second part is monocular camera tracking, which can realize single camera target deduplication based on tracking algorithm and Reid algorithm; the third part is cross-camera target deduplication and tracing, which is based on GPS for spatial positioning of detection equipment, combined with laser radar for three-dimensional coordinate calculation of target, can obtain accurate target real space coordinate information, so as to obtain the first label of the target, in addition to the type label and date label of the target, and compare the obtained label with the target detection in the database. For targets with the same category label and close coordinate labels, the Reid algorithm is further used to perform feature similarity analysis, and finally decide whether to retain the newly detected target results.

[0053] Next, we will introduce each module and the specific implementation steps:

[0054] (1) Multimodal fusion target detection module

[0055] Step 1.1: Distortion processing of visual images. Before data collection, the camera needs to be calibrated for distortion. Here, Zhang's calibration method is used, and a chessboard with alternating black and white squares is selected as the calibration object for subsequent calibration. Since two-dimensional objects lack certain position information for three-dimensional calibration objects. First, you need to use a camera to take pictures of the chessboard at different angles. In practical applications, the total number of chessboards taken at different angles needs to be greater than 10. The distortion calibration process has been integrated into the opencv library, and the use of related functions can greatly simplify the overall calculation process. First, determine the number of horizontal and vertical grids contained in the designed chessboard (num w , num h ), the calculation process is divided into three steps. First, use the findChessboardCorners() function to extract the corner points in the chessboard. The extracted corner points do not include the edge of the chessboard. Use the find4QuadCornerSubpix() function to further extract the sub-pixel information of the corner points to improve the accuracy of corner point extraction. After detection, the drawChessboardCorners() function can be used to visualize the extraction of the corner points and obtain the corresponding corner point image coordinate system (pix x ,pix y ), using the obtained corner image coordinate set and the number of real grids, calibrateCamera() is used to obtain the camera's intrinsic parameter matrix R0 and the corresponding distortion matrix P. In undistort(), the final calibrated image can be calculated using the intrinsic parameter matrix and the distortion matrix.

[0056] Step 1.2: Generate a two-dimensional projection of the point cloud. In multimodal fusion, the point cloud data must be processed first. The detected point cloud data contains four parameters (x, y, z, f), of which the first three parameters represent the spatial three-dimensional coordinates of the point cloud, and the last one represents the reflection intensity of the point cloud. Since the data dimension of the laser point cloud does not match the dimension of the image, the point cloud needs to be converted in combination with the relevant parameters of the laser point cloud. The core parameters include the conversion matrix from the laser radar coordinate system to the image coordinate system. The rotation and translation matrix R and translation matrix t of the camera, the distortion calibration matrix P of the camera, combined with the above parameters, the corresponding process of the point cloud data v to the data y of the corresponding camera coordinate system is:

[0057]

[0058] Combining the above formulas, we can get the camera 2D projection image of the point cloud projected in the camera coordinate system, and remove the point cloud data outside the image. The obtained camera 2D projection image of the point cloud can be further divided into height, depth, and reflection intensity images.

[0059] Step 1.3: Construction of Multimodal Convolutional Neural Network Model (Muti-CNN). Based on the YOLO algorithm, the single-channel BackBone structure is designed as a dual-input feature extraction network. The visual image collected from the vehicle-mounted camera, the calibrated image after step 1.1, and the point cloud projection image generated in step 1.2 are respectively input into their respective BackBone, and their respective feature layers are derived in their respective BackBone. FPN is subsequently introduced to achieve multi-modal feature-level fusion. The attention mechanism is introduced in the feature fusion process to optimize the feature fusion method and strengthen the weights of important features. The dual-channel BackBone design can take into account feature layers of different levels and perform a variety of feature fusion methods. The final detection output adds the detection head Detect to realize the detection of targets of different scales. The overall model structure adopts a modular design, and each part can be modularly replaced or modularly added according to needs. The specific structure is as follows: Figure 1 As shown in the figure. During the detection process, a large number of candidate regions are preset, and most of the pre-detection boxes are eliminated through confidence values ​​and IoU detection. The loss function includes category loss and regression result loss. The final output result includes the detection center of the corresponding target (cent x , cent y ), the corresponding detection box size (w, h) and the confidence s of the detected target.

[0060] (2) Monocular camera tracking module

[0061] Step 2.1: Monocular tracking algorithm composition. The Deepocsort tracking algorithm detects the results including image calibration, external feature matching, detection box prediction and target association, and finally realizes the tracking of the detected target. The calculation process of Deepocsort is as follows: Figure 2 As shown in Figure 2, the Deepocsort tracking algorithm combines visual information with the prediction and estimation information of OC-SORT, and modifies the prediction results by adding target appearance features.

[0062] Step 2.2: CMC camera compensation of the image. Based on the target detection results generated in step 1.3, a target detection set (cent x -w / 2, cent y +h / 2, centx +w / 2, cent y -h / 2,s), where cent x -w / 2, centy+h / 2 represent the coordinates of the upper left corner of the detection frame, and centx+w / 2, centy-h / 2 represent the coordinates of the lower right corner of the detection frame. At the same time, the CMC technology is used to provide camera compensation for each frame for the detected results.

[0063] Step 2.3: Calculation of appearance features. The detection model is identified by the Reid algorithm, and the detection targets with relatively high confidence s are selected from the detection targets. The appearance features of these detection targets are dynamically introduced into the deep visual track of the target re-identification module, and the weight ratio of the two is dynamically adjusted.

[0064] Step 2.4: Adaptive weight assignment. The appearance feature matrix is ​​introduced to adaptively change the feature appearance weights, and the appearance matrix composition is determined according to the number of detection boxes and the current number of tracks.

[0065] Step 2.5: Detection box prediction. The OC-SORT algorithm combines the OCM, OCR and OOS modules to optimize the error in the camera motion model, uses the Kalman filter to predict the results, and uses the Hungarian algorithm to associate the prediction results. By introducing the CMC technology in step 2.2 to compensate for the image, the prediction results are more accurate.

[0066] Step 2.6: Target matching: Calculate the feature similarity and association results between the detected target and the predicted target, and use adaptive weighting to associate the detected target with the predicted target to achieve the best match.

[0067] (3) Cross-camera target deduplication and tracing module

[0068] Step 3.1: Obtaining detection target parameters. First, perform target recognition on the data collected by the current train i, and perform single target tracking to remove duplicates, and obtain the target detection set detections collected by the current train. Each object in the detections contains five parameters (cent x -w / 2, cent y +h / 2, cent x +w / 2, cent y -h / 2,s). Further obtain the GPS latitude and longitude information including the current train number (C lon , C lat ), since the positioning interval of GPS is short and within the error range, it can be approximately considered that the current vehicle is traveling in a straight line. Combining the GPS information before and after the two times, the current vehicle driving direction angle B can be further calculated, where the direction angle ranges from 0-360°.

[0069] Step 3.2: Detect the spatial three-dimensional coordinates of the target. Based on the detection, the coordinates of the upper left corner and lower right corner of the detection frame are further calculated to obtain the image coordinates of the four points of the detection frame (cent x– w / 2, cent y +h / 2), (cent x +w / 2, cent y +h / 2), (cent x +w / 2, cent y –h / 2), (cent x +w / 2, cent y +h / 2). The camera 2D projection image of the point cloud obtained in step 1.2, so the point cloud as discrete data does not correspond to each pixel in the entire image one by one, here the current detection box corner point and the point cloud 2D projection point with the nearest Euclidean distance in the image are used as the mapping relationship between the two. The real 3D coordinates (x, y, z) of the pixel point can be read from the real space point cloud corresponding to the point cloud 2D projection point.

[0070] Step 3.3: Detect the longitude and latitude coordinates of the target. In step 3.2, the three-dimensional coordinates of the target relative to the vehicle are obtained. Combined with the vehicle's deflection angle, the longitude and latitude information can be used to obtain the longitude and latitude coordinates of the target:

[0071] (D lon , D lat )=(C lon , C lat )+(C lon , C lat )·(cos(B), sin(B)) T

[0072] Since both GPS positioning and LiDAR distance measurement have high precision, the high-precision latitude and longitude coordinates of the detection target are finally obtained.

[0073] Step 3.4: Multi-label design for detection targets. Based on step 3.3, the corresponding longitude and latitude coordinates of each target in detections are obtained. Further, the target type id and target detection date are added to each target to form a complete target detection object label (D lon , D lat , id, date), and obtain a new set of detection target labels. It is used to distinguish data collected on different dates from different trains.

[0074] Step 3.5: Obtaining database targets and deduplication of detection targets. After obtaining the detection targets of the current date and the current train, traverse all the detection targets stored previously. First, compare the target type and longitude and latitude coordinates in the detection target. If the id information of the newly collected target is different from that of all targets in all databases or the longitude and latitude coordinate information is outside the error range, the current collection is considered to be a new target and stored in the database. If it is within the error range, combined with the Reid algorithm in step 2.3, calculate the appearance features of the current target, and calculate the cosine similarity cosD of the targets with the same id and longitude and latitude error range in the database. If the similarity between the two is still within the error range, it is considered that the target is the target stored in the current database. Finally, determine the date of collection, and determine whether the current target needs to be eliminated based on the difference in collection dates and the analysis requirements for the development of diseases with dates. Implement cross-camera target detection deduplication and traceability. The specific implementation process is as follows: Figure 3 shown.

[0075] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cross-camera disease detection and positioning tracing method based on multimodal fusion as described above is implemented.

[0076] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the cross-camera disease detection and positioning tracing method based on multimodal fusion as described above.

[0077] In addition to diseases, the present invention can also be applied to other target detections, such as providing a feasible path method for subsequent detection, identification, deduplication and tracing of fixed resources such as road assets.

[0078] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0079] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A cross-camera disease detection and positioning tracing method based on multimodal fusion, characterized in that: include: The multi-modal fusion target detection is realized based on the fusion of two data sources: LiDAR point cloud data and visual image data; Monocular camera tracking based on tracking algorithms, including the use of Reid algorithm to achieve target deduplication for a single camera; Based on the spatial positioning of the detection equipment by GPS, combined with the three-dimensional coordinate calculation of the target by LiDAR, the real spatial coordinate information of the target is obtained; the Reid algorithm is used to calculate the feature similarity with the existing targets in the same spatial position to decide whether to retain the newly detected target results, thereby achieving deduplication and tracing of targets across cameras.

2. The cross-camera disease detection and positioning tracing method based on multimodal fusion according to claim 1 is characterized by: The target detection based on the fusion of two data sources, LiDAR point cloud data and visual image data, to achieve multimodal fusion includes: Distortion processing of visual images: Zhang calibration method is used to calibrate the distortion and obtain the calibrated visual image; Generate a two-dimensional projection of LiDAR point cloud data: Since the dimension of the laser point cloud does not match the dimension of the visual image, the LiDAR point cloud data of the laser point cloud needs to be converted. The two-dimensional projection correspondence between the LiDAR point cloud data and the data of the corresponding camera coordinate system is: in, represents the transformation matrix from the LiDAR coordinate system to the image coordinate system, R represents the rotation and translation matrix of the camera, t represents the translation matrix of the camera, v represents the LiDAR point cloud data, y represents the data of the camera coordinate system, and P cam Represents the distortion calibration matrix of the camera; thereby obtaining the point cloud camera two-dimensional projection image of the laser point cloud projected in the camera coordinate system; Construct a multimodal convolutional neural network: Based on the YOLO algorithm, the single-channel BackBone structure is optimized into a dual-input feature extraction network. The visual image collected from the on-board camera is distorted, the calibration image, and the two-dimensional projection image of the point cloud camera after data conversion are respectively input into their respective BackBone. In each BackBone, the respective feature layer is derived, and FPN is introduced later to achieve multimodal feature-level fusion. In the feature fusion process, the attention mechanism is introduced to optimize the feature fusion method and strengthen the weight of important features. The final detection output is achieved by adding a detection head Detect to detect targets of different scales. The output results of the multimodal convolutional neural network include: the detection center of the corresponding target (centre x , cent y ), the corresponding detection box size (w, h) and the confidence s of the detected target.

3. The cross-camera disease detection and positioning tracing method based on multimodal fusion according to claim 2 is characterized by: The monocular camera tracking based on the tracking algorithm includes a Reid algorithm to achieve target deduplication of a single camera, including: Based on the Deepocsort algorithm, the detection output results are subjected to image calibration, external feature matching, detection box prediction and target association to achieve tracking of the detected target; the Deepocsort tracking algorithm is based on OC-SORT and adds a target re-identification module containing deep visual information.

4. The cross-camera disease detection and positioning tracing method based on multimodal fusion according to claim 3 is characterized by: In the target deduplication of a single camera, the output results of the detection are formed into a target detection set (cent x -w / 2, cent y +h / 2, cent x +w / 2, cent y -h / 2,s), where cent x -w / 2, centy+h / 2 are the coordinates of the upper left corner of the detection box, centx+w / 2, centy-h / 2 are the coordinates of the lower right corner of the detection box; CMC technology is used to provide camera compensation between each frame; the Reid algorithm is used to recognize the appearance features of the detection target, and the detection targets with relatively high confidence s are selected among the detection targets. The appearance features of these detection targets are dynamically introduced into the deep visual track of the target re-identification module, and the weight ratio between the OC-SORT module and the target re-identification module is dynamically adjusted.

5. The cross-camera disease detection and positioning tracing method based on multimodal fusion according to claim 4 is characterized by: The deduplication of targets for a single camera also includes: adaptive weight allocation, adaptively changing the feature appearance weights by introducing the appearance feature matrix, and determining the appearance matrix composition based on the number of detection frames and the current number of tracks; detection frame prediction: the OC-SORT algorithm combines the OCM, OCR and OOS modules to optimize the error in the camera motion model, uses the Kalman filter to predict the results, and associates the prediction results through the Hungarian algorithm; target matching: calculates the feature similarity and target association results of the detected target and the predicted target, and uses adaptive weighting to associate the detected target with the predicted target to achieve the best match.

6. The cross-camera disease detection and positioning tracing method based on multimodal fusion according to claim 4 is characterized by: The spatial positioning of the detection equipment based on GPS is read with direction information, and the three-dimensional coordinates of the target are calculated by laser radar to obtain the real spatial coordinate information of the target; The Reid algorithm is used to calculate the feature similarity of existing targets with the same spatial position, and decide whether to retain the newly detected target results, so as to achieve cross-camera target deduplication and tracing. Specifically: The obtained target real space coordinate information is used as the target coordinate label; the current date is obtained to form a date label; the target type is added to each target to form a type label; the target coordinate label, type label, and date label are formed into a complete target detection object label, which is compared with the target detection in the database: for targets with the same category label and similar coordinate labels, the Reid algorithm is used to perform feature similarity. If the similarity is within the error range, the target is considered to be the target stored in the current database; then, based on the difference in the acquisition date and the analysis requirements for the development of the disease with the date, it is determined whether the current target needs to be eliminated, and finally it is decided whether to retain the newly detected target result.

7. The cross-camera disease detection and positioning tracing method based on multimodal fusion according to claim 6 is characterized by: The method for spatial positioning of the detection equipment and reading the direction information is as follows: First, target recognition is performed on the data collected by the current i-th detection vehicle, and the targets of the single camera are deduplicated to obtain the target detection set collected by the current vehicle; the GPS latitude and longitude information of the current detection vehicle where the detection equipment is located is obtained; the image coordinates of the four corner points of the detection frame of the detection target are calculated; the current detection frame corner points are mapped to the point cloud two-dimensional projection points of the nearest Euclidean distance in the two-dimensional projection image of the point cloud camera, and the real three-dimensional coordinates of the detection frame corner points are read through the real space point cloud corresponding to the point cloud two-dimensional projection points to obtain the three-dimensional coordinates of the detection target relative to the detection vehicle; the longitude and latitude coordinates of the detection target are obtained by combining the deflection angle and longitude and latitude information of the detection vehicle.

8. A cross-camera disease detection and positioning tracing system based on multimodal fusion, characterized in that: include: Multimodal data fusion target detection module, which is used to realize multimodal fusion target detection based on the fusion of two data sources: LiDAR point cloud data and visual image data; The monocular camera tracking module is used to track the monocular camera based on the tracking algorithm, including the use of the Reid algorithm to achieve target deduplication of the single camera; The cross-camera target deduplication and tracing module is used to obtain the target's real spatial coordinate information based on the spatial positioning of the detection equipment based on GPS and the three-dimensional coordinate calculation of the target by the laser radar. The target with the same spatial position as the existing one is calculated by the Reid algorithm for feature similarity, and it is decided whether to retain the newly detected target result, so as to achieve cross-camera target deduplication and tracing. The system is used to execute the cross-camera disease detection and positioning tracing method based on multimodal fusion as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for cross-camera disease detection and positioning tracing based on multimodal fusion as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the cross-camera disease detection and positioning tracing method based on multimodal fusion as described in any one of claims 1 to 7.

Images