Target searching and positioning method and system based on multimode data fusion

Abstract

The invention discloses a target searching and positioning method and system based on multimode data fusion, and belongs to the technical field of data fusion. The method comprises the following steps: acquiring detection data of a detection device for a specific landmark object, performing fusion processing on the detection data to form a synthetic image, acquiring positioning data of the detection device, calculating the positioning data of the specific landmark object and system positioning deviation by combining the synthetic image, and outputting the positioning data of the specific landmark object. Correcting the positioning data of the detection equipment according to the positioning data of the specific landmark object; carrying out mobile phone signal acquisition at a plurality of positioning points by utilizing detection equipment to generate a mobile phone positioning data set; and judging whether the mobile phone is in a motion state according to the mobile phone positioning data set and the system positioning deviation, and correcting the mobile phone positioning data based on the corrected positioning data of the detection equipment under the condition that the mobile phone is in a non-motion state to obtain a final positioning result of the mobile phone. The method can be used for building a rapid search and rescue system.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion, and in particular relates to a target search and positioning method and system based on multi-mode data fusion. Background Art

[0002] After a disaster, the earliest and most accurate detection of buried personnel is a core technical requirement for disaster emergency rescue. Due to the destructive nature of disaster sites, rapidly obtaining personnel information and accurately locating them is a significant challenge. Against this backdrop, my country's emergency response industry is developing a variety of search and detection equipment to achieve these goals, leveraging projects such as the "Key Technology and Equipment for Precise Search of Life at Large Scale and Depth Based on Multi-Source Information Fusion" (2023YFC3011500).

[0003] On the other hand, with the continuous advancement of social science, technology, and economic development, smart devices, represented by mobile phones, have gradually become indispensable tools in people's daily lives and production. According to incomplete statistics, the average number of mobile phones per person in my country currently reaches 1.28. my country has built the world's largest 4G network and has essentially achieved nationwide 4G coverage. By the end of 2021, the number of 4G mobile users in China had reached 1.52 billion, with a coverage rate exceeding 97%. The number of 5G base stations exceeded 4.1 million, with 995 million users and a coverage rate of 70%. Smart devices continuously generate signals during operation. Due to the unique relationship between smart devices and people, these signals can serve as clues to people's movements and status. These signal clues can guide the discovery and location of trapped people. Therefore, equipment that searches for mobile communication signals has gradually become a key technical tool in the emergency response industry.

[0004] However, mobile communication signals themselves reflect the terminal's operating status and location. Therefore, this method can be considered an indirect clue to the presence of a person. In practical applications, mobile terminal information collected over a wide range of searches cannot directly indicate whether a person is trapped. In emergency scenarios, applications must identify the person based on the terminal's motion status and its relationship to the damaged area or building space.

[0005] Due to this deepening application requirement, terminal search equipment often requires continuous searches of specific areas. Based on the serialization of terminal positioning data from multiple searches, the terminal's motion characteristics are determined. Furthermore, the search and identification results are spatially superimposed with the trapped area at the scene to form the final analysis. Therefore, to achieve more accurate results during the search process, the search equipment often needs to continuously collect signals and process data in a specific area using a specific operating method, and then fuse the multiple processing results to form the final conclusion.

[0006] However, due to the complexity of disaster-stricken environments, human targets themselves may be in random motion. Furthermore, during the multi-point sampling process, the search equipment faces difficulties in merging multi-point measurements due to errors in the positioning of its own carrier platform (e.g., drone) and numerical processing calculations. This makes it difficult to establish an accurate motion state discrimination mechanism for positioning data in a time series, leading to erroneous conclusions. These erroneous conclusions can lead to incorrect target guidance, delays, and even search and rescue failures. These issues significantly impact the rapid analysis and assessment of disaster targets in disaster areas. Summary of the Invention

[0007] In combination with the characteristics of post-disaster emergency rescue business, on the basis of multi-mode data fusion processing, the present invention proposes a target search and positioning method and system based on multi-mode data fusion, which specifically solves the technical difficulties currently faced by emergency management departments and ensures the construction and application of a rapid search and rescue system.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0009] A target search and positioning method based on multimodal data fusion, the method comprising:

[0010] Acquire detection data of a specific landmark by detection equipment, and fuse the detection data to form a synthetic image; wherein the detection data includes radar data, optical data, and infrared data;

[0011] Acquiring positioning data of the detection equipment, and calculating positioning data of a specific landmark and a system positioning deviation based on the synthetic image, and then correcting the positioning data of the detection equipment based on the positioning data of the specific landmark;

[0012] Use detection equipment to collect mobile phone signals at multiple positioning points to generate a collection of mobile phone positioning data;

[0013] Based on the mobile phone positioning data set and the system positioning deviation, it is determined whether the mobile phone is in motion. When the mobile phone is in a non-motion state, the mobile phone positioning data is corrected based on the positioning data of the corrected detection equipment to obtain the final positioning result of the mobile phone.

[0014] Furthermore, the detection data is fused to form a synthetic image, including:

[0015] Taking the detection data with the highest imaging frequency as a benchmark, extract the synchronous frames in each detection data to generate a synchronous frame data set;

[0016] The synchronous frames in the synchronous frame dataset are fused to obtain a synthetic image.

[0017] Furthermore, taking the detection data with the highest imaging frequency as a benchmark, the synchronous frames in each detection data are extracted to generate a synchronous frame data set, including:

[0018] Setting a time difference tolerance and establishing a window according to the time difference tolerance;

[0019] Obtain the detection data with the highest imaging frequency at time T i The most recent dataframe Among them, the data frame The corresponding time T i ′ and time T i The difference is no greater than the window length constraint;

[0020] Establishment time T i ′’s forward window TW fw ′ and the backward window TW bk ';

[0021] For detection data with a non-highest imaging frequency, in the forward window TW fw ′ and backward window TW bk ′ and time T i 'The most recent data frame R fw ′ and data frame R bk ';

[0022] Comparing data frames in R fw ' and the data frame R bk ′, and the distance time T i ′The closest data frame is taken as data frame T pi ′ synchronization frame.

[0023] Furthermore, the synchronized frames in the synchronized frame dataset are fused to obtain a synthetic image, including:

[0024] Traverse the data frame R in the synchronized frame dataset and obtain the maximum pixel value u of the resolution max and the maximum pixel value v mxx ;

[0025] Based on the maximum pixel value u mxx and the maximum pixel value v mxx The data frame R is stretched to obtain the data frame R′; wherein the pixel values ​​of the four corners (p1, p2, p3, p4) of the data frame R′ are: p1=(u′ i ,v′ j ), p2=(u′ i+1 ,v′ j ), p3=(u′ i+1 ,v′ j+1 ), p4=(u′ i ,v′j+1 );

[0026] For pixel value u′ i and pixel value v′ j Round up and calculate the pixel value u′ i+1 and pixel value v′ j+1 Round down to form a dot matrix data set uvc = {(uc x ,vc y )}|x=uidx1,…,uidx2,y=vidx1,…,vidx2}; where uidx1, uidx2, vidx1 and vidx2 are the pixel values ​​u′ i , pixel value u′ i+1 , pixel value v′ j and pixel value v′ j+1 The rounded result of the value of each newly generated pixel in the dot matrix data set uvc is obtained by bilinear interpolation and encapsulated in the corresponding attribute of the newly generated pixel;

[0027] Based on the dot matrix data set uvc, the synchronization frames of different detection data are aligned to obtain the data frame Rc;

[0028] Based on the data frame Rc, the synchronous frames of different detection data are fused to obtain a synthetic image.

[0029] Furthermore, the positioning data of the detection equipment is obtained and combined with the synthetic image to calculate the positioning data of specific landmarks and the system positioning deviation, including:

[0030] Acquire detection equipment to form a synthetic image i Positioning data expression EPPos i ={ROPGPS i ,α i ,β i ,θ i}, α i ,β i ,θ i is the rotation angle of the first load coordinate system axis, ROPGPS i ={Lat i ,Lng i ,Z i} is the positioning data expression of the detection equipment in the world coordinate system, Lat i Longitude of the detection equipment, Lng i is the longitude of the detection equipment, Z i To detect the height of the equipment, the first load coordinate system is formed by forming a synthetic image image i The polar coordinate system is formed by taking the position of the detection equipment at the time as the origin;

[0031] Get a specific landmark in the synthetic image i Pixel coordinates The corresponding radar data depth value Depth i , and calculate the pixel coordinates The relative coordinate angle A i and relative coordinate angle B i After that, the relative coordinates ROPRPos between the specific landmark and the detection equipment are obtained i ={A i ,B i ,Depth i}; Among them, the relative coordinate angle A i The angle between the vector between the specific landmark and the origin of the first load coordinate system and the XY plane of the world coordinate system, the relative coordinate angle B i The angle between the X-axis and the projection of the vector between the specific landmark and the origin of the first load coordinate system in the XY plane of the world coordinate system;

[0032] To form a synthetic image i The position of the detection equipment at that time is the origin, a Cartesian coordinate system is established, and the relative coordinate ROPRPos i Converted to Cartesian coordinates ROPRPos′ i ;

[0033] According to the rotation angle α i , rotation angle β i and the rotation angle θ i Construct the inverse transformation matrix and express it according to the positioning data Construct a space matrix and transform the coordinate ROPRPos′ based on the inverse transformation matrix and the space transformation matrix i Transform world coordinates to ROPGPS i ;

[0034] All world coordinate systems are based on ROPGPS i A minimum enclosing circle CR is established, and the center CPos of the minimum enclosing circle CR is used as the positioning data of the specific landmark, and the radius cr of the minimum enclosing circle CR is used as the system positioning deviation.

[0035] Furthermore, the positioning data of the detection equipment is corrected according to the positioning data of the specific landmark, including:

[0036] Establish a Cartesian coordinate system with the positioning data of a specific landmark as the origin, and convert the positioning data of the detection equipment into the coordinate value CRpos in the Cartesian coordinate system i ;

[0037] Convert the world coordinate system to ROPGPS i Deviation vector RPVec from the center CPos i As the coordinate value CRpos i The correction amount is used to obtain the corrected positioning data of the detection equipment.

[0038] Furthermore, judging whether the mobile phone is in motion based on the mobile phone positioning data set and the system positioning deviation includes:

[0039] Calculate the distance between each positioning point and other positioning points in the mobile phone positioning data set;

[0040] The detection interval value dist = 2*cr is established with the system positioning deviation as the radius, where cr is the system positioning deviation;

[0041] When the distance between any positioning point and other positioning points is greater than the detection interval value dist, it is determined that the mobile phone is in motion.

[0042] Furthermore, the mobile phone positioning data is corrected based on the corrected positioning data of the detection equipment to obtain the final positioning result of the mobile phone, including:

[0043] generating a second load coordinate system, wherein the second load coordinate system is a polar coordinate system with the corrected positioning data of the detection equipment as an origin;

[0044] For the kth mobile phone positioning data PPos T , generate the coordinates of the mobile phone in the second load coordinate system (ρ k ,ω k ,dis k ); where ρ k ,ω k are the horizontal and vertical plane deflection angles of the kth mobile phone relative to the center line of the detection equipment’s field of view, respectively. k Indicates relative distance;

[0045] According to the k-th mobile phone positioning data PPos T At the time point of , obtain the corresponding corrected positioning data pos′ of the detection equipment k ;

[0046] Let x k is the corrected positioning data pos′ of the detection equipment k Longitude in y k is the corrected positioning data pos′ of the detection equipment k Longitude in x T PPos for mobile phone positioning data T Longitude in y T PPos for mobile phone positioning data TLatitude in, distance d k =dis k *cos(ρ k );

[0047] Constructing a system of equations

[0048] Convert the system of equations into the equation EX=F; where the matrix matrix matrix

[0049] The least squares method is used to solve X in the equation EX=F to obtain the final positioning result of the mobile phone.

[0050] A target search and positioning system based on multi-mode data fusion, the system comprising:

[0051] A synthetic image generation module is used to obtain detection data of a specific landmark from a detection device and fuse the detection data to form a synthetic image; wherein the detection data includes radar data, optical data, and infrared data;

[0052] a first positioning data correction module, configured to obtain positioning data of the detection equipment, calculate positioning data of a specific landmark and a system positioning deviation based on the synthetic image, and correct the positioning data of the detection equipment based on the positioning data of the specific landmark;

[0053] The second positioning data correction module is used to collect mobile phone signals at multiple positioning points using detection equipment to generate a mobile phone positioning data set; based on the mobile phone positioning data set and the system positioning deviation, it is used to determine whether the mobile phone is in motion, and when the mobile phone is in a non-motion state, the mobile phone positioning data is corrected based on the positioning data of the corrected detection equipment to obtain the final positioning result of the mobile phone.

[0054] An electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned target search and positioning methods based on multi-mode data fusion.

[0055] Compared with the prior art, the present invention has at least the following beneficial effects.

[0056] The present invention proposes to process multi-point continuous search data based on multimodal data fusion to improve the comprehensiveness and accuracy of data processing. By fusing data from different modalities, the changing trends and internal connections of continuous data can be understood more accurately, thereby improving target search efficiency.

[0057] By extracting calibration objects from synthetic images as references, we can ensure that clear and stable reference points can be obtained under various environmental conditions, which helps to correct the positioning errors of the vehicle in a timely manner and improve positioning accuracy.

[0058] By establishing a detection window through the vehicle platform system deviation, the motion characteristics of the mobile phone can be more effectively captured to screen out moving mobile phones, thereby improving detection accuracy, enhancing robustness and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a technical framework diagram of a target search and positioning method based on multimodal data fusion according to an exemplary embodiment.

[0060] Figure 2 The main technical process.

[0061] Figure 3 For the search process and coordinate system establishment.

[0062] Figure 4 Schematic diagram of different detection payload data frames.

[0063] Figure 5 For the overall processing flow.

[0064] Figure 6 Pixel stretching and alignment.

[0065] Figure 7 Generate a processing pipeline for synthetic imagery.

[0066] Figure 8 It shows the relationship between the calibration object and the equipment.

[0067] Figure 9 This is a schematic diagram of multi-point observation ROP.

[0068] Figure 10 Schematic diagram of ROP deviation of multi-point observation.

[0069] Figure 11 Schematic diagram of ROP deviation prediction.

[0070] Figure 12 ROP relative coordinate system spatial relationship.

[0071] Figure 13 Vehicle spatial positioning correction process.

[0072] Figure 14 Schematic diagram of the spatial relationship between mobile phone and search equipment. DETAILED DESCRIPTION

[0073] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0074] like Figure 1 As shown, the technical framework of the present invention consists of four layers: access layer, preprocessing layer, correction layer and analysis layer.

[0075] in:

[0076] Access layer: Search equipment uses a vehicle platform (drone) to conduct searches in designated areas. During this process, multiple points of the target area are continuously sampled according to operating specifications. Vehicle positioning data and laser / imaging data generated during the sampling process are accessed as a real-time stream. Mobile phone signal searches are accessed in batches. Mobile phone signal search data is expressed in a relative coordinate system. During the access process, the data set is serialized and organized to facilitate subsequent processing.

[0077] Preprocessing layer: During the underlying data access process, the corresponding frames are extracted from the visible light image\radar detection data based on the timestamp corresponding to the search data batch. In this process, the laser and image frame extraction and synchronization processing are realized in a minimum window manner to form a synthetic image CVMap. At the same time, at a specific time point T0, a specific point in the synthetic image field of view is calibrated as the positioning reference point ROP0. In the subsequent sequence T i In the process of ROP calibration, the corresponding calibration ROP in the synthetic image at that moment is obtained i , thus providing a reference basis for subsequent positioning solution repair;

[0078] Correction layer: Based on the preprocessing of multiple synthetic frame data, the positioning distribution of ROP is extracted. Since ROP is a stationary object, a distribution surrounding CR can be established based on this. The center position of CR is used as the approximate positioning solution of ROP. The radius of CR is used as the system positioning deviation. At the same time, the position of ROP relative to the center of CR in different frames is used as relative motion, and the distance between the vehicle in the frame and the center of CR is used as the state parameter, thereby establishing the state equation and measurement equation, and then forming a Kalman filter. The filter is used to predict and correct the positioning deviation of the vehicle at different times. At the same time, the CR is used to establish discrimination constraints, screen the collected mobile phone positioning data, and mark and detect moving mobile phones;

[0079] Analysis Layer: After determining the motion characteristics of the mobile phone, the positioning results for non-moving mobile phones are corrected. During this process, a spatial relationship is established based on the relative distance between the mobile phone and the vehicle at the time of sampling and the vehicle's positioning coordinates (after corrections in the previous layer). A least squares solution is constructed to obtain an approximate true value of the current mobile phone signal location and output the result. This method improves the detection accuracy of non-moving mobile phones.

[0080] Through the above framework, the present invention processes multi-point continuous search data on the basis of multi-mode data fusion. The data it accesses include the vehicle's own high-precision positioning data, radar detection data, image data and mobile phone signal search data. During the access process, the vehicle's high-precision positioning data is encapsulated with radar detection data and image data for spatiotemporal attributes. In the incremental process, a time window is established with the sampling operation of the mobile phone search equipment as the moment point, and key frames are extracted from the radar detection and image to form a synthetic image. The reference point ROP is marked in the synthetic image. Thereafter, at different sampling moments, the relative coordinate values ​​of the reference point ROP in the corresponding synthetic image are sequentially extracted as observation quantities, and the corresponding positioning data is used as self-measurement. The positioning value of the vehicle in the world coordinate system and the positioning error of the system are obtained by Kalman filtering. On this basis, the mobile phone positioning data in multiple sampling data sets are subjected to coordinate transformation and time series processing. On the basis of the time series, spatial constraints are established, and position change detection is performed on the multiple moment positioning data of each mobile phone to form a detection and discrimination of motion characteristics, and finally a search result output application system is formed. Its overall structure is as follows: Figure 2 shown.

[0081] During the search process, the search payload arrives at the operation site via the carrier platform. The target area is searched according to the business specifications. Usually, the search operation is carried out in a closed path with the target area as the center. Figure 3 As shown, the vehicle establishes an operation path with the target area as the center, passing through multiple points ( Figure 3 During the search operation, the data to be processed include the following:

[0082] 1. Vehicle positioning data: This is the GPS positioning data of the vehicle, which uses the world coordinate system.

[0083] 2. Detection data: This includes radar detection and image acquisition data. This primarily involves sampling and detecting spatial objects within the field of view, as well as visual acquisition. During the incremental process, radar detection and imagery are fused to form a spatial calibration using relative coordinates, meaning the coordinates of the pixel calibration are relative to the current sampling point.

[0084] 3. Mobile phone search data: Mobile phone information and positioning data collected at specific points in the target area. Relative coordinates are used.

[0085] For ease of description, the coordinates in the present invention are expressed as follows:

[0086] World coordinate system: expressed in a coordinate system, such as longitude, latitude, and elevation in the WG84 coordinate system, where Lat is latitude, Lng is longitude, and Z is elevation.

[0087] Relative coordinate system: Td is the relative distance between the target and the origin of the coordinate system; A is the angle between the vector between the target and the origin of the coordinate system and the XY plane; B is the angle between the XY projection of the vector and the X axis.

[0088] During the data increment process, positioning data fusion, conversion and other processing are performed based on the mutual generation relationship between different data.

[0089] 1. Unified preprocessing of multimodal data.

[0090] 1.1 Unification of multimodal imaging data.

[0091] Although visible light, radar, and other detection methods have different imaging mechanisms, their imaging results are similar. Specifically, their imaging data is generated in frames and is a 2D structured dataset. This includes the coordinates of the 2D imaging plane and the corresponding physical quantities. These physical quantities correspond to optical and depth measurements. Based on this characteristic, a unified 2D imaging data representation is established, namely:

[0092] R={type,time,pos,{px ij |i=1,2,..n,j=1,2..m}}

[0093] px=(u,v,value)

[0094] in:

[0095] type is the imaging type, including visible light imaging, radar detection, infrared imaging, etc.

[0096] px is the pixel point, ij is the horizontal and vertical position number of the pixel;

[0097] m, n are horizontal and vertical pixel values;

[0098] time is the timestamp of the current imaging data frame;

[0099] pos is the external parameter data of the current imaging, which is defined as follows:

[0100] pos = {lat, lng, z, α, β, θ}, where lat, lng, and z are the spatial position coordinates of the imaging device in the world coordinate system at the current moment, and α, β, and θ are the rotation angles of the imaging device from the coordinate axis of the coordinate system;

[0101] u, v are the coordinate values ​​in the pixel coordinate system in the 2D imaging plane;

[0102] value is the sampling value corresponding to the current pixel. When the current image is visible light, its value is R\G\B value. When it is radar detection, its value is depth value Depth. When the current image is infrared imaging, it is infrared value Infra.

[0103] During operation, data frames generated by different means are uniformly encapsulated according to this data structure.

[0104] 1.2 Data synchronization and image synthesis.

[0105] During search operations, radar, visible light detection payloads, and mobile search equipment are typically used in combination. To improve processing efficiency, radar imaging and visible light imaging are combined to achieve a "holographic" two-dimensional image. This method can obtain depth, thermal infrared, and other detection values ​​for any pixel in the image's field of view, facilitating subsequent positioning and correction of the vehicle platform.

[0106] Due to the different detection mechanisms, the “frame rates” of data imaging vary. Figure 4 As shown, different imaging methods result in frame asynchrony. Furthermore, the relative motion between the target and the probe payload leads to mismatched target holographic information during pixel fusion and image synthesis. To address this issue, the present invention utilizes a sliding window as the basis, incorporating an accuracy assurance mechanism to achieve frame synchronization, meeting the requirements for temporal consistency of pixel-fused holographic information during multi-method image synthesis.

[0107] First, establish the detection equipment parameter expression, which is defined as follows:

[0108] DDev={type, ID, imgFreq, Rp}

[0109] Rp={R j |j=1,2,..m}

[0110] in:

[0111] Type is the detection payload type (such as image acquisition, radar detection, etc.), ID is the identifier of the payload, imgFreq is the data imaging frequency, and Rp is the frame sequence. When multiple means are combined, a payload data set is formed: DDevSet = {DDev i |i=1, 2, ..n}, according to the basic principle of target detection, the data with the highest imaging frequency (such as visible light image) is selected as the frame synchronization reference object, that is:

[0112] DDev main ∈DDevSet,imgFreq main ≥imgFreq, imgFreq∈DDev, DDev∈DDevSet

[0113] In T i When the vehicle moves to a specific position, in order to achieve positioning correction, it is necessary to synchronously obtain the corresponding data from the image and radar to create a synthetic image. Figure 5 As shown, the process is as follows:

[0114] 1. First, establish the window TWin based on the time difference tolerance fP;

[0115] 2. From DDev main Get the current T i The most recent data frame The corresponding time is T i 'Right now:

[0116] (T i ′-T i )<(T j -T i ),T j =Time(Tp j ), Tp i ∈Rp,Rp∈DDev main

[0117] Time is the time extraction operation, is the data frame at time j.

[0118] 3. If the current data frame corresponds to time T i ′ and T i The time difference is greater than the current window length constraint, indicating synchronization failure, and reselect T i time or exit the current operation.

[0119] 4. According to T i 'Establish forward window TW fw ′ and the backward window TW bk ′.

[0120] 5. From DDev k In the data sequence, extract TW fw ′\TW bk ′ and T i 'The most recent data frame R fw ′\R bk ';

[0121] T fw ′=Time(R fw ′), T bk ′=Time(R bk ′)

[0122] 6. Compare T fw ′ and T bk ′, take away T i The frame corresponding to the latest time is taken as the synchronization frame R k,i ';

[0123] iif(|T fw ′-T i |<|T bk ′-T i |)then R=R fw ′else R=R bk '

[0124] R fw ′∈Rp,R bk ′∈Rp,Rp∈DDev k

[0125] 7. After completing the above operations, obtain the different DDev in the current T i Synchronous frame dataset at time.

[0126] Rs={Tp i ′,{R k,i ′|k=1,2,...n}},R k ∈DDev k ,DDev k ≠DDev main ,DDev k ∈DDevSet

[0127] After the synchronization frame is extracted, the synthesis processing operation can be performed.

[0128] Due to the differences in technical mechanisms, the imaging resolutions of different means such as visible light image acquisition and radar detection are different. After the corresponding key frames are extracted, in order to facilitate subsequent processing operations, the above-mentioned imaging with different resolutions needs to be synthesized. As mentioned above, the key frame set RS includes T i Optical images, radar images, infrared images, and other frames within a specific range at a given moment. First, the frame with the highest resolution is obtained from the synchronized frame dataset Rs. The data of the other frames is processed based on its resolution to achieve pixel alignment across the different imaging frames. The sampled physical quantities corresponding to the pixels in the different frames are then matched to the pixels to form a composite image.

[0129] 1. Traverse RS and obtain the maximum value of 2D uv from all data frames:

[0130] u max =Max({u p |p=1,2,...n}),u p =size(px.u),px∈R k ,R k ∈Rs

[0131] v max =Max({v p|p=1,2,...n}),v p =size(px.v),px∈R k ,R k ∈Rs

[0132] 2. According to u max With v max Construct a resolution dataset of synthetic images:

[0133] CVMap={px u,v |u=1,2,...u max ,v=1,2,...v max}

[0134] 3. Based on the uv of CVMap, align each data frame R in each RS. First, stretch the data frame R proportionally, and stretch the u direction to u. max , v direction stretching is v max During the stretching process, firstly, the maximum value of u and v (u rmax , v rmax ) and (u max , v max ) to calculate the linear stretch ratio:

[0135] ur=u max / u rmax ,vr=v max / v rmax

[0136] 4. According to the stretching ratio, the original pixels in R are first stretched to form R′.

[0137] u'=ur*u,v'=vr*v,(u,v,value)∈px ij ,px ij ∈R

[0138] R'={type,time,pos,{px' o,p |o=1,...umax,p=1,...vmax}}

[0139] px'=(u',v',value)

[0140] After completing the above treatment, the R′ formed is as follows Figure 6 gesture.

[0141] like Figure 6 As shown in the figure, since ur and vr are usually not integer ratios, u′ and v′ in R after stretching are usually not the correct pixel position values. Therefore, linear interpolation is required to obtain the correct pixel position (uc, vc).

[0142] 5. Take the adjacent four corners p1=(u′ i ,v′ j ), p2=(u′ i+1 ,v′ j ), p3=(u′ i+1 ,v′ j+1 ), p4=(u′ i ,v′ j+1 ),i′ i The value is rounded up uidx1=ceil(u′ i ), u′ i+1 The value is rounded down uidx2=floor(u′ i+1 ), v′ j The value is rounded up vidx1=ceil(v′ j ), v′ j+1 The value is rounded down vidx2=floor(v′ j+1 ), forming a new point matrix data set:

[0143] uvc={(uc x ,vc y )|x=uidx1,..uidx2,y=vidx1,...vidx2}

[0144] The value of each newly generated point in the dot matrix uvc is calculated by the aforementioned linear interpolation of px′, that is:

[0145] vaule1=(uc x -u' i,j )(value i+1,j -value i,j ) / (u' i+1,j -u' i,j )

[0146] vaule2=(uc x -u' i,j+1 )(value i+1,j+1 -value i,j+1 ) / (u' i+1,j+1 -u' i,j+1 )

[0147] value=(value2-value1) / (v' i+1,j+1 -v' i+1,j )

[0148] This gives pxc x,y =(uc x ,uc y ,value).

[0149] 6. After all alignment operations of the current data frame are completed, a new data frame Rc is formed.

[0150] u'=ur*u,v'=vr*v,(u,v,value)∈px ij ,px ij ∈R

[0151] R'c={type,time,pos,{pxc x,y |x=1,...umax,y=1,...vmax}}

[0152] type∈R',time∈R',pos∈R',pxc x,y =(uc x ,vc y ,value)

[0153] 7. After processing all frames in Rs, the pixels of these frames are completely aligned and valid values ​​are obtained. These frames are merged based on the pixels. Each pixel ultimately forms a composite image that is a result of multi-method collaboration. Its pixel expression is defined as follows:

[0154] CVMap={time,pos,{px x,y |x=1,2,...u max ,y=1,2,...v max}}

[0155] px={x,y,vals},vals={value k x,y |k=1,2..n}

[0156] in:

[0157] The synthetic image CVMap consists of a set of pixels px that retain the holographic information vals.

[0158] vals is the physical quantity extracted from different frames, including visible light RGB value, radar detection depth value, infrared value, etc., which uses pixel reconstruction to realize the fusion processing operation of multimodal data imaging, laying the foundation for subsequent target calibration. This process is as follows Figure 7 shown.

[0159] 2. Vehicle platform positioning correction and system deviation distribution acquisition.

[0160] During operation, the vehicle platform itself suffers from positioning system deviations. Therefore, a mechanism is needed to eliminate these deviations. This invention utilizes synthetic imagery to extract a calibration object as a reference (ROP). During operation, the relative positional relationship between the calibration object and the vehicle is extracted while searching at multiple operating points. This information is then filtered and combined with the vehicle's own positioning to calculate the spatial position. This information also captures system deviations during operation, facilitating target positioning corrections.

[0161] The calibration object is usually a landmark object on site, which is usually a stationary object. The calibration object is extracted and locked by image. The extraction and locking technology itself is not within the scope of this invention, and the specific technical principles are not described here. In this process, the relationship between coordinates and space is as follows: Figure 8 As shown in Figure 2, in this spatial relationship, the expression of the calibration object ROP in the world coordinate system is:

[0162] ROPGPS = {Lat, Lng, Z}

[0163] Among them: Lat, Lng are longitude and latitude, and Z is elevation.

[0164] The load is expressed in the world coordinate system as:

[0165] EPPos={EPPGS, α, β, θ}, EPGPS= {Lat, Lng, Z}

[0166] The definitions of Lat, Lng, and Z are consistent with those of ROPGPS, and α, β, and θ are the rotation angles of the load from the coordinate axis of the coordinate system.

[0167] The pixel coordinates of ROP in the current synthetic image CVMap are

[0168] ROPCMPPos={u ROP , v ROP}

[0169] During operation, the following coordinate processing is performed:

[0170] 1. Get the depth value Depth from the corresponding vals of (UROP, VROP) from the synthetic image CVMap. Calculate the relative coordinate angles A and B of (UROP, VROP) based on the payload imaging parameters. Where:

[0171] A is the angle between the vector between ROP and the origin of the load coordinate system and the XY plane;

[0172] B is the angle between the XY projection of the vector and the X axis.

[0173] Thus, the relative coordinates of ROP and load are formed

[0174] ROPRePos={A,B,Depth}

[0175] 2. With the load as the origin, form a Cartesian coordinate system and convert ROPRePos into Cartesian coordinates under the load coordinate system:

[0176] ROPPos'={X',Y',Z'}

[0177] X'=Depth*cos(A)*cos(B)

[0178] Y'=Depth*cos(A)*sin(B)

[0179] Z'=Depth*sin(A)

[0180] 3. Construct the inverse transformation matrix AT based on the rotation angles α, β, and θ in the payload EPPOS, and construct the space transformation matrix based on EPGPS. The specific spatial geometry algorithm will not be described here. Through the above operations, ROPPos' is converted to the world coordinate: ROPGPS.

[0181] During operation, the deviation composition relationship can be established: synthetic image error + system deviation. Synthetic image error can be derived from payload parameters, while system deviation is influenced by multiple factors during operation. It is highly dynamic and has a significant impact on target positioning. Therefore, it is the primary basis for positioning correction calculations.

[0182] 2.1 System deviation acquisition.

[0183] ROP is a stationary landmark. Therefore, during the multi-point continuous observation process, its world coordinates should be the same. Figure 9 Middle ROPGPS1=ROPGPS2=ROPGPS3=ROPGPS4....

[0184] In actual operation, due to the positioning deviation of the system itself, there are deviations in the ROP obtained from the synthetic images at different points, such as Figure 10 As shown:

[0185] Therefore, the system deviation leads to inconsistent positioning of static ROP during multi-point observation, which then forms a distribution. To address this situation, the present invention extracts the coordinate distribution of ROP from multiple points as the system deviation, providing calculation support for subsequent target positioning correction. The process is as follows:

[0186] 1. During the operation of multiple points, establish a corresponding synthetic image set:

[0187] CVMaps = {CVMap i |i=1,2,...n}

[0188] POS i ∈CVMap i ,pos i =EPGPS i ,EPGPS i ∈EPPos i

[0189] 2. Extract the coordinates of ROP from each synthetic image to form a sequence:

[0190] ROPGPSs = {ROPGPS i |i=1,2,...n}

[0191] 3. Establish the minimum enclosing circle CR, that is, with CPos as the center and cr as the radius, and the entire ROPGPS is surrounded by CR:

[0192] CR={CPos, cr}, CPos=(Lat, Lng, Z)

[0193] In the present invention, CPos is regarded as the approximate positioning of ROP, and cr is the system deviation.

[0194] 2.2 Establishment of positioning deviation prediction model and correction of vehicle platform positioning data.

[0195] Once the CR is obtained, the system deviation can be considered. This deviation can be used to correct the positioning of the vehicle platform, thereby improving the target positioning accuracy. During the search operation, the vehicle platform is in motion. Its own positioning is calculated through satellite, inertial navigation, and other methods to form a self-measurement. This is recorded as:

[0196] iGPos={(pos, time) i |i=1,2,..n}

[0197] pos={Lat,Lng,Z}

[0198] In combination with the static characteristics of ROP and the change of the relative position relationship between the vehicle and ROP, the present invention establishes a coordinate system with CPos as the origin, and uses different time points T k The relative changes of ROPGPSk and ROPGPSk-1 extracted from the synthetic image frame form an observation sequence. Figure 11 shown.

[0199] Based on this sequence, a Cartesian coordinate system is established with CPos as the origin, and the positioning coordinates of the vehicle are converted into coordinate values ​​in this coordinate system, namely:

[0200] cRPos i ={cRLat,cRLng,cRZ} i ,

[0201] cRLat i =pos.Lat i -CPos.Lat i ,cRLng i =pos.Lng i -CPos.Lng i ,cRZ i =pos.Z i -CPos.Z i

[0202] In this coordinate system, T i Time ROPGPS i Deviation vector RPVec from CPos i Can be used as CRpos i The correction amount, namely CRPos′ i =CRpos i -RPVec i , such as Figure 12 gesture.

[0203] Therefore, a prediction model can be established on the time series, that is, i At this moment, the vehicle coordinates are converted to CRpos i Then, use this value as the initial value and set the vector RPVec at this time i As the compensation amount, the correction amount CRPos′ is obtained by adding the two together. i A Kalman state prediction model can be established on the time series to achieve the specific moment T k Compensation RPVec k Make a prediction and then realize the positioning coordinate CRpos at that moment k Make corrections and convert them into pos in the world coordinate system k The process is as follows:

[0204] 1. Establish a Cartesian coordinate system with CPos as the origin, convert the pos value in iGPos to CRPos, and then get the sequence:

[0205] {(CRPos, time) i |i=1,2,.....m}

[0206] 2. Take time T i With T i+1 , due to T i With T i+1 The interval is small, and this motion has the characteristics of uniformly accelerated linear motion in this time interval. The following motion parameter calculation is established:

[0207]

[0208] 3. Establish the following motion relationship in the time series

[0209]

[0210] Generally speaking, these variables can be considered to be stationary random sequences with a normal distribution of mean 0 and variance 2 in the time dimension, and are uncorrelated with each other. That is:

[0211] E{ax(i)}=0

[0212]

[0213] Comprehensively consider the distance between the vehicle and CPos and T i The state equation corresponding to the position relationship of ROPPos relative to CPos at each moment is:

[0214] X(i+1)=ΦX(i)+ΓW(i)

[0215] in:

[0216]

[0217] 4. In T i At this moment, the following correction relationship is established:

[0218] CRPos' i =CRPos i -RPVec i ,RPVec i =RPOGPS i -CPos

[0219] From this we can establish the measurement equation:

[0220] Z(i)=H(i)X(i)+V(i)

[0221] Among them, Z(i) is the corrected coordinate at that moment, that is, GRPos′ i , X(i) is the value of the state equation, V(i) is the RPVec corresponding to the current moment i The value of , H(i) is the measurement matrix for model solution.

[0222] On the basis of the above equations, a Kalman filter is established. The method for establishing the Kalman filter is beyond the scope of the present invention and will not be described in detail here.

[0223] 5. Using this filter, take T0 as the starting frame and establish deviation prediction with multi-point detection process to achieve T kThe prediction of time V(k), that is, the offset vector RPVec k :

[0224] The offset vector is used to calculate the positioning deviation correction of the vehicle platform at that moment:

[0225] iGPos'={(pos', time) i |i=1,2,..n}

[0226] pos' k =pos k -RPVec k

[0227] This process is like Figure 13 shown.

[0228] 2.3 Mobile phone search data fusion processing and analysis.

[0229] During operation, the mobile phone search equipment collects, calculates and processes mobile phone signals at specific points and generates positioning results. Usually, the positioning generated is the relative coordinates with the equipment as the origin, that is:

[0230] phoneOPPos=(ρ,ω,dis)

[0231] Where ρ and ω are the horizontal and vertical plane deflection angles relative to the center line of the mobile phone search equipment's field of view, and dis is the relative distance. After the coordinates are converted into Cartesian coordinates (x, y, z), they need to be superimposed with the vehicle coordinates pos to form the positioning in the world coordinate system, that is:

[0232]

[0233] During a search operation, the vehicle uses a multi-point search method around the target area. During this process, the mobile phone search positioning data forms a time series. It is defined as follows:

[0234] phoneResults={phone i |i=1,2.......m}

[0235] phone={IMSI,type,PPoss}

[0236] Among them: IMSI is the mobile phone identification, type indicates whether it is a mobile phone, the default value is 1, and the value is 1 when it is determined to be a sports phone.

[0237] PPoss={(phoneOPPos,time) i |i=1,2,...n}

[0238] Due to the superposition of multiple factors, mobile phone positioning data at multiple moments usually has certain deviations. At the same time, since the mobile phone is in an uncertain state, it is not possible to simply eliminate the difference of the positioning result data in this sequence. To address this problem, the present invention establishes a detection window through the aforementioned vehicle platform system deviation to identify the motion characteristics of the mobile phone. The details are as follows:

[0239] 2.3.1 Identification and screening of mobile phone motion characteristics.

[0240] As mentioned above, during operation, the mobile phone searches for positioning result data at multiple times to form a sequence

[0241] 1. Calculate the distance between each anchor point in the sequence and other anchor points to form a data set:

[0242] PPosDists={(||phoneOPPos i ,phoneOPPos k ||) i,k |i=1,2,...n-1,k=i+1,i+2,...n,i≠k}

[0243] 2. Use the cr in the system deviation CR obtained above as the radius to establish the detection interval value. If any value in PPosDists is greater than cr*2, the current mobile phone is identified as a sports phone and marked, that is:

[0244] iif(dist>2*cr)then phone.type=1,dist∈PPosDists

[0245] 3. After all mobile phones are processed, the mobile phones marked as sports will be deleted from the current search results, that is:

[0246] phoneResults'∈phoneResults

[0247] type=0,type∈phone,phone∈phoneResults'

[0248] Correct the positioning results of the remaining mobile phones.

[0249] 2.3.2 Correction of non-mobile phone positioning results.

[0250] Mobile phone positioning is mainly based on 2D. Therefore, in the present invention, the multi-point positioning data of the mobile phone is corrected in the horizontal plane, as shown in the figure below. Figure 14 As shown:

[0251] 1. First, obtain the complete mobile phone positioning result data set and establish the mobile phone positioning true value as PPo sT

[0252] PPoss={(phoneOPPos,time) i |i=1,2,...n}

[0253] PPos T =(Lat,lng,Z)

[0254] 2. Get the vehicle position data pos corresponding to the time point in PPoss and create a data set:

[0255] PCPoss={PCPos i |i=1,2,…,n},PCPos=(phoenOPPos,time,pos)

[0256] 3. Set i = k and obtain data from PCPoss to form the following relationship:

[0257] x k =pos k .Lat,y k =pos k .Lng

[0258] x T =PPos T .Lat,y T =PPos T .Lng

[0259] d k =dis k *cos(α),dis k ,α∈phoneOPPos k

[0260] 4. Complete the establishment of all solution equations:

[0261]

[0262] From this we can get: EX=F

[0263]

[0264] 5. Use least squares to solve X(x T ,y T ) is solved to form the correction of the mobile phone positioning result. The specific calculation process of least squares will not be repeated here.

[0265] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A target search and positioning method based on multimodal data fusion, characterized in that: The method comprises: Acquire detection data of a specific landmark by detection equipment, and fuse the detection data to form a synthetic image; wherein the detection data includes radar data, optical data, and infrared data; Acquiring positioning data of the detection equipment, and calculating positioning data of a specific landmark and a system positioning deviation based on the synthetic image, and then correcting the positioning data of the detection equipment based on the positioning data of the specific landmark; Use detection equipment to collect mobile phone signals at multiple positioning points to generate a collection of mobile phone positioning data; Based on the mobile phone positioning data set and the system positioning deviation, it is determined whether the mobile phone is in motion. When the mobile phone is in a non-motion state, the mobile phone positioning data is corrected based on the positioning data of the corrected detection equipment to obtain the final positioning result of the mobile phone.

2. The method according to claim 1, characterized in that The detection data is fused and processed to form a synthetic image, including: Taking the detection data with the highest imaging frequency as a benchmark, extract the synchronous frames in each detection data to generate a synchronous frame data set; The synchronous frames in the synchronous frame dataset are fused to obtain a synthetic image.

3. The method according to claim 2, characterized in that Taking the detection data with the highest imaging frequency as the benchmark, the synchronous frames in each detection data are extracted to generate a synchronous frame data set, including: Setting a time difference tolerance and establishing a window according to the time difference tolerance; Obtain the detection data with the highest imaging frequency at time T i The most recent dataframe Among them, the data frame The corresponding time T i ′ and time T i The difference is no greater than the window length constraint; Establishment time T i ′’s forward window TW fw ′ and the backward window TW bk '; For detection data with a non-highest imaging frequency, in the forward window TW fw ′ and backward window TW bk ′ and time T i 'The most recent data frame R fw ′ and data frame R bk '; Comparing data frames in R fw ' and the data frame R bk ′, and the distance time T i ′The closest data frame is taken as the data frame synchronization frame.

4. The method according to claim 2, characterized in that The synchronized frames in the synchronized frame dataset are fused to obtain a synthetic image, including: Traverse the data frame R in the synchronized frame dataset and obtain the maximum pixel value u of the resolution mxx and the maximum pixel value v mxx ; Based on the maximum pixel value u mxx and the maximum pixel value v mxx The data frame R is stretched to obtain the data frame R′; wherein the pixel values ​​of the four corners (p1, p2, p3, p4) of the data frame R′ are: p1=(u′ i ,v′ j ), p2=(u′ i+1 ,v′ j ), p3=(u′ i+1 ,v′ j+1 ), p4=(u′ i ,v′ j+1 ); For pixel value u′ i and pixel value v′ j Round up and calculate the pixel value u′ i+1 and pixel value v′ j+1 Round down to form a dot matrix data set uvc = {(uc x ,vc y )}|x=uidx1,…,uidx2,y=vidx1,…,vidx2}; where uidx1, uidx2, vidx1 and vidx2 are the pixel values ​​u′ i , pixel value u′ i+1 , pixel value v′ j and pixel value v′ j+1 The rounded result of the value of each newly generated pixel in the dot matrix data set uvc is obtained by bilinear interpolation and encapsulated in the corresponding attribute of the newly generated pixel; Based on the dot matrix data set uvc, the synchronization frames of different detection data are aligned to obtain the data frame Rc; Based on the data frame Rc, the synchronous frames of different detection data are fused to obtain a synthetic image.

5. The method according to claim 1, wherein Obtain positioning data from detection equipment and combine it with synthetic imagery to calculate the positioning data of specific landmarks and system positioning deviations, including: Acquire detection equipment to form a synthetic image i Positioning data expression EPPos i ={ROPGPS i ,α i ,β i ,θ i }, α i ,β i ,θ i is the rotation angle of the first load coordinate system axis, ROPGPS i ={Lat i ,Lng i ,Z i } is the positioning data expression of the detection equipment in the world coordinate system, Lat i Longitude of the detection equipment, Lng i is the longitude of the detection equipment, Z i To detect the height of the equipment, the first load coordinate system is formed by forming a synthetic image image i The polar coordinate system is formed by taking the position of the detection equipment at the time as the origin; Get a specific landmark in the synthetic image i Pixel coordinates The corresponding radar data depth value Depth i , and calculate the pixel coordinates The relative coordinate angle A i and relative coordinate angle B i After that, the relative coordinates ROPRPos between the specific landmark and the detection equipment are obtained i ={A i ,B i ,Depth i }; Among them, the relative coordinate angle A i The angle between the vector between the specific landmark and the origin of the first load coordinate system and the XY plane of the world coordinate system, the relative coordinate angle B i The angle between the X-axis and the projection of the vector between the specific landmark and the origin of the first load coordinate system in the XY plane of the world coordinate system; To form a synthetic image i The position of the detection equipment at that time is the origin, a Cartesian coordinate system is established, and the relative coordinate ROPRPos i Converted to Cartesian coordinate system ROPRPos i '; According to the rotation angle α i , rotation angle β i and the rotation angle θ i Construct the inverse transformation matrix and express it according to the positioning data Construct a spatial matrix and transform the coordinate ROPRPos based on the inverse transformation matrix and the spatial transformation matrix i 'Convert world coordinate system coordinates ROPGPS i ; All world coordinate systems are based on ROPGPS i A minimum enclosing circle CR is established, and the center CPos of the minimum enclosing circle CR is used as the positioning data of the specific landmark, and the radius cr of the minimum enclosing circle CR is used as the system positioning deviation.

6. The method according to claim 5, characterized in that Correct the positioning data of the detection equipment based on the positioning data of specific landmarks, including: Establish a Cartesian coordinate system with the positioning data of a specific landmark as the origin, and convert the positioning data of the detection equipment into the coordinate value CRpos in the Cartesian coordinate system i ; Convert the world coordinate system to ROPGPS i Deviation vector RPVec from the center CPos i As the coordinate value CRpos i The correction amount is used to obtain the corrected positioning data of the detection equipment.

7. The method according to claim 1, characterized in that Determine whether the phone is in motion based on the phone's location data set and system location deviation, including: Calculate the distance between each positioning point and other positioning points in the mobile phone positioning data set; The detection interval value dist = 2*cr is established with the system positioning deviation as the radius, where cr is the system positioning deviation; When the distance between any positioning point and other positioning points is greater than the detection interval value dist, it is determined that the mobile phone is in motion.

8. The method according to claim 1, characterized in that The mobile phone positioning data is corrected based on the corrected positioning data of the detection equipment to obtain the final positioning result of the mobile phone, including: generating a second load coordinate system, wherein the second load coordinate system is a polar coordinate system with the corrected positioning data of the detection equipment as an origin; For the kth mobile phone positioning data PPos T , generate the coordinates of the mobile phone in the second load coordinate system (ρ k ,ω k ,dis k ); where ρ k ,ω k are the horizontal and vertical plane deflection angles of the kth mobile phone relative to the center line of the detection equipment’s field of view, respectively. k Indicates relative distance; According to the k-th mobile phone positioning data PPos T At the time point of , obtain the corresponding corrected positioning data pos′ of the detection equipment k ; Let x k is the corrected positioning data pos′ of the detection equipment k Longitude in y k is the corrected positioning data pos′ of the detection equipment k Longitude in x T PPos for mobile phone positioning data T Longitude in y T PPos for mobile phone positioning data T Latitude in, distance d k =dis k *cos(ρ k ); Constructing a system of equations Convert the system of equations into the equation EX=F; where the matrix matrix matrix The least squares method is used to solve X in the equation EX=F to obtain the final positioning result of the mobile phone.

9. A target search and positioning system based on multi-mode data fusion, characterized in that: The system comprises: A synthetic image generation module is used to obtain detection data of a specific landmark from a detection device and fuse the detection data to form a synthetic image; wherein the detection data includes radar data, optical data, and infrared data; a first positioning data correction module, configured to obtain positioning data of the detection equipment, calculate positioning data of a specific landmark and a system positioning deviation based on the synthetic image, and correct the positioning data of the detection equipment based on the positioning data of the specific landmark; The second positioning data correction module is used to collect mobile phone signals at multiple positioning points using detection equipment to generate a mobile phone positioning data set; based on the mobile phone positioning data set and the system positioning deviation, it is used to determine whether the mobile phone is in motion, and when the mobile phone is in a non-motion state, the mobile phone positioning data is corrected based on the positioning data of the corrected detection equipment to obtain the final positioning result of the mobile phone.

10. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the target search and positioning method based on multimodal data fusion as described in any one of claims 1-8.

Images