Multi-view three-dimensional target detection method, device, equipment and storage medium

Abstract

The present invention provides a multi-view three-dimensional target detection method, device, equipment and storage medium based on a lightweight remote sensing basic model, which can be applied to the field of image processing technology. The method includes: inputting an optical remote sensing image to be detected of the same scene shot from multiple viewpoints into a trained lightweight remote sensing basic model to obtain multiple feature maps under the multiple viewpoints, and the lightweight remote sensing basic model is used to extract the feature map of the optical remote sensing image; according to the feature pixel depth of each feature map, all the feature maps are converted to the same world coordinate system to generate a bird's-eye view feature map; the bird's-eye view feature map is sent to an FPN network to obtain a bird's-eye view feature map after scale feature fusion; and the bird's-eye view feature map after scale feature fusion is used to perform three-dimensional target detection, which can improve the accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the field of optical communications, and in particular to a multi-view three-dimensional target detection method, device, equipment and storage medium based on a lightweight remote sensing basic model. Background Art

[0002] At present, remote sensing basic models have achieved excellent performance in various downstream tasks, but the high demand for computing resources limits the application of these models on edge devices. Therefore, supporting on-orbit remote sensing image interpretation based on lightweight basic models is a key research direction.

[0003] Among the downstream tasks of remote sensing basic models, multi-view 3D object detection is a cutting-edge research field. It uses remote sensing images taken from different perspectives of the same area to detect the position and range of the target in a unified 3D world coordinate system, thereby making better use of the perception scene of remote sensing images. The current multi-view 3D object detection method has poor detection accuracy. Summary of the invention

[0004] In view of the above problems, the present invention provides a multi-view three-dimensional target detection method, device, equipment and storage medium based on a lightweight remote sensing basic model.

[0005] According to a first aspect of the present invention, a multi-view three-dimensional target detection method based on a lightweight remote sensing basic model is provided, comprising:

[0006] Inputting optical remote sensing images to be detected of the same scene taken from multiple perspectives into the trained lightweight remote sensing basic model to obtain a plurality of feature maps under the multiple perspectives, wherein the lightweight remote sensing basic model is used to extract the feature map of the optical remote sensing image;

[0007] According to the feature pixel depth of each feature map, all the feature maps are converted into the same world coordinate system to generate a bird's-eye view feature map;

[0008] Sending the bird's-eye view feature map to the FPN network to obtain a bird's-eye view feature map after scale feature fusion;

[0009] The bird's-eye view feature map obtained by fusion of the scale features is used to perform three-dimensional target detection.

[0010] In some embodiments, converting all the feature maps to the same world coordinate system to generate a bird's-eye view feature map according to the feature pixel depth of each feature map comprises:

[0011] Performing reverse projection on the feature pixels on the feature map to obtain a reverse projection result;

[0012] A two-dimensional bird's-eye view network is set in a world coordinate system, wherein the two-dimensional bird's-eye view network includes a plurality of two-dimensional bird's-eye view grids;

[0013] Project the reverse projection of each feature map onto the two-dimensional bird's-eye view network to obtain the projection result;

[0014] Average pooling is used on the projection results so that there is only one feature vector in each two-dimensional bird's-eye view grid, and a two-dimensional bird's-eye view feature map is obtained.

[0015] In some embodiments, performing reverse projection on the feature pixels on the feature map to obtain a reverse projection result includes:

[0016] Setting a discrete depth range for feature pixels on the feature map to form a three-dimensional viewing cone for each feature map;

[0017] Weighting the feature vector of the feature pixel on the feature map with a preset depth probability to obtain a weighted feature vector;

[0018] Placing the weighted feature vector in a corresponding three-dimensional viewing cone to obtain a three-dimensional viewing cone for each feature map in different image coordinate systems;

[0019] The three-dimensional view cone of each feature map based on different image coordinate systems is converted to a unified world coordinate system to obtain a reverse projection of each feature map.

[0020] In some embodiments, the method further comprises:

[0021] Calculate the reprojection loss of all the points with the same name in the feature map;

[0022] The three-dimensional frustum of each of the feature maps is optimized using the reprojection loss.

[0023] In some embodiments, calculating the reprojection loss of all the same-name points in the feature maps comprises:

[0024] Calculating phase characteristic diagrams of all the characteristic diagrams using a phase consistency method;

[0025] Performing phase feature map matching between all the feature maps in pairs using a normalized cross-correlation method to obtain multiple pairs of matching results, each pair of the matching results including two phase feature maps;

[0026] For each pair of matching results, a feature point is selected on one phase feature map, and a target area is set with the feature point as the center, and the other phase feature map is the area to be matched;

[0027] Slide the phase characteristic map of the target area in the phase characteristic map of the area to be matched, calculate the normalized mutual correlation coefficient, and the peak value of the correlation coefficient is the same name point;

[0028] Calculate the reprojection loss of a pair of points with the same name located on the two feature maps respectively.

[0029] In some embodiments, the method further comprises:

[0030] Based on multiple sets of optical remote sensing images of the same scene taken from multiple perspectives, a multi-view 3D object detection dataset is constructed;

[0031] Using the multi-view three-dimensional target detection data set, training the lightweight remote sensing basic model to be trained, and obtaining the trained lightweight remote sensing basic model;

[0032] Wherein, the position and range of the target object are marked in the world coordinate system where the scene is located as the training truth value.

[0033] In some embodiments, constructing a multi-view 3D object detection dataset based on multiple sets of optical remote sensing images of the same scene taken from multiple viewpoints includes:

[0034] Collect multiple sets of optical remote sensing images of the same scene taken from multiple perspectives;

[0035] For each of the optical remote sensing images, annotate a transformation matrix of the camera coordinate system of the optical remote sensing image relative to the world coordinate system;

[0036] According to the transformation matrix, the multi-view three-dimensional object detection data set is constructed.

[0037] A second aspect of the present invention provides a multi-view three-dimensional target detection device based on a lightweight remote sensing basic model, comprising:

[0038] An input module is used to input the optical remote sensing images to be detected of the same scene taken from multiple perspectives into the trained lightweight remote sensing basic model to obtain a plurality of feature maps under the multiple perspectives, and the lightweight remote sensing basic model is used to extract the feature map of the optical remote sensing image;

[0039] A generating module, used for converting all the feature maps into the same world coordinate system to generate a bird's-eye view feature map according to the feature pixel depth of each feature map;

[0040] A feeding module is used to feed the bird's-eye view feature map into the FPN network to obtain a bird's-eye view feature map after scale feature fusion;

[0041] The detection module is used to perform three-dimensional target detection using the bird's-eye view feature map after the scale feature is fused.

[0042] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0043] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0044] According to the multi-view three-dimensional target detection method, device, equipment and medium based on the lightweight remote sensing basic model provided by the present invention, according to the feature pixel depth of each feature map, all feature maps are converted to the same world coordinate system to generate a bird's-eye view feature map, that is, through the perspective transformation mechanism, the multi-view feature maps are matched and the same-name points are extracted, and the reprojection errors of these same-name points are used as depth supervision signals to participate in the depth optimization to obtain better depth estimation, and at the same time, there is no need to consume additional computing resources, which significantly improves the accuracy and robustness of target detection, and is significantly better than the existing methods in terms of the three indicators of mAP, mTE and mSE. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0046] Figure 1 A flowchart of a multi-view 3D target detection method based on a lightweight remote sensing basic model according to an embodiment of the present invention is schematically shown;

[0047] Figure 2 The schematic diagram shows an architecture diagram of a multi-view 3D target detection method based on a lightweight remote sensing basic model according to an embodiment of the present invention;

[0048] Figure 3 A structural block diagram of a multi-view 3D target detection device based on a lightweight remote sensing basic model according to an embodiment of the present invention is schematically shown;

[0049] Figure 4 A block diagram of an electronic device suitable for implementing a multi-view three-dimensional target detection method based on a lightweight remote sensing basic model according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0050] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0051] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0052] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0053] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0054] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision, invention and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0055] In the technical solution of the present invention, the acquisition, collection, storage, use, processing, transmission, provision, invention and application of data all comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0056] Figure 1 The flowchart of the multi-view 3D target detection method based on the lightweight remote sensing basic model according to an embodiment of the present invention is schematically shown. Figure 2 The schematic diagram shows an architecture diagram of a multi-view 3D target detection method based on a lightweight remote sensing basic model according to an embodiment of the present invention.

[0057] like Figure 1As shown, the multi-view 3D target detection method based on a lightweight remote sensing basic model of this embodiment includes operations S110 to S140.

[0058] In operation S110, optical remote sensing images to be detected of the same scene shot from multiple perspectives are input into the trained lightweight remote sensing basic model to obtain a plurality of feature maps under the multiple perspectives.

[0059] In operation S120, according to the feature pixel depth of each feature map, all the feature maps are converted to the same world coordinate system to generate a bird's-eye view feature map.

[0060] In operation S130, the bird's-eye view feature map is sent to the FPN network to obtain a bird's-eye view feature map after scale feature fusion.

[0061] In operation S140, three-dimensional object detection is performed using the bird's-eye view feature map obtained by fusion of the scale features.

[0062] In some embodiments, the lightweight remote sensing basic model in the above operation S110 is used to extract the feature map of the optical remote sensing image. The lightweight remote sensing basic model can be trained in the following manner: based on multiple groups of optical remote sensing images of the same scene taken from multiple perspectives, a multi-perspective three-dimensional target detection data set is constructed; using the multi-perspective three-dimensional target detection data set, the lightweight remote sensing basic model to be trained is trained to obtain the trained lightweight remote sensing basic model, wherein the position and range of the target object are marked in the world coordinate system where the scene is located as the training truth value.

[0063] Specifically, multiple groups of optical remote sensing images of the same scene taken from multiple perspectives are collected, a world coordinate system is set for the scene, and the position (x, y, z) and range (w, h, l) of the target object are marked in this coordinate system as the training truth value. For each optical remote sensing image, the transformation matrix of its camera coordinate system relative to the world coordinate system is marked. The internal parameters of the camera come from the information carried by the remote sensor itself, so as to construct a multi-perspective 3D target detection dataset based on a lightweight remote sensing basic model.

[0064] In some embodiments, operation S120, based on the feature pixel depth of each feature map, converting all the feature maps to the same world coordinate system to generate a bird's-eye view feature map includes: reversely projecting the feature pixels on the feature map to obtain a reverse projection result; setting a two-dimensional bird's-eye view network in the world coordinate system, the two-dimensional bird's-eye view network including multiple two-dimensional bird's-eye view grids; projecting the reverse projection of each feature map onto the two-dimensional bird's-eye view network to obtain a projection result; using average pooling on the projection result so that there is only one feature vector in each two-dimensional bird's-eye view grid to obtain a two-dimensional bird's-eye view feature map.

[0065] In some embodiments, the above-mentioned reverse projection of the feature pixels on the feature map to obtain the reverse projection result includes: setting a discrete depth range for the feature pixels on the feature map to form a three-dimensional viewing cone for each of the feature maps; weighting the feature vectors of the feature pixels on the feature map with a preset depth probability to obtain a weighted feature vector; placing the weighted feature vector in the corresponding three-dimensional viewing cone to obtain a three-dimensional viewing cone for each of the feature maps based on different image coordinate systems; converting the three-dimensional viewing cone of each of the feature maps based on different image coordinate systems to a unified world coordinate system to obtain a reverse projection of each of the feature maps.

[0066] Specifically, a discrete depth range is set for the feature pixels on the feature map to form a three-dimensional viewing cone for each feature map. In order for the present invention to estimate a more accurate depth probability distribution, the feature vector of the feature pixel is weighted with the depth probability and then placed in the corresponding three-dimensional viewing cone grid. Thereafter, the three-dimensional viewing cone based on different image coordinate systems is converted to a unified world coordinate system.

[0067] In some embodiments, Figure 1 The method shown also includes: calculating the reprojection loss of all points with the same name in the feature map; and optimizing the three-dimensional viewing cone of each feature map using the reprojection loss.

[0068] Calculating the reprojection loss of all the same-name points in the feature map includes: calculating the phase feature maps of all the feature maps using the phase consistency method; matching the phase feature maps between all the feature maps using the normalized cross-correlation method to obtain multiple pairs of matching results, each pair of matching results includes two phase feature maps; for each pair of matching results, selecting a feature point on a phase feature map, and setting a target area with the feature point as the center, and the other phase feature map is the area to be matched; sliding the phase feature map of the target area on the phase feature map of the area to be matched, calculating the normalized cross-correlation coefficient, and the peak value of the correlation coefficient is the same-name point; calculating the reprojection loss of a pair of same-name points located on the two feature maps respectively.

[0069] The existing three-dimensional target detection method optimizes the depth probability distribution through downstream task loss supervision, which actually performs monocular depth estimation for each perspective, which is essentially an ill-conditioned problem. At the same time, the target detection loss needs to be back-propagated to the depth estimation module over a long distance, and there may be problems such as gradient vanishing. In order to solve the problems of poor depth estimation effect and gradient vanishing, the present invention adopts the reprojection error of the same-name points in the feature map, and uses the reprojection error of the same-name points as a depth supervision signal to participate in the optimization of the three-dimensional view cone of the feature map, so as to obtain better depth estimation without consuming additional computing resources.

[0070] In an example, the specific process of the same-name point reprojection error is:

[0071] The phase feature maps of different feature maps of the same scene are calculated using the phase consistency method. The feature maps are matched with each other using the normalized cross-correlation method. Feature points are selected on one phase feature map, and the target area is set with the feature points as the center. The other phase feature map is the area to be matched. The phase feature map of the target area is slid in the phase feature map of the area to be matched, and the normalized cross-correlation coefficient is calculated. The peak of the correlation coefficient is the same-name point.

[0072] For a pair of points p and q with the same name located on two feature maps respectively, the expected depth zp is calculated for p, and then the reprojection point p' of point p on the other feature map is obtained by the homography method. The Euclidean distance between p' and p is calculated as the error, and all matching points are traversed to calculate the reprojection loss of the points with the same name.

[0073] In some embodiments, in operation S140, three-dimensional target detection is performed using the bird's-eye view feature map after the scale feature fusion, and specifically, the feature of each voxel is extracted through a 3D convolutional neural network.

[0074] The network outputs a center point heatmap that indicates the center location of the object. Each voxel is classified as being the center of the object or not. In practice, the center point is usually the bottom center of the object. For each detected center point, it is also necessary to predict the properties of the 3D bounding box associated with it, such as size, orientation, and the exact location of the object. Finally, post-processing steps such as non-maximum suppression (NMS) are used to remove overlapping predictions to ensure that each detected object has only one bounding box.

[0075] Based on the above-mentioned multi-view 3D target detection method based on lightweight remote sensing basic model, the present invention also provides a wavelength routing device. Figure 3 The device is described in detail.

[0076] Figure 3 The structural block diagram of a multi-view three-dimensional target detection device based on a lightweight remote sensing basic model according to an embodiment of the present invention is schematically shown.

[0077] like Figure 3 As shown, the multi-view three-dimensional target detection device 300 based on the lightweight remote sensing basic model of this embodiment includes an input module 310, a generation module 320, a feeding module 330 and a detection module 340.

[0078] The input module 310 is used to input the optical remote sensing images to be detected of the same scene taken from multiple perspectives into the trained lightweight remote sensing basic model to obtain multiple feature maps under the multiple perspectives, and the lightweight remote sensing basic model is used to extract the feature map of the optical remote sensing image. In one embodiment, the input module 310 can be used to perform the operation S110 described above, which will not be repeated here.

[0079] The generating module 320 is used to convert the structure diagram of the optical topology network into an auxiliary diagram. In one embodiment, the generating module 320 can be used to perform the operation S120 described above, which will not be described in detail here.

[0080] The input module 330 is used to input the bird's-eye view feature map into the FPN network to obtain the bird's-eye view feature map after the scale feature is fused. In one embodiment, the input module 330 can be used to perform the operation S130 described above, which will not be repeated here.

[0081] The detection module 340 is used to perform 3D target detection using the bird's-eye view feature map after the scale feature fusion. In one embodiment, the detection module 340 can be used to perform the operation S140 described above, which will not be described in detail here.

[0082] According to an embodiment of the present invention, converting all the feature maps to the same world coordinate system to generate a bird's-eye view feature map according to the feature pixel depth of each feature map includes:

[0083] Perform reverse projection on the feature pixels on the feature map to obtain a reverse projection result;

[0084] A two-dimensional bird's-eye view network is set in a world coordinate system, wherein the two-dimensional bird's-eye view network includes a plurality of two-dimensional bird's-eye view grids;

[0085] Project the reverse projection of each feature map onto the two-dimensional bird's-eye view network to obtain the projection result;

[0086] Average pooling is used on the projection results so that there is only one feature vector in each two-dimensional bird's-eye view grid, and a two-dimensional bird's-eye view feature map is obtained.

[0087] According to an embodiment of the present invention, performing reverse projection on the feature pixels on the feature map to obtain a reverse projection result includes:

[0088] Setting a discrete depth range for feature pixels on the feature map to form a three-dimensional viewing cone for each feature map;

[0089] The feature vector of the feature pixel on the feature map is weighted by a preset depth probability to obtain a weighted feature vector;

[0090] The weighted feature vector is placed in the corresponding three-dimensional viewing cone to obtain a three-dimensional viewing cone of each feature map based on different image coordinate systems;

[0091] The three-dimensional view cone of each feature map based on different image coordinate systems is converted to a unified world coordinate system to obtain the reverse projection of each feature map.

[0092] According to an embodiment of the present invention, the method further comprises:

[0093] Calculate the reprojection loss of all points with the same name in the feature map;

[0094] Using the reprojection loss, the 3D frustum of each feature map is optimized.

[0095] According to an embodiment of the present invention, calculating the reprojection loss of all the points of the same name in the feature map includes:

[0096] Calculate the phase characteristic diagram of all the characteristic diagrams by using the phase consistency method;

[0097] Performing phase feature map matching between all the feature maps by using a normalized cross-correlation method, and obtaining multiple pairs of matching results, each pair of matching results includes two phase feature maps;

[0098] For each pair of matching results, a feature point is selected on one phase feature map, and a target area is set with the feature point as the center, and the other phase feature map is the area to be matched;

[0099] Slide the phase feature map of the target area in the phase feature map of the area to be matched, calculate the normalized mutual correlation coefficient, and the peak value of the correlation coefficient is the same name point;

[0100] Calculate the reprojection loss of a pair of points with the same name located on the two feature maps.

[0101] According to an embodiment of the present invention, the method further comprises:

[0102] Based on multiple sets of optical remote sensing images of the same scene taken from multiple perspectives, a multi-view 3D object detection dataset is constructed;

[0103] Using the multi-view three-dimensional target detection data set, training a lightweight remote sensing basic model to be trained, and obtaining the trained lightweight remote sensing basic model;

[0104] The position and range of the target object are marked in the world coordinate system of the scene as the training truth value.

[0105] According to an embodiment of the present invention, the method of constructing a multi-view 3D object detection dataset based on multiple sets of optical remote sensing images of the same scene taken from multiple viewpoints includes:

[0106] Collect multiple sets of optical remote sensing images of the same scene taken from multiple perspectives;

[0107] For each of the optical remote sensing images, annotate the transformation matrix of the camera coordinate system of the optical remote sensing image relative to the world coordinate system;

[0108] According to the transformation matrix, the multi-view 3D object detection dataset is constructed.

[0109] According to an embodiment of the present invention, any multiple modules of the input module 310, the generation module 320, the input module 330 and the detection module 340 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the input module 310, the generation module 320, the input module 330 and the detection module 340 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the input module 310, the generation module 320, the input module 330 and the detection module 340 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0110] Figure 4 A block diagram of an electronic device suitable for implementing a multi-view three-dimensional target detection method based on a lightweight remote sensing basic model according to an embodiment of the present invention is schematically shown.

[0111] like Figure 4 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0112] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other through bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0113] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0114] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0115] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0116] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0117] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0118] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0119] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0120] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0123] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A multi-view three-dimensional target detection method based on a lightweight remote sensing basic model, characterized in that: include: Inputting optical remote sensing images to be detected of the same scene taken from multiple perspectives into the trained lightweight remote sensing basic model to obtain a plurality of feature maps under the multiple perspectives, wherein the lightweight remote sensing basic model is used to extract the feature map of the optical remote sensing image; According to the feature pixel depth of each feature map, all the feature maps are converted into the same world coordinate system to generate a bird's-eye view feature map; The bird's-eye view feature map is sent to a feature pyramid FPN network to obtain a bird's-eye view feature map after scale feature fusion; Utilizing the bird's-eye view feature map after the scale feature fusion to perform three-dimensional target detection; The method also includes: calculating the reprojection loss of the same-name points in all the feature maps; using the reprojection loss of the same-name points as a deep supervision signal to optimize the three-dimensional cone of each feature map, wherein the three-dimensional cone is formed in the process of generating a bird's-eye view feature map; wherein calculating the reprojection loss of the same-name points in all the feature maps includes: calculating the phase feature maps of all the feature maps using a phase consistency method; matching the phase feature maps between all the feature maps using a normalized cross-correlation method to obtain multiple pairs of matching results, each pair of the matching results including two phase feature maps; for each pair of the matching results, selecting a feature point on one phase feature map, and setting a target area with the feature point as the center, and the other phase feature map is the area to be matched; sliding the phase feature map of the target area in the phase feature map of the area to be matched, and calculating the normalized cross-correlation coefficient, wherein the peak value of the correlation coefficient is the same-name point; and calculating the reprojection loss of a pair of same-name points located on the two feature maps respectively.

2. The multi-view three-dimensional target detection method based on a lightweight remote sensing basic model according to claim 1, characterized in that: The step of converting all the feature maps to the same world coordinate system to generate a bird's-eye view feature map according to the feature pixel depth of each feature map comprises: Performing reverse projection on the feature pixels on the feature map to obtain a reverse projection result; A two-dimensional bird's-eye view network is set in a world coordinate system, wherein the two-dimensional bird's-eye view network includes a plurality of two-dimensional bird's-eye view grids; Project the reverse projection of each feature map onto the two-dimensional bird's-eye view network to obtain the projection result; Average pooling is used on the projection results so that there is only one feature vector in each two-dimensional bird's-eye view grid, and a two-dimensional bird's-eye view feature map is obtained.

3. The multi-view three-dimensional target detection method based on a lightweight remote sensing basic model according to claim 2 is characterized in that: The reverse projection of the feature pixels on the feature map to obtain the reverse projection result includes: Setting a discrete depth range for feature pixels on the feature map to form a three-dimensional viewing cone for each feature map; Weighting the feature vector of the feature pixel on the feature map with a preset depth probability to obtain a weighted feature vector; Placing the weighted feature vector in a corresponding three-dimensional viewing cone to obtain a three-dimensional viewing cone for each feature map in different image coordinate systems; The three-dimensional view cone of each feature map based on different image coordinate systems is converted to a unified world coordinate system to obtain a reverse projection of each feature map.

4. The multi-view three-dimensional target detection method based on a lightweight remote sensing basic model according to claim 1, characterized in that: The method further comprises: Based on multiple sets of optical remote sensing images of the same scene taken from multiple perspectives, a multi-view 3D object detection dataset is constructed; Using the multi-view three-dimensional target detection data set, training the lightweight remote sensing basic model to be trained, and obtaining the trained lightweight remote sensing basic model; Wherein, the position and range of the target object are marked in the world coordinate system where the scene is located as the training truth value.

5. The multi-view three-dimensional target detection method based on a lightweight remote sensing basic model according to claim 4 is characterized in that: The method of constructing a multi-view 3D object detection dataset based on multiple sets of optical remote sensing images of the same scene taken from multiple viewpoints includes: Collect multiple sets of optical remote sensing images of the same scene taken from multiple perspectives; For each of the optical remote sensing images, annotate a transformation matrix of the camera coordinate system of the optical remote sensing image relative to the world coordinate system; According to the transformation matrix, the multi-view three-dimensional object detection data set is constructed.

6. A multi-view three-dimensional target detection device based on a lightweight remote sensing basic model, used to execute the method according to any one of claims 1 to 5, characterized in that: include: An input module is used to input the optical remote sensing images to be detected of the same scene taken from multiple perspectives into the trained lightweight remote sensing basic model to obtain a plurality of feature maps under the multiple perspectives, and the lightweight remote sensing basic model is used to extract the feature map of the optical remote sensing image; A generating module, used for converting all the feature maps into the same world coordinate system to generate a bird's-eye view feature map according to the feature pixel depth of each feature map; A feeding module is used to feed the bird's-eye view feature map into the FPN network to obtain a bird's-eye view feature map after scale feature fusion; The detection module is used to perform three-dimensional target detection using the bird's-eye view feature map after the scale feature is fused.

7. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 5.

Images