Three-dimensional point cloud reconstruction method and system based on ultra-wideband signal multi-view fusion

Abstract

The invention discloses a three-dimensional point cloud reconstruction method and system based on ultra-wideband signal multi-view fusion. A commercial pulse wireless ultra-wideband radar is adopted to collect signals of an object under multi-view; background reflection reduction and effective distance box positioning are carried out on the reflection signals; the radar signal is subjected to data enhancement through virtual antenna offset and phase rotation; designing a signal multi-view depth reconstruction network model, extracting three-dimensional structure information of an object from a radar signal, and generating a multi-view object depth map; and performing inverse projection on the multi-view depth map, converting the depth map into local point cloud, and generating global three-dimensional reconstruction point cloud of the target object through point cloud fusion. The method provided by the invention still has relatively high robustness in a scene in which the object is shielded, and can be widely applied to the fields of logistics sorting, security check and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud reconstruction, and in particular relates to a three-dimensional point cloud reconstruction method and system based on multi-view fusion of ultra-wideband signals. Background Art

[0002] 3D reconstruction technology has achieved significant breakthroughs over the past few decades and has been widely applied in fields such as industrial manufacturing, cultural heritage preservation, and autonomous driving. Its core advantage lies in its ability to efficiently construct 3D geometric models of objects or scenes from 2D images or sensor data through techniques such as multi-view geometric analysis, feature matching, and deep learning, thereby providing high-precision spatial information support for subsequent analysis, simulation, and decision-making. However, while existing methods perform well in open, well-visible scenes, their technical limitations are becoming increasingly prominent in certain complex scenarios. This is particularly true in the areas of logistics sorting and security inspection, where traditional methods face key challenges.

[0003] In logistics sorting scenarios, the performance bottlenecks of 3D reconstruction technology stem from two main aspects: First, the objects to be sorted are often covered by multiple layers of packaging materials (such as cartons and plastic film). As a result, vision-based 3D reconstruction methods cannot effectively obtain the true geometric information of the target objects due to missing surface texture or reflection interference. Second, image-based 3D reconstruction methods focus only on the surface of the object and cannot effectively reconstruct the internal structure of the object. Image-based methods also have a high computational cost. Similar problems are also prominent in the field of security inspection: dangerous goods or contraband may be deliberately wrapped in luggage or cargo. Although traditional X-ray imaging can penetrate the packaging, its two-dimensional projection image lacks three-dimensional spatial information, making it difficult to accurately identify the three-dimensional posture and internal structure of the object. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned existing technologies and provide a three-dimensional point cloud reconstruction method and system based on multi-perspective fusion of ultra-wideband signals. The method can realize three-dimensional point cloud reconstruction of objects at a relatively low cost when the objects are obscured by external packaging, and provide information support for downstream tasks such as logistics sorting and security inspection. The method is used to solve the technical problem that the existing three-dimensional reconstruction technology based on computer vision cannot be used when objects may be obscured by packaging in logistics sorting and security inspection.

[0005] The present invention adopts the following technical solutions: A 3D point cloud reconstruction method based on multi-view fusion of ultra-wideband signals includes the following steps: A single-transmitter, single-receiver commercial pulse ultra-wideband radar is used to collect multi-perspective reflection signals of moving target objects and construct a virtual antenna array. The obtained multi-view reflection signals in motion are subjected to background reflection subtraction and effective range box positioning to extract the denoised target signals. Based on the virtual antenna array, data enhancement is performed on the denoised target signal, including virtual antenna array offset and phase rotation operations, to generate an enhanced radar signal dataset; Build a signal multi-view depth reconstruction network model, input the enhanced radar signal dataset, and output a multi-view depth map and a mask of the pixels occupied by the object in the depth map; The multi-view depth map obtained in the first step is inversely projected, and the mask of the depth map pixels occupied by the object is combined to generate the local 3D reconstruction point cloud of the target object under multiple viewpoints; then the global 3D reconstruction point cloud of the object is obtained by point cloud fusion.

[0006] Preferably, the multi-view reflected signals of the target object in motion are collected and a virtual antenna array is constructed, specifically as follows: A virtual antenna array is generated by the uniform motion of an object relative to the radar. The signal data collected by the ultra-wideband radar is a two-dimensional complex matrix, with the two dimensions corresponding to the virtual antenna and the range bin. To obtain the three-dimensional features of the target object from four different perspectives, the radar signals from the four perspectives of the object are collected.

[0007] Preferably, background reflection reduction and effective range box positioning are as follows: Collect the background reference signal when there is no target object; linearly subtract the background reference signal from the original signal to eliminate environmental reflections and antenna crosstalk interference; An effective range bin positioning algorithm based on the sliding window idea is used to locate the nine adjacent range bins with the largest reflected signal amplitude. These bins are used as effective range bins. The initial radar signal matrix is ​​cropped along the range bin dimension, and only the effective range bins are retained.

[0008] Preferably, the virtual antenna array offset and phase rotation operations are specifically: The virtual antenna array offset simulates the horizontal displacement of the object by randomly clipping the virtual antenna dimension of the radar signal matrix; Phase rotation simulates the vertical displacement of an object by adding a global phase offset to the complex signal.

[0009] Preferably, the signal multi-view depth reconstruction network model includes: Multiple weight-sharing complex signal encoder modules are used to extract three-dimensional features of objects from the complex signal matrix; The decoder module is used to map the spliced ​​features into depth maps and object masks under multiple views; A joint loss function based on the L1 loss of the predicted depth value and the binary cross entropy loss of the predicted object mask.

[0010] Preferably, the complex signal encoder module comprises: Two-way parallel multi-scale one-dimensional convolutional blocks with kernel sizes of 3 / 5 / 7 / 9 and padding of 1 / 2 / 3 / 4 to maintain output size; five-level downsampling structure, each level including MaxPool1D and channel multiplication; output features are compressed to 128 dimensions through three layers of full connection, and the dimension after splicing the four-view features is 512.

[0011] Preferably, the decoder module comprises: Fully connected layer and feature reorganization module: The concatenated global feature vector is upgraded to 4096 dimensions through three fully connected layers and reorganized into a 256×4×4 two-dimensional feature map; Multi-level upsampling module: This module gradually expands the feature map size to 128×128 through bilinear interpolation and two-dimensional convolution operations. It outputs a final feature map with 2N channels, where the first N channels are multi-view depth map predictions and the last N channels are object mask predictions. It also optimizes depth regression by using a preset depth value bias. The final decoder outputs the depth map prediction under N perspectives And the mask of the pixels occupied by the object in the depth map under N perspectives .

[0012] Preferably, the joint loss function for:

[0013] in, is the binary cross entropy loss function with Logits The weight of is the L1 loss function The weight of .

[0014] Preferably, the depth map inverse projection and point cloud fusion process includes: Reverse-project each pixel in the depth map at each viewpoint into the camera's 3D coordinate system to obtain the 3D point cloud corresponding to the depth map at that viewpoint in the camera coordinate system. Convert the 3D point clouds in multiple camera coordinate systems to the world coordinate system. Obtain a local point cloud in the world coordinate system for each viewpoint through coordinate conversion. Merge the local point clouds from all viewpoints to obtain the final global point cloud. Homogeneous three-dimensional coordinates in world coordinates for:

[0015] in, is the three-dimensional homogeneous coordinate of the depth map in the camera coordinate system under the corresponding viewing angle, , , are the corresponding coordinate values ​​on the three coordinate axes, For the The homogeneous transformation matrix from the world coordinate system to the camera coordinate system under each perspective, is the rotation matrix calculated by rotating the quaternion in the camera external parameters, is the translation vector in the camera extrinsic parameters.

[0016] In a second aspect, an embodiment of the present invention provides a three-dimensional point cloud reconstruction system based on multi-view fusion of ultra-wideband signals, comprising: The signal module uses a single-transmitter, single-receiver commercial pulse ultra-wideband radar to collect multi-perspective reflection signals of the target object in motion and construct a virtual antenna array; The denoising module performs background reflection subtraction and effective range box positioning on the multi-view reflection signals obtained in the motion state, and extracts the denoised target signal; The enhancement module performs data enhancement on the denoised target signal based on the virtual antenna array, including virtual antenna array offset and phase rotation operations to generate an enhanced radar signal dataset; The network module builds a signal multi-view depth reconstruction network model, inputs the enhanced radar signal dataset, and outputs a multi-view depth map and a mask of the pixels occupied by the object in the depth map; The reconstruction module performs inverse projection and point cloud fusion on the obtained multi-view depth map, and generates a 3D reconstructed point cloud of the target object by combining the mask of the depth map pixels occupied by the object.

[0017] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned three-dimensional point cloud reconstruction method based on multi-perspective fusion of ultra-wideband signals are implemented.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned three-dimensional point cloud reconstruction method based on multi-perspective fusion of ultra-wideband signals.

[0019] In a fifth aspect, a chip comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned three-dimensional point cloud reconstruction method based on multi-perspective fusion of ultra-wideband signals are implemented.

[0020] In a sixth aspect, an embodiment of the present invention provides an electronic device, comprising a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned three-dimensional point cloud reconstruction method based on multi-perspective fusion of ultra-wideband signals.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: A three-dimensional point cloud reconstruction method based on multi-perspective fusion of ultra-wideband signals. This method uses a single-transmitter, single-receiver IR-UWB radar combined with target motion to construct a virtual antenna array, replacing traditional multi-antenna systems and significantly reducing hardware cost and complexity. Background subtraction and data enhancement are used to improve signal quality and generate a multi-perspective radar dataset. A multi-perspective signal deep reconstruction network is designed to directly extract the target's three-dimensional features from the radar signal, output a depth map and mask, and combine inverse projection with a point cloud fusion algorithm to reconstruct a high-precision three-dimensional structure. This invention overcomes the limitations of traditional vision and radio frequency technologies, providing a lightweight, highly robust solution for three-dimensional perception in complex scenes.

[0022] Furthermore, a single-transmitter, single-receiver ultra-wideband radar is used to replace traditional multi-antenna RF systems (such as 3-transmitter, 4-receiver millimeter-wave radars), reducing hardware costs. A virtual antenna array is constructed through the object's own movement, eliminating the need for mechanical slides or synthetic aperture radar (SAR) scanning devices, significantly simplifying the deployment process.

[0023] Furthermore, the signal processing module removes environmental interference signals (such as antenna crosstalk and static object reflections) through background reflection reduction and effective range box positioning technology, ensuring that effective features can still be stably extracted in complex environments.

[0024] Furthermore, a virtual antenna offset and phase rotation data enhancement algorithm was designed. By randomly cropping the slow time dimension and global phase offset of the signal matrix, the impact of small displacements of objects on radar signals was effectively simulated, effectively improving the diversity of signal samples in the dataset.

[0025] Furthermore, a signal multi-view deep reconstruction network (SMDRNet) is proposed. Through the multi-scale one-dimensional convolution feature extraction of the complex signal encoder and the bilinear interpolation upsampling technology of the decoder, it effectively extracts the three-dimensional structural features of the object from the multi-view radar complex signal and outputs the depth information of the object from multiple viewpoints.

[0026] Furthermore, a global three-dimensional point cloud is generated through multi-view depth map inverse projection and point cloud fusion technology, combined with mask filtering and coordinate system alignment. Compared with directly generating point cloud data through neural networks, this method avoids problems such as uneven point cloud distribution, poor surface continuity, and difficulty in ensuring topological structure.

[0027] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0028] In summary, the three-dimensional point cloud reconstruction method of the present invention comprehensively considers hardware cost and occlusion robustness, and can be widely used in logistics, security inspection and other fields.

[0029] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 Build a schematic diagram for the IR-UWB radar virtual antenna; Figure 2 This is a schematic diagram of the two-dimensional matrix of radar receiving signals; Figure 3 This is the SMDRNet network model framework diagram; Figure 4 This is the structure diagram of the complex signal encoder of the SMDRNet network model; Figure 5 This is the structure diagram of the SMDRNet network model decoder; Figure 6 Schematic diagram of the experimental scene layout; Figure 7 Reconstruct the effect of the object's 3D point cloud; Figure 8 A schematic diagram of a computer device provided in accordance with an embodiment of the present invention; Figure 9 A block diagram of an electronic device provided according to an embodiment of the present invention; Figure 10 It is a schematic diagram of the process of the present invention.

[0032] Among them, 60. Computer device; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / Utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0034] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0036] It should be further understood that the term "and / or" as used in the present specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0037] It should be understood that although the terms "first," "second," and "third" may be used to describe preset ranges in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are merely used to distinguish one preset range from another. For example, without departing from the scope of embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0038] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0039] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0040] The present invention provides a three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals. The method uses a commercial impulse radio ultra-wideband (IR-UWB) radar to collect signals from objects at multiple viewpoints, performs background reflection reduction and effective range box positioning on the reflected signals, and enhances the radar signal data through virtual antenna offset and phase rotation. Then, a Signal Multi-view Depth Reconstruction Network (SMDRNet) model is designed to extract the three-dimensional structural information of the object from the radar signal and generate a depth map of the object from multiple viewpoints. The multi-view depth map is then inversely projected and point cloud fused to generate a three-dimensional reconstructed point cloud of the target object. The method can be widely used in logistics sorting, security inspection and other fields.

[0041] Example 1 See also Figure 10 The present invention provides a three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals, comprising the following steps: S1. Use a single-transmitter, single-receiver commercial pulse ultra-wideband (IR-UWB) radar to collect multi-view reflected signals from moving target objects and construct a virtual antenna array. See also Figure 1 , the object moves in the horizontal direction, and the ultra-wideband radar is placed on the side of the object. By equating the movement of the object to the movement of the radar, an equivalent virtual antenna array is constructed; See also Figure 2 ,The received signal of the radar is a two-dimensional matrix, where the two dimensions are the range bin (fast time index) and the virtual antenna index (slow time index).

[0042] S2. Collect a background reference signal when there is no target object; linearly subtract the background reference signal from the original signal to eliminate environmental reflection and antenna crosstalk interference.

[0043] An effective range bin positioning algorithm based on the sliding window idea is used to locate the nine adjacent range bins with the largest reflected signal amplitude. These bins are used as effective range bins. The initial radar signal matrix is ​​cropped along the range bin dimension, and only the effective range bins are retained.

[0044] S3. Simulate the horizontal displacement of the object by performing a virtual antenna offset on the initial signal data, and simulate the vertical displacement of the object by performing a phase rotation on the complex signal, thereby generating an enhanced radar signal data set; S4. Construct a signal multi-view depth reconstruction network model (SMDRNet) that inputs a multi-view radar signal and outputs a multi-view depth map and a mask of the pixels occupied by objects in the depth map; the network model includes a complex signal encoder module, a decoder module, and a joint loss function based on an L1 loss of the predicted depth value and a binary cross entropy loss of the predicted object mask; See also Figure 3 ,The overall structure of SMDRNet consists of a complex signal encoder module and a decoder module ,which share weights under multiple signal perspectives; See also Figure 4 ,The complex encoder module consists of a dual-parallel multi-scale one-dimensional convolution block, a five-level downsampling structure, and three fully connected layers, and finally outputs a 128-dimensional feature vector; See also Figure 5 The decoder module consists of a fully connected layer and a multi-level upsampling module, and finally outputs the depth map prediction under N viewing angles. And the mask of the pixels occupied by the object in the depth map under N perspectives ; The loss function used by the SMDRNet network model is a two-part joint loss function: For the predicted depth map, only valid pixels are constrained, and the valid pixels are filtered out using the binary mask. The binary mask formula is:

[0045] The L1 loss function is then used to constrain the depth values ​​of valid pixels:

[0046] in, is the true value of the depth value, is the depth map height, is the depth map width.

[0047] For the object mask corresponding to the depth map, a binary cross entropy loss function with Logits is used for constraint:

[0048] The final loss function is the joint loss of the two loss functions:

[0049] S5. Inversely project each pixel in the depth map at each viewing angle into the camera's three-dimensional coordinate system to obtain the three-dimensional point cloud corresponding to the depth map at that viewing angle in the camera's coordinate system:

[0050] in, is the homogeneous pixel coordinate of the pixel in the depth map, is the homogeneous three-dimensional coordinate corresponding to the pixel point in the camera coordinate system, is the homogeneous extension of the orthogonal camera intrinsic parameter matrix.

[0051] In order to fuse 3D point clouds from multiple perspectives, it is necessary to uniformly transform the 3D point clouds from multiple camera coordinate systems into the world coordinate system:

[0052] in, is the homogeneous three-dimensional coordinate in world coordinates, For the The homogeneous transformation matrix from the world coordinate system to the camera coordinate system under the perspective is composed of the rotation quaternion in the camera external parameter and translation vectors Through coordinate transformation, the world coordinate local point cloud under each perspective is obtained, and the local point clouds under all perspectives are merged together to obtain the final global point cloud.

[0053] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Accordingly, various aspects of the present invention may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuits," "modules," or "platforms."

[0054] Example 2 The present invention provides a three-dimensional point cloud reconstruction system based on multi-perspective fusion of ultra-wideband signals. The system can be used to implement the above-mentioned three-dimensional point cloud reconstruction method based on multi-perspective fusion of ultra-wideband signals. Specifically, the three-dimensional point cloud reconstruction system based on multi-perspective fusion of ultra-wideband signals includes a signal module, a denoising module, an enhancement module, a network module and a reconstruction module.

[0055] The signal module uses a single-transmitter, single-receiver commercial pulse ultra-wideband radar to collect multi-perspective reflection signals of the target object in motion and construct a virtual antenna array. The denoising module performs background reflection subtraction and effective range box positioning on the multi-view reflection signals obtained in the motion state, and extracts the denoised target signal; The enhancement module performs data enhancement on the denoised target signal based on the virtual antenna array, including virtual antenna array offset and phase rotation operations to generate an enhanced radar signal dataset; The network module builds a signal multi-view depth reconstruction network model, inputs the enhanced radar signal dataset, and outputs a multi-view depth map and a mask of the pixels occupied by the object in the depth map; The reconstruction module performs inverse projection and point cloud fusion on the obtained multi-view depth map, and generates a 3D reconstructed point cloud of the target object by combining the mask of the depth map pixels occupied by the object.

[0056] Example 3 The present invention provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of a three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals, including: A single-transmitter, single-receiver commercial pulse ultra-wideband radar is used to collect multi-perspective reflection signals of the target object in motion, and a virtual antenna array is constructed. The background reflection subtraction and effective range box positioning of the obtained multi-perspective reflection signals in motion are performed to extract the denoised target signal. Based on the virtual antenna array, the denoised target signal is data enhanced, including virtual antenna array offset and phase rotation operations, to generate an enhanced radar signal dataset. A signal multi-perspective depth reconstruction network model is constructed, which inputs the enhanced radar signal dataset and outputs a multi-perspective depth map and a mask of the pixels occupied by the object in the depth map. The obtained multi-perspective depth map is subjected to inverse projection and point cloud fusion, and the mask of the pixels occupied by the object in the depth map is combined to generate a three-dimensional reconstructed point cloud of the target object.

[0057] See also Figure 8 The terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable by the processor 61. When the computer program 63 is executed by the processor 61, the method for reconstructing a 3D point cloud based on multi-view fusion of ultra-wideband signals in the embodiment is implemented. To avoid repetition, the details are not described here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the system for reconstructing a 3D point cloud based on multi-view fusion of ultra-wideband signals in the embodiment are implemented. To avoid repetition, the details are not described here.

[0058] The computer device 60 may be a desktop computer, a notebook computer, a PDA, a cloud server, or other computing devices. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. It will be understood by those skilled in the art that Figure 10 This is merely an example of the computer device 60 and does not constitute a limitation of the computer device 60 . The computer device 60 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0059] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0060] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 60.

[0061] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is about to be output.

[0062] See also Figure 9 The terminal device is an electronic device 600, which is implemented as a general-purpose computing device. The components of the electronic device may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0063] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 10 Follow the steps shown in .

[0064] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0065] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0066] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0067] The electronic device 600 may also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem). Such communication may occur via an input / output interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network, a wide area network, and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0068] Example 4 The present invention also provides a storage medium, specifically a computer-readable storage medium. The computer-readable storage medium is a memory device in a terminal device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. It may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that more specific examples of the computer-readable storage medium herein include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0069] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program code. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the above.

[0070] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network or a wide area network, or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0071] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals in the above embodiment; one or more instructions in the computer-readable storage medium are loaded by the processor and executed as follows: using a single-transmitter, single-receiver commercial pulse ultra-wideband radar to collect multi-view reflection signals of a target object in motion and construct a virtual antenna array; performing background reflection subtraction and effective range bin positioning on the obtained multi-view reflection signals in motion to extract a denoised target signal; based on the virtual antenna array, performing data enhancement on the denoised target signal, including virtual antenna array offset and phase rotation operations, to generate an enhanced radar signal data set; constructing a signal multi-view depth reconstruction network model, inputting the enhanced radar signal data set, and outputting a multi-view depth map and a mask of the pixels of the depth map occupied by the object; performing inverse projection and point cloud fusion on the obtained multi-view depth map, and combining the mask of the pixels of the depth map occupied by the object to generate a three-dimensional reconstructed point cloud of the target object.

[0072] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] The three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals in this invention was tested in the laboratory: See also Figure 6 The figure shows the experimental scenes when the object is not blocked and when the object is blocked. The object is placed on the loading platform on the slide and moves with the slide, simulating the movement of the object on the conveyor belt in the field of logistics or security inspection. The radar is placed on the side of the slide.

[0075] See also Figure 7 ,The figure shows the reconstructed 3D point cloud of some objects and the corresponding ,real 3D point cloud.

[0076] Experimental results show that when the object is not obstructed, the chamfer distance (CD) error of the reconstructed 3D point cloud of the object is 4.837cm. When the object is obstructed, the CD error of the reconstructed 3D point cloud of the object is 5.814cm, which is only 0.997cm higher than that of the non-obstructed scene. This shows that the present invention can achieve highly robust 3D point cloud reconstruction of objects under object occlusion. In summary, the present invention provides a three-dimensional point cloud reconstruction method and system based on multi-perspective fusion of ultra-wideband signals, which uses a commercial pulse wireless ultra-wideband (IR-UWB) radar to achieve three-dimensional point cloud reconstruction of objects. The method of the present invention still has good performance when the object is completely occluded, achieving high robustness in occlusion scenarios. Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0077] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0078] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0079] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0081] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0082] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0086] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals, characterized in that: The following steps are involved: A single-transmitter, single-receiver commercial pulse ultra-wideband radar is used to collect multi-perspective reflection signals of moving target objects and construct a virtual antenna array. The obtained multi-view reflection signals in motion are subjected to background reflection subtraction and effective range box positioning to extract the denoised target signals. Based on the virtual antenna array, data enhancement is performed on the denoised target signal, including virtual antenna array offset and phase rotation operations, to generate an enhanced radar signal dataset; Build a signal multi-view depth reconstruction network model, input the enhanced radar signal dataset, and output a multi-view depth map and a mask of the pixels occupied by the object in the depth map; The multi-view depth map obtained in the first step is inversely projected, and the mask of the depth map pixels occupied by the object is combined to generate the local 3D reconstruction point cloud of the target object under multiple viewpoints; then the global 3D reconstruction point cloud of the object is obtained by point cloud fusion.

2. The three-dimensional point cloud reconstruction method based on multi-view fusion of ultra-wideband signals according to claim 1 is characterized in that: Collect multi-view reflected signals of the target object in motion and construct a virtual antenna array, specifically: A virtual antenna array is generated by the uniform motion of an object relative to the radar. The signal data collected by the ultra-wideband radar is a two-dimensional complex matrix, with the two dimensions corresponding to the virtual antenna and the range bin. To obtain the three-dimensional features of the target object from four different perspectives, the radar signals from the four perspectives of the object are collected.

3. The three-dimensional point cloud reconstruction method based on ultra-wideband signal multi-view fusion according to claim 1 is characterized in that: Background reflection reduction and effective range box positioning are as follows: Collect the background reference signal when there is no target object; linearly subtract the background reference signal from the original signal to eliminate environmental reflections and antenna crosstalk interference; An effective range bin positioning algorithm based on the sliding window idea is used to locate the nine adjacent range bins with the largest reflected signal amplitude. These bins are used as effective range bins. The initial radar signal matrix is ​​cropped along the range bin dimension, and only the effective range bins are retained.

4. The 3D point cloud reconstruction method based on multi-view fusion of ultra-wideband signals according to claim 1, characterized in that: The virtual antenna array offset and phase rotation operations are specifically: The virtual antenna array offset simulates the horizontal displacement of the object by randomly clipping the virtual antenna dimensions of the radar signal matrix; Phase rotation simulates the vertical displacement of an object by adding a global phase offset to the complex signal.

5. The 3D point cloud reconstruction method based on multi-view fusion of ultra-wideband signals according to claim 1, characterized in that: The signal multi-view deep reconstruction network model includes: Multiple weight-sharing complex signal encoder modules are used to extract three-dimensional features of objects from the complex signal matrix; The decoder module is used to map the spliced ​​features into depth maps and object masks under multiple views; A joint loss function based on the L1 loss of the predicted depth value and the binary cross entropy loss of the predicted object mask.

6. The three-dimensional point cloud reconstruction method based on ultra-wideband signal multi-view fusion according to claim 5, characterized in that: The complex signal encoder module includes: Two-way parallel multi-scale one-dimensional convolutional blocks with kernel sizes of 3 / 5 / 7 / 9 and padding of 1 / 2 / 3 / 4 to maintain output size; five-level downsampling structure, each level including MaxPool1D and channel multiplication; output features are compressed to 128 dimensions through three layers of full connection, and the dimension after splicing the four-view features is 512.

7. The three-dimensional point cloud reconstruction method based on ultra-wideband signal multi-view fusion according to claim 5, characterized in that: The decoder module includes: Fully connected layer and feature reorganization module: The concatenated global feature vector is upgraded to 4096 dimensions through three fully connected layers and reorganized into a 256×4×4 two-dimensional feature map; Multi-level upsampling module: This module gradually expands the feature map size to 128×128 through bilinear interpolation and two-dimensional convolution operations. It outputs a final feature map with 2N channels, where the first N channels are multi-view depth map predictions and the last N channels are object mask predictions. It also optimizes depth regression by using a preset depth value bias. The final decoder outputs the depth map prediction under N perspectives And the mask of the pixels occupied by the object in the depth map under N perspectives .

8. The three-dimensional point cloud reconstruction method based on ultra-wideband signal multi-view fusion according to claim 5, characterized in that: Joint loss function for: in, is the binary cross entropy loss function with Logits The weight of is the L1 loss function The weight of .

9. The three-dimensional point cloud reconstruction method based on ultra-wideband signal multi-view fusion according to claim 1, characterized in that: The depth map inverse projection and point cloud fusion process includes: Reverse-project each pixel in the depth map at each viewpoint into the camera's 3D coordinate system to obtain the 3D point cloud corresponding to the depth map at that viewpoint in the camera coordinate system. Convert the 3D point clouds in multiple camera coordinate systems to the world coordinate system. Obtain a local point cloud in the world coordinate system for each viewpoint through coordinate conversion. Merge the local point clouds from all viewpoints to obtain the final global point cloud. Homogeneous three-dimensional coordinates in world coordinates for: in, is the three-dimensional homogeneous coordinate of the depth map in the camera coordinate system under the corresponding viewing angle, , , are the corresponding coordinate values ​​on the three coordinate axes, For the The homogeneous transformation matrix from the world coordinate system to the camera coordinate system under each perspective, is the rotation matrix calculated by rotating the quaternion in the camera external parameters, is the translation vector in the camera extrinsic parameters.

10. A 3D point cloud reconstruction system based on multi-view fusion of ultra-wideband signals, characterized in that: include: The signal module uses a single-transmitter, single-receiver commercial pulse ultra-wideband radar to collect multi-perspective reflection signals of the target object in motion and construct a virtual antenna array; The denoising module performs background reflection subtraction and effective range box positioning on the multi-view reflection signals obtained in the motion state, and extracts the denoised target signal; The enhancement module performs data enhancement on the denoised target signal based on the virtual antenna array, including virtual antenna array offset and phase rotation operations to generate an enhanced radar signal dataset; The network module builds a signal multi-view depth reconstruction network model, inputs the enhanced radar signal dataset, and outputs a multi-view depth map and a mask of the pixels occupied by the object in the depth map; The reconstruction module performs inverse projection and point cloud fusion on the obtained multi-view depth map, and generates a 3D reconstructed point cloud of the target object by combining the mask of the depth map pixels occupied by the object.

Images