A three-dimensional reconstruction method and related apparatus

Abstract

A three-dimensional reconstruction method can be applied to the field of artificial intelligence, and the method comprises: acquiring a first feature representation and a second feature representation, the first feature representation being a feature representation corresponding to a pixel image collected by an RGB camera, and the second feature representation being a feature representation corresponding to event data collected by an event camera; the pixel image and the event data are collected for the same scene; the first feature representation and the second feature representation are fused to obtain a third feature representation; and a three-dimensional reconstruction network is used to perform three-dimensional reconstruction on the scene according to the third feature representation to obtain a three-dimensional reconstruction result. The three-dimensional reconstruction result with high time resolution, low noise and high spatial resolution can be obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a three-dimensional reconstruction method and related apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Structured light imaging, as a crucial component of 3D reconstruction, reconstructs the 3D structure of objects through actively projected coded patterns. It boasts advantages such as simplicity, non-contact operation, high robustness, and low loss. Compared to passive methods like binocular 3D reconstruction, structured light imaging offers superior imaging performance and higher accuracy in extreme scenarios such as darkness and texture loss. Traditional structured light imaging achieves high-resolution, high-quality 3D reconstruction using a stereo system comprised of a conventional active pixel sensor (APS) camera and an active light source (laser emitter, digital projector).

[0004] However, as the requirements for imaging quality and speed in structured light imaging become increasingly demanding, the limitations of traditional cameras in terms of resolution and frame rate prevent the achievement of fast, high-quality 3D imaging. Furthermore, active projection is susceptible to ambient light interference, and the color and reflection of object surfaces also affect the decoding of the projected pattern, reducing the accuracy of active structured light imaging.

[0005] Event cameras (dynamic vision sensors, DVS), as a visual sensor that has attracted widespread attention in recent years, borrow from bionics in their imaging method, driving imaging through changes in light intensity. They possess advantages such as low latency, high dynamic range, low redundancy, and independent pixel excitation, making them ideal sensors for structured light imaging. Firstly, their low latency allows for extremely high data acquisition bandwidth, effectively compensating for the insufficient frame rate of traditional cameras. Secondly, their high dynamic range (120dB) enables the sensor to still acquire valid information even under overexposure conditions. However, due to the independent excitation of camera pixels, they are susceptible to significant noise from changes in natural lighting, severely reducing the accuracy of 3D reconstruction. Furthermore, due to the trade-off between low latency and high resolution, existing event cameras have relatively low spatial resolution, failing to achieve high-resolution 3D reconstruction. Summary of the Invention

[0006] This application provides a three-dimensional reconstruction method that can obtain three-dimensional reconstruction results with high temporal resolution, low noise, and high spatial resolution.

[0007] In a first aspect, embodiments of this application provide a three-dimensional reconstruction method, the method comprising: obtaining a first feature representation and a second feature representation, wherein the first feature representation is a feature representation corresponding to a pixel image acquired by an RGB camera, and the second feature representation is a feature representation corresponding to event data acquired by an event camera; wherein the pixel image and the event data are acquired for the same scene; performing feature fusion on the first feature representation and the second feature representation to obtain a third feature representation; and performing three-dimensional reconstruction of the scene using a three-dimensional reconstruction network based on the third feature representation to obtain a three-dimensional reconstruction result.

[0008] In this embodiment, the feature representations of pixel images and event data are fused. Since the event data acquired by the event camera has a high temporal resolution and the image frames acquired by the RGB camera have low noise and high spatial resolution, the reconstruction result obtained by 3D reconstruction based on the fused feature representation can have high temporal resolution, low noise and high spatial resolution.

[0009] To ensure the accuracy of the fusion object, it is necessary to ensure that the pixel image and event data are matched in both time and space. That is, the pixel image and event data corresponding to feature fusion can be obtained by the RGB camera and the event camera respectively from the same scene, and the event data is obtained from multiple frames of event data collected by the event camera within the target time period, and the pixel image is obtained from the image collected by the RGB camera within the target time period.

[0010] In other words, the exposure time of the pixel image is within the target time period, specifically a sub-time period within the target time period. Within the target time period, the event camera can acquire multiple frames of event data. For example, phase calculation can be performed on the multiple frames of event data to obtain the event data.

[0011] In one possible implementation, the first feature representation and the second feature representation have the same size in the spatial dimension, and the features are fused into a splicing operation in the channel dimension.

[0012] In one possible implementation, obtaining the second feature representation includes: obtaining target information through a neural network based on the third feature representation, wherein the third feature representation is the feature representation corresponding to the pixel image, and the target information represents the weight of noise contained in the location point in the fourth feature representation, wherein the fourth feature representation is the feature representation corresponding to the event data; and obtaining the second feature representation based on the fourth feature representation and the target information.

[0013] In one possible implementation, since the pixel images captured by the camera have lower noise than the event data captured by the event camera, in order to achieve the fusion of the two modalities, a neural network can be used to extract target information from the feature representation of the pixel images. This target information can represent the weight of the noise contained in each location point of the feature representation. The target information and the fourth feature representation are then fused to achieve effective suppression of event noise.

[0014] In one possible implementation, the neural network is a first attention network. Based on a third feature representation, target information is obtained through the first attention network, and the target information includes first attention information. The first attention information is the spatial correlation information of the pixel image. The third feature representation can be the shallow feature representation corresponding to the pixel image described in the above embodiments, or the feature representation corresponding to the pixel image obtained through a certain attention module. The first attention network can belong to the next adjacent attention module of that attention module. The spatial correlation information of the pixel image can represent the weight ratio of noise.

[0015] Due to the unique nature of event stream data and the resolution difference between event cameras and traditional cameras, a self-attention mechanism was added to the network to achieve modality fusion, and a high-resolution reconstruction module was used to obtain fast and high-quality results.

[0016] In one possible implementation, a second attention network can be used to process the fourth feature representation to obtain second attention information. The second attention information is the spatial correlation information of the event data. The third feature representation can be the shallow feature representation corresponding to the event data described in the above embodiments, or the feature representation corresponding to the event data obtained through a certain attention module. The second attention network can belong to the next adjacent attention module of the attention module. The second attention information is used to perform attention fusion with the first attention information to obtain third attention information. The third attention information is used to obtain the second feature representation.

[0017] In one possible implementation, the attention fusion is a summation operation at corresponding positions.

[0018] In one possible implementation, the first attention information, the second attention information, and the third attention information can be used to represent the correlation between the Q vector and the K vector; the third attention information can be used to fuse with the K vector obtained by processing the fourth feature representation through the second attention network to obtain the first feature representation.

[0019] Compared to multi-layer convolutions for receptive field expansion, global relevance calculation of feature maps can be performed using two-dimensional matrix multiplication to obtain the global receptive field. However, this method only calculates global relevance weights, losing local relevance at specific pixel locations, and cannot effectively guarantee the unity of local and global self-attention mechanisms. To obtain a local receptive field, a corresponding feedforward network (FFN) can be added. Its basic components can include an input layer, hidden layers, activation layers, and an output layer. The shallow network in the hidden layers can realize the calculation of local relevance.

[0020] In one possible implementation, the first feature representation can also be processed by a feedforward neural network (FFN) to obtain a processed first feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the processed first feature representation and the second feature representation; or, the second feature representation can be processed by a feedforward neural network (FFN) to obtain a processed second feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the first feature representation and the processed second feature representation.

[0021] In one possible implementation, multiple cascaded attention modules can be used to process the feature representations corresponding to the pixel image and the event data to obtain the first feature representation and the second feature representation. The output of each attention module serves as the input to the next adjacent attention module for attention-based processing. In other words, each attention module can perform attention-based processing based on the feature representations of the pixel image and the event data output by the previous connected attention module (the first attention module obtains a shallow feature representation of the pixel image and a shallow feature representation of the event data, while the last attention module obtains the first and second feature representations).

[0022] In one possible implementation, the output of each attention module includes a feature representation corresponding to the pixel image and a feature representation corresponding to the event data; the first feature representation is obtained by fusing the feature representations corresponding to the pixel image obtained by some or all of the attention modules; or, the second feature representation is obtained by fusing the feature representations corresponding to the event data obtained by some or all of the attention modules.

[0023] In one possible implementation, the resolution of the 3D reconstruction result is consistent with the resolution of the pixel image.

[0024] In one possible implementation, the 3D reconstruction result includes multiple location points and depth information corresponding to each location point.

[0025] In one possible implementation, the method further includes:

[0026] Phase calculation is performed on the event data collected by the event camera to obtain the event phase;

[0027] The second feature is represented as the feature representation corresponding to the event phase.

[0028] In one possible implementation, median filtering can be applied to the event phase. Specifically, the event phase includes multiple phase values ​​corresponding to the target location point, where different phase values ​​correspond to different timestamps. Median filtering can be applied to the event phase to obtain a processed event phase. The second feature representation is the feature representation corresponding to the processed event phase. The processed event phase includes the target phase value corresponding to the target location point, which is obtained by median filtering of the multiple phase values ​​corresponding to the target location point. Due to the characteristics of the event camera imaging principle, effective data is accompanied by a large amount of random noise. Generally, random fluctuations in event points can be effectively removed by simple filtering, specifically by applying median filtering to the timestamps of all event points within the same pixel within a period.

[0029] Secondly, this application provides a three-dimensional reconstruction apparatus, the apparatus comprising:

[0030] The acquisition module is used to acquire a first feature representation and a second feature representation, wherein the first feature representation is the feature representation corresponding to the pixel image acquired by the RGB camera, and the second feature representation is the feature representation corresponding to the event data acquired by the event camera; the pixel image and the event data are acquired for the same scene;

[0031] The fusion module is used to fuse the first feature representation and the second feature representation to obtain a third feature representation;

[0032] The 3D reconstruction module is used to perform 3D reconstruction of the scene based on the third feature representation through a 3D reconstruction network to obtain the 3D reconstruction result.

[0033] In one possible implementation, the event data is obtained from multiple frames of event data collected by the event camera within a target time period, and the pixel image is obtained from an image collected by the RGB camera within the target time period.

[0034] In one possible implementation, the first feature representation and the second feature representation have the same size in the spatial dimension, and the features are fused into a splicing operation in the channel dimension.

[0035] In one possible implementation, the acquisition module is specifically used for:

[0036] Based on the third feature representation, target information is obtained through a neural network. The third feature representation is the feature representation corresponding to the pixel image. The target information represents the weight of noise contained in the position point in the fourth feature representation. The fourth feature representation is the feature representation corresponding to the event data.

[0037] The second feature representation is obtained based on the fourth feature representation and the target information.

[0038] In one possible implementation, the neural network is a first attention network, and the acquisition module is specifically used for:

[0039] Based on the third feature representation, target information is obtained through a first attention network, wherein the target information includes first attention information; the first attention information is the spatial correlation information of the pixel image.

[0040] The fourth feature representation is processed through a second attention network to obtain second attention information. This second attention information represents the spatial correlation of the event data. It is then fused with the first attention information to obtain third attention information.

[0041] The third attention information is used to obtain the second feature representation.

[0042] In one possible implementation, the attention fusion is a summation operation at corresponding positions.

[0043] In one possible implementation, the first attention information, the second attention information, and the third attention information are used to represent the correlation between the Q vector and the K vector;

[0044] The third attention information is used to obtain the first feature representation, including:

[0045] The third attention information is used to fuse with the K vector obtained by processing the fourth feature representation through the second attention network to obtain the first feature representation.

[0046] In one possible implementation, the device further includes:

[0047] The FFN module is used to process the first feature representation through a feedforward neural network (FFN) to obtain a processed first feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the processed first feature representation and the second feature representation; or,

[0048] The second feature representation is processed by a feedforward neural network (FFN) to obtain a processed second feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the first feature representation and the processed second feature representation.

[0049] In one possible implementation, the acquisition module is specifically used for:

[0050] Multiple attention modules are connected in series to process the feature representations corresponding to the pixel images and the event data to obtain the first feature representation and the second feature representation; wherein, the output of each attention module is used as the input of the next adjacent attention module and is processed based on the attention mechanism.

[0051] In one possible implementation, the output of each attention module includes a feature representation corresponding to the pixel image and a feature representation corresponding to the event data;

[0052] The first feature is obtained by fusing the feature representations corresponding to the pixel image obtained from some or all of the attention modules; or,

[0053] The second feature is obtained by fusing the feature representations corresponding to the event data obtained from some or all of the attention modules.

[0054] In one possible implementation, the resolution of the 3D reconstruction result is consistent with the resolution of the pixel image.

[0055] In one possible implementation, the 3D reconstruction result includes multiple location points and depth information corresponding to each location point.

[0056] In one possible implementation, the device further includes:

[0057] The phase calculation module is used to perform phase calculation on the event data acquired by the event camera to obtain the event phase;

[0058] The second feature is represented as the feature representation corresponding to the event phase.

[0059] In one possible implementation, the event phase includes multiple phase values ​​corresponding to the target location point, wherein different phase values ​​correspond to different timestamps; the device further includes:

[0060] A filtering module is used to perform median filtering on the event phase to obtain a processed event phase; the second feature representation is the feature representation corresponding to the processed event phase; wherein...

[0061] The processed event phase includes the target phase value corresponding to the target location point, which is obtained by median filtering of multiple phase values ​​corresponding to the target location point.

[0062] Thirdly, embodiments of this application provide a system including an RGB camera, an event camera, and the apparatus as described in any of the second aspects above.

[0063] Fourthly, embodiments of this application provide a three-dimensional reconstruction apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform any of the optional methods described in the first aspect above.

[0064] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the first aspect and any optional method described above.

[0065] Sixthly, embodiments of this application provide a computer program product, including code, which, when executed, is used to implement the first aspect and any optional method described above.

[0066] Seventhly, this application provides a chip system including a processor for supporting a 3D reconstruction device in performing the functions involved in the foregoing aspects, such as transmitting or processing data or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0067] Figure 1A A structural diagram illustrating the main framework of artificial intelligence;

[0068] Figure 1B and Figure 2 This is a schematic diagram of the application system framework of the present invention;

[0069] Figure 3 This is a schematic diagram of an optional hardware structure for a terminal.

[0070] Figure 4 This is a schematic diagram of the structure of a server;

[0071] Figure 5 This is a schematic diagram of a system architecture according to this application;

[0072] Figure 6 A process for providing a cloud service;

[0073] Figure 7 This is a schematic diagram of the structure of a neural network model in an embodiment of this application;

[0074] Figure 8 This is a schematic diagram of the structure of a neural network model in an embodiment of this application;

[0075] Figure 9 This is a schematic diagram of a transformer layer architecture;

[0076] Figure 10 This is a flowchart illustrating a three-dimensional reconstruction method.

[0077] Figure 11 This is a schematic diagram of a three-dimensional reconstruction method;

[0078] Figure 12 This is a schematic diagram of a transformer layer structure;

[0079] Figure 13 This is a schematic diagram of a transformer layer structure;

[0080] Figure 14 A schematic diagram of the operation of an attention head;

[0081] Figure 15 This is a schematic diagram of a processing flow;

[0082] Figure 16A This is a schematic diagram of an NSB network;

[0083] Figure 16B A schematic diagram of an FFN;

[0084] Figures 17 to 21B This is a schematic diagram illustrating the effect of a three-dimensional reconstruction method provided in an embodiment of this application;

[0085] Figure 22 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application;

[0086] Figure 23 A schematic diagram of an execution device provided in an embodiment of this application;

[0087] Figure 24 A schematic diagram of a training device provided in an embodiment of this application;

[0088] Figure 25 This is a schematic diagram of a chip provided in an embodiment of this application. Detailed Implementation

[0089] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0090] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0091] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0092] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1A , Figure 1AThe diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0093] (1) Infrastructure

[0094] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0095] (2) Data

[0096] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0097] (3) Data processing

[0098] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0099] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0100] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0101] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0102] (4) General ability

[0103] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0104] (5) Smart Products and Industry Applications

[0105] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0106] The embodiments of this application can be applied to three-dimensional reconstruction (or downstream tasks based on the results of three-dimensional reconstruction), such as terminal unlocking or smart physical locks based on facial recognition FACE ID, counting of goods in unmanned factories based on three-dimensional reconstruction, gesture control such as contactless control based on three-dimensional reconstruction in vehicle scenarios, photography scenarios related to depth processing, 3D video generation (such as digital human generation), and other fields.

[0107] The application scenarios of this application will be introduced first. This application can be applied, but is not limited to, to applications with 3D reconstruction capabilities or cloud services provided by cloud-side servers. These will be introduced separately below:

[0108] I. 3D Reconstruction Applications

[0109] The product form of this application embodiment can be a 3D reconstruction application. The 3D reconstruction application can run on a terminal device or a cloud-based server.

[0110] In one possible implementation, 3D reconstruction applications can perform 3D reconstruction tasks or tasks based on 3D reconstruction results.

[0111] In one possible implementation, a user can open an application with 3D reconstruction capabilities installed on a terminal device. The application can acquire data collected by a camera (e.g., image data collected by an RGB camera and event data collected by an event camera). The 3D reconstruction application can obtain processing results based on the input data using the method provided in this application embodiment, and present the 3D reconstruction results or the results of downstream tasks based on the 3D reconstruction results to the user (the presentation method may include, but is not limited to, displaying, saving, uploading to the cloud, etc.).

[0112] In one possible implementation, a user can open a 3D reconstruction application installed on a terminal device. The application can acquire data collected by a camera (e.g., image data collected by an RGB camera and event data collected by an event camera). The 3D reconstruction application can send the data (or the result obtained after processing the data) to a cloud-based server. The cloud-based server generates a 3D reconstruction result based on the image using the method provided in this application embodiment, and sends the 3D reconstruction result or the result of a downstream task based on the 3D reconstruction result back to the terminal device. The terminal device can present the 3D reconstruction result or the result of a downstream task based on the 3D reconstruction result to the user (the presentation method may include, but is not limited to, displaying, saving, uploading to the cloud, etc.).

[0113] The following sections will introduce the 3D reconstruction application in this application from the perspectives of functional architecture and product architecture that implements the functions.

[0114] Reference Figure 1B , Figure 1B This is a schematic diagram of the functional architecture of the 3D reconstruction application in the embodiments of this application:

[0115] In one possible implementation, such as Figure 1B As shown, the 3D reconstruction application 102 can receive input data 101 (e.g., images and event data) and generate a 3D reconstruction result 103. The 3D reconstruction application 102 can be executed on at least one computer system (for example) and includes computer code that, when executed by one or more computers, causes the computers to perform the 3D reconstruction methods described herein.

[0116] Reference Figure 2 , Figure 2 This is a schematic diagram of the entity architecture for running a 3D reconstruction application in this embodiment of the application:

[0117] See Figure 2 , Figure 2 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 2 (The example includes a server). The server 200 can provide 3D reconstruction services to one or more terminals or perform downstream tasks based on the 3D reconstruction results.

[0118] The terminal 100 may have a 3D reconstruction application installed, or a webpage related to 3D reconstruction or downstream tasks based on 3D reconstruction results opened. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the 3D reconstruction or downstream tasks based on 3D reconstruction results interface, and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0119] It should be understood that in some optional implementations, the terminal 100 can also complete the data processing results based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0120] The following description Figure 2 The product form of the mid-terminal 100;

[0121] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle-mounted device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0122] Figure 3 A schematic diagram of an optional hardware structure for terminal 100 is shown.

[0123] refer to Figure 3 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art will understand that... Figure 3 These are merely examples of terminals or multi-functional devices and do not constitute a limitation on terminals or multi-functional devices. They may include more or fewer components than shown in the illustration, or combine certain components, or use different components.

[0124] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit 130 may include a touchscreen 131 (optional) and / or other input devices 132. The touchscreen 131 can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers, knuckles, styluses, or any suitable object on or near the touchscreen), and drive the corresponding connection devices according to a pre-set program. The touchscreen can detect the user's touch actions, convert the touch actions into touch signals and send them to the processor 170, and can receive and execute commands sent by the processor 170; the touch signal includes at least touch point coordinate information. The touchscreen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, various types of touchscreens, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touchscreen. Besides the touchscreen 131, the input unit 130 may also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0125] Other input devices 132 can acquire data collected by the camera, etc.

[0126] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In this embodiment, the display unit 140 can be used to display the interface of applications related to 3D reconstruction, etc.

[0127] The memory 120 can be used to store instructions and data. The memory 120 may primarily include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extended sets thereof. It may also include non-volatile random access memory. It provides the processor 170 with hardware, software, and data resources for managing the computing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.

[0128] The processor 170 is the control center of the terminal 100. It connects various parts of the terminal 100 via various interfaces and lines. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it performs various functions and processes data of the terminal 100, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 170. In some embodiments, the processor and memory can be implemented on a single chip; in some embodiments, they can also be implemented separately on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that the various functional modules therein perform corresponding functions, thereby controlling the corresponding components to act according to the instructions.

[0129] The memory 120 can be used to store software code related to the 3D reconstruction method, and the processor 170 can execute the steps of the chip's 3D reconstruction method. It can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve the corresponding functions.

[0130] The radio frequency unit 110 (optional) can be used for receiving and transmitting signals during information transmission or calls. For example, it can receive downlink information from the base station and process it for the processor 170; additionally, it can transmit uplink data to the base station. Typically, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the radio frequency unit 110 can also communicate wirelessly with network devices and other devices. This wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0131] In this embodiment of the application, the radio frequency unit 110 can send the data collected by the camera to the server 200 and receive the processing result information sent by the server 200.

[0132] It should be understood that the radio frequency unit 110 is optional and can be replaced with other communication interfaces, such as a network port.

[0133] The terminal 100 also includes a power supply 190 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0134] Terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0135] Although not shown, terminal 100 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with various functions, etc., which will not be described in detail here. Some or all of the methods described below can be applied to, for example... Figure 3 In the terminal 100 shown.

[0136] The following description Figure 2 The product form of the mid-range server 200;

[0137] Figure 4 A structural diagram of a server 200 is provided, as follows: Figure 4 As shown, server 200 includes bus 201, processor 202, communication interface 203, and memory 204. Processor 202, memory 204, and communication interface 203 communicate with each other via bus 201.

[0138] Bus 201 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0139] The processor 202 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0140] Memory 204 may include volatile memory, such as random access memory (RAM). Memory 204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0141] The memory 204 can be used to store software code related to the 3D reconstruction method, and the processor 202 can execute the steps of the chip's 3D reconstruction method, and can also schedule other units to achieve corresponding functions.

[0142] It should be understood that the aforementioned terminal 100 and server 200 can be centralized or distributed devices. The processors (e.g., processor 170 and processor 202) in the aforementioned terminal 100 and server 200 can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the processor can be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0143] It should be understood that the steps related to the model inference process in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture described above. The following section will further explain... Figure 5 The system architecture provided in the embodiments of this application will be described in detail.

[0144] Please see Figure 5 , Figure 5 A system architecture diagram of the system provided in the embodiments of this application is shown. Figure 5 In this system, the task processing system 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition device 560. The execution device 510 includes a computing module 511. The data acquisition device 560 acquires the large-scale open-source dataset (i.e., training set) required by the user and stores it in the database 530. The training device 520 trains the target model / rule 501 based on the training set maintained in the database 530. The trained neural network is then applied on the execution device 510. The execution device 510 can access data, code, etc., in the data storage system 550, and can also store data, instructions, etc., in the data storage system 550. The data storage system 550 can be located within the execution device 510, or it can be an external storage device relative to the execution device 510.

[0145] The trained neural network, obtained by training the target model / rule 501 via training device 520, can be applied to different systems or devices (i.e., execution device 510), specifically edge devices or end-side devices, such as mobile phones, tablets, laptops, monitoring systems (e.g., cameras), security systems, etc. Figure 5 In this embodiment, the execution device 510 is equipped with an I / O interface 512 for data interaction with external devices. A "user" can input data to the I / O interface 512 through a client device 540. For example, the client device 540 can be a camera in a monitoring system. Images and event data captured by the camera are input to the calculation module 511 of the execution device 510. The calculation module 511 detects the input target image and obtains a detection result, which is then output to the camera or displayed directly on the display interface of the execution device 510 (if applicable). Furthermore, in some embodiments of this application, the client device 540 can also be integrated into the execution device 510. For example, when the execution device 510 is a mobile phone, the target task can be directly obtained through the mobile phone (e.g., images and event data captured by the phone's camera) or received from other devices (e.g., another mobile phone). The calculation module 511 within the mobile phone then detects the target task, obtains a detection result, and directly displays the result on the mobile phone's display interface. The product form of execution device 510 and customer device 540 is not limited here.

[0146] It is worth noting that, Figure 5 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 5 In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0147] From the inference side of the model:

[0148] In this embodiment, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in this embodiment.

[0149] In this embodiment of the application, the computing module 511 of the execution device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0150] Specifically, the computing module 511 of the execution device 520 can be a hardware system with the function of executing instructions. The steps related to the model inference process provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model inference process provided in this application embodiment.

[0151] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the model reasoning process provided in the embodiments of this application can also be implemented by the hardware system in the computing module 511 of the execution device 520 without the function of executing instructions, which is not limited here.

[0152] From the training side of the model:

[0153] In this embodiment of the application, the training device 520 can access the memory ( Figure 5 (Not shown in the diagram, but can be integrated into the training device 520 or deployed separately from the training device 520) The code stored in the diagram can be used to implement the steps related to model training in the embodiments of this application.

[0154] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0155] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the training of the neutralization model provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0156] II. Cloud services for 3D reconstruction provided by the server:

[0157] In one possible implementation, the server can provide services such as 3D reconstruction or downstream tasks based on 3D reconstruction results to the client through an application programming interface (API).

[0158] In this process, the terminal device can send relevant parameters (such as images and event data) to the server through the API provided by the cloud. The server can obtain the processing result based on the received parameters and return the processing result to the terminal.

[0159] The descriptions of the terminal and server are the same as those in the above embodiments, and will not be repeated here.

[0160] like Figure 6 The process of using a cloud-based 3D reconstruction service provided by a cloud platform is illustrated.

[0161] 1. Activate and purchase content moderation services.

[0162] 2. Users can download the software development kit (SDK) corresponding to the content moderation service. Cloud platforms usually provide multiple development versions of the SDK for users to choose from according to their development environment needs, such as JAVA version SDK, Python version SDK, PHP version SDK, Android version SDK, etc.

[0163] 3. After downloading the corresponding version of the SDK to their local machine according to their needs, users can import the SDK project into their local development environment, configure and debug it in the local development environment, and develop other functions in the local development environment to form an application that integrates 3D reconstruction capabilities.

[0164] 4. When a 3D reconstruction application needs to perform 3D reconstruction or downstream tasks based on the 3D reconstruction results, it can trigger API calls for 3D reconstruction or downstream tasks based on the 3D reconstruction results. When an application triggers 3D reconstruction or downstream tasks based on the 3D reconstruction results, it initiates an API request to the running instance of the 3D reconstruction service in the cloud environment. The API request carries an image, which is then processed by the running instance in the cloud environment to obtain the processing result.

[0165] 5. The cloud environment returns the processing results to the application, thereby completing a 3D reconstruction or invoking downstream task services based on the 3D reconstruction results.

[0166] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0167] (1) Neural Network

[0168] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0169]

[0170] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0171] (2) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn reasonable weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0172] CNN is a very common type of neural network. Below, we will combine... Figure 7 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0173] like Figure 7As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.

[0174] Convolutional / pooling layers 220:

[0175] Convolutional layers:

[0176] like Figure 7 The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0177] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0178] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0179] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0180] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0181] Pooling layer:

[0182] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 7Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0183] Fully connected layer 230:

[0184] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 can include multiple hidden layers (such as...). Figure 7 As shown in 231, 232 to 23n), the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0185] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 7 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 7 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0186] It should be noted that, as Figure 7The convolutional neural network 200 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, for example, including only... Figure 7 As shown in the network structure, for example, the convolutional neural network used in the embodiments of this application may only include an input layer 210, a convolutional / pooling layer 220, and an output layer 240.

[0187] It should be noted that, as Figure 7 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 8 The multiple convolutional / pooling layers shown are implemented in parallel, and the extracted features are all input into the fully connected layer 230 for processing.

[0188] (3) Deep Neural Networks

[0189] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0190] (4) Transformer layer

[0191] Reference Figure 9 , Figure 9 This is a schematic diagram of a transformer layer architecture, such as Figure 9 As shown, the neural network includes an embedding layer and at least one transformer layer. The at least one transformer layer can be N transformer layers (N being an integer greater than 0). Each transformer layer includes sequentially adjacent attention layers, add and normalize layers, feed-forward layers, and add and normalize layers. In the embedding layer, the current input is embedded to obtain multiple feature vectors. In the attention layer, P input vectors are obtained from the layer above the transformer layer. Using any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained. This process determines P intermediate vectors corresponding to the P input vectors. In the pooling layer, the P intermediate vectors are pooled into Q output vectors, where the multiple output vectors obtained from the last transformer layer in at least one transformer layer are used as feature representations of the current input.

[0192] Next, we will explain each of the above steps in detail with specific examples.

[0193] First, in the embedding layer, the current input is embedded to obtain multiple feature vectors.

[0194] The embedding layer can be called the input embedding layer. The current input can be text, such as a paragraph or a sentence. The text can be Chinese, English, or other languages. After obtaining the current input, the embedding layer can perform embedding processing on each word in the current input to obtain the feature vector of each word. In some embodiments, such as... Figure 9As shown, the embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each word in the current input, so as to obtain a word embedding vector for each word. In the positional encoding layer, the position of each word in the current input can be acquired, and then a position vector is generated for the position of each word. In some examples, the position of each word may be the absolute position of each word in the current input. Taking the current input "which date should I repay the Ant Credit Pay" as an example, the position of the character "which" can be expressed as the first position, the position of the character "date" can be expressed as the second position, and so on. In some examples, the position of each word may be the relative position between each word. Still taking the current input "which date should I make the repayment" as an example, the position of the character "which" can be expressed as before "date", the position of the character "date" can be expressed as after "which" and before "should", and so on. When the word embedding vector and the position vector of each word in the current input are obtained, the position vector of each word and the corresponding word embedding vector can be combined to obtain the feature vector of each word, that is, a plurality of feature vectors corresponding to the current input are obtained. The plurality of feature vectors can be represented as an embedding matrix with a preset dimension. If the number of feature vectors in the plurality of feature vectors is set to M and the preset dimension is H dimensions, the plurality of feature vectors can be represented as an M×H embedding matrix.

[0195] Next, P input vectors are acquired from the previous layer of the first transformer layer, with any first input vector among the P input vectors as the center, based on the correlation degree between each input vector within a preset attention window range and the first input vector, an intermediate vector corresponding to the first input vector is obtained, so that P intermediate vectors corresponding to the P input vectors are determined. The attention layer may also be referred to as a multi-head attention layer. In an example, the attention layer may be a fixed window multi-head attention layer.

[0196] In some embodiments, the first transformer layer may be the next layer of the above-mentioned embedding layer, and the P input vectors are the plurality of feature vectors obtained from the embedding layer. In some embodiments, at least one transformer layer in the neural network provided by the embodiments of this specification further includes a second transformer layer. The second transformer layer is the previous layer of the first self-attention, and the P input vectors are P output vectors output by the second transformer layer. In the last transformer layer of the neural network, the plurality of output vectors obtained through the above steps can be used as the feature representation of the current input. This feature representation is a feature representation of the current input suitable for computer processing.

[0197] (5) Attention mechanism

[0198] Attention mechanisms mimic the internal processes of biological observation—aligning internal experience with external senses to increase the precision of observation in specific areas. They enable the rapid sifting of high-value information from a large volume of data using limited attentional resources. Attention mechanisms can quickly extract important features from sparse data and are therefore widely used in natural language processing tasks, particularly machine translation. Self-attention mechanisms (SE1F-attention) are an improvement on attention mechanisms, reducing reliance on external information and excelling at capturing the internal correlations of data or features. The essential idea of ​​attention mechanisms can be rewritten as follows:

[0199] In this formula, Lx = ||Source|| represents the length of the Source. The meaning is that the elements in the Source are imagined as a series of data pairs. Given a Query element in the Target, the similarity or relevance between the Query and each Key is calculated to obtain the weight coefficient of the Value corresponding to each Key. Then, the Values ​​are weighted and summed to obtain the final Attention value. Therefore, the Attention mechanism essentially performs a weighted sum of the Values ​​of the elements in the Source, while the Query and Key are used to calculate the weight coefficients of their corresponding Values. Conceptually, Attention can be understood as selectively filtering a small amount of important information from a large amount of information and focusing on this important information, ignoring most of the unimportant information. The focusing process is reflected in the calculation of the weight coefficients; the larger the weight, the more focused it is on its corresponding Value. That is, the weight represents the importance of the information, and the Value is the corresponding information. Self-attention can be understood as intra attention. The attention mechanism occurs between the elements of the Target (Query) and all elements of the Source. Self-attention refers to the attention mechanism that occurs between elements within the Source or between elements within the Target. It can also be understood as the attention calculation mechanism in the special case where Target = Source. The specific calculation process is the same, only the calculation object changes.

[0200] (6) Event cameras

[0201] It is a bio-inspired visual sensor that works in a completely different way from standard cameras. Instead of outputting intensity image frames at a constant rate, the event camera outputs only information related to local pixel-level brightness changes.

[0202] (7) Warp operation

[0203] Operations that are typically performed in conjunction with a stream (such as optical flow, and scene flow in this application) represent an affine transformation of an image relative to a stream (such as optical flow, scene flow), such as rotation, translation, scaling, etc.

[0204] (8) Loss Function

[0205] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0206] (9) Backpropagation algorithm

[0207] Backpropagation (BP) can be used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0208] Structured light imaging, as a crucial component of 3D reconstruction, reconstructs the 3D structure of objects through actively projected coded patterns. It boasts advantages such as simplicity, non-contact operation, high robustness, and low loss. Compared to passive methods like binocular 3D reconstruction, structured light imaging offers superior imaging performance and higher accuracy in extreme scenarios such as darkness and texture loss. Traditional structured light imaging achieves high-resolution, high-quality 3D reconstruction using a stereo system comprised of a conventional active pixel sensor (APS) camera and an active light source (laser emitter, digital projector).

[0209] However, as the requirements for imaging quality and speed in structured light imaging become increasingly demanding, the limitations of traditional cameras in terms of resolution and frame rate prevent the achievement of fast, high-quality 3D imaging. Furthermore, active projection is susceptible to ambient light interference, and the color and reflection of object surfaces also affect the decoding of the projected pattern, reducing the accuracy of active structured light imaging.

[0210] Event cameras (dynamic vision sensors, DVS), as a visual sensor that has attracted widespread attention in recent years, borrow from bionics in their imaging method, driving imaging through changes in light intensity. They possess advantages such as low latency, high dynamic range, low redundancy, and independent pixel excitation, making them ideal sensors for structured light imaging. Firstly, their low latency allows for extremely high data acquisition bandwidth, effectively compensating for the insufficient frame rate of traditional cameras. Secondly, their high dynamic range (120dB) enables the sensor to still acquire valid information even under overexposure conditions. However, due to the independent excitation of camera pixels, they are susceptible to significant noise from changes in natural lighting, severely reducing the accuracy of 3D reconstruction. Furthermore, due to the trade-off between low latency and high resolution, existing event cameras have relatively low spatial resolution, failing to achieve high-resolution 3D reconstruction.

[0211] To address the aforementioned issues, this application provides a three-dimensional reconstruction method, which can be either a feedforward process for model training or an inference process.

[0212] Reference Figure 10 , Figure 10 A three-dimensional reconstruction method provided in the embodiments of this application, such as Figure 10 As shown, the three-dimensional reconstruction method provided in this application includes:

[0213] 1001. Obtain a first feature representation and a second feature representation, wherein the first feature representation is the feature representation corresponding to the pixel image captured by the RGB camera, and the second feature representation is the feature representation corresponding to the event data captured by the event camera; the pixel image and the event data are acquired for the same scene.

[0214] In this embodiment of the application, the execution subject of step 1001 can be a terminal device. The terminal device can be a portable mobile device, such as, but not limited to, mobile or portable computing devices (such as smartphones), personal computers, server computers, handheld devices (such as tablets) or laptop devices, multiprocessor systems, game consoles or controllers, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices with wearable or accessory form factors (such as watches, glasses, headphones or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0215] In this embodiment of the application, the execution subject of step 1001 can be a cloud-side server. The server can receive pixel images sent from the terminal device and event data collected by the event camera, and thus the server can obtain pixel images and event data collected by the event camera.

[0216] In one possible implementation, the pixel image and the event data are acquired within the same time period for the same scene. For example, the pixel image can be an image captured by an RGB camera on the terminal device, and the event data can be an image captured by an event camera on the terminal device for the same scene.

[0217] The pixel image can be the average of multiple frames (images obtained within the target exposure time period). In other words, the pixel image can be obtained by averaging multiple consecutive frames to obtain a single composite pixel image. The data collected by the event camera can include event data at multiple time points. In this embodiment, the event data can be the event data at a single event point within the target exposure time period corresponding to the pixel image.

[0218] In this context, an event point, also known as an event, is the basic principle of an event camera. When the cumulative brightness change of a pixel reaches a trigger condition (the change reaches a certain level), an event point is output. Therefore, an event point can be understood as an expression of an event: at what time (timestamp), which pixel (pixel coordinates), the brightness increased or decreased (brightness change) occurred.

[0219] The following section describes how to obtain event data and the preprocessing steps performed on it:

[0220] In one possible implementation, for the area where the 3D reconstructed object is located, a dynamic projection pattern can be generated in the area using an active light source (such as a laser emitter, digital projector, etc.). For example, the size of the projection pattern can be H×W.

[0221] In one possible implementation, a traditional coded multi-projection pattern can be used. Dynamic projection simplifies the projection content through time coding. The coding method can be as shown in equation (1), and the projection result is as follows: Figure 11 As shown in pattern (a):

[0222] F(x, y, t) = A*s(ω) c x+ω t t), x∈[1, W], y∈[1, H] (1)

[0223] Where A is the grayscale value of the projected pattern (optional, A = 255). s(·) represents the square wave projection, ω c and ω t These represent the spatial angular frequency and the temporal angular frequency, respectively. The size of the projected pattern is (W×H). As shown in the figure, the dynamic moving pattern of black and white stripes can effectively generate changes in light intensity, helping the event camera to acquire projection changes, thereby achieving effective phase decoding.

[0224] In one possible implementation, the RGB camera can acquire multiple image frames for the target scene, and the event camera can acquire event data for multiple event points for the target scene. In this embodiment, the feature representations of pixel images and event data are fused. Since the event data acquired by the event camera has a high temporal resolution and the image frames acquired by the RGB camera have the characteristics of low noise and high spatial resolution, the reconstruction result obtained by 3D reconstruction based on the fused feature representation can have the characteristics of high temporal resolution, low noise and high spatial resolution.

[0225] To ensure the accuracy of the fusion object, it is necessary to ensure that the pixel image and event data are matched in both time and space. That is, the pixel image and event data corresponding to feature fusion can be obtained by the RGB camera and the event camera respectively from the same scene, and the event data is obtained from multiple frames of event data collected by the event camera within the target time period, and the pixel image is obtained from the image collected by the RGB camera within the target time period.

[0226] In other words, the exposure time of the pixel image is within the target time period, specifically a sub-time period within the target time period. Within the target time period, the event camera can acquire multiple frames of event data. For example, phase calculation can be performed on the multiple frames of event data to obtain the event data.

[0227] In one possible implementation, phase calculation can be performed on the event data acquired by the event camera to obtain the event phase; the second feature representation is the feature representation corresponding to the event phase.

[0228] In one possible implementation, median filtering can be applied to the event phase. Specifically, the event phase includes multiple phase values ​​corresponding to the target location point, where different phase values ​​correspond to different timestamps. Median filtering can be applied to the event phase to obtain a processed event phase. The second feature representation is the feature representation corresponding to the processed event phase. The processed event phase includes the target phase value corresponding to the target location point, which is obtained by median filtering of the multiple phase values ​​corresponding to the target location point. Due to the characteristics of the event camera imaging principle, effective data is accompanied by a large amount of random noise. Generally, random fluctuations in event points can be effectively removed by simple filtering, specifically by applying median filtering to the timestamps of all event points within the same pixel within a period.

[0229] Next, we will introduce a schematic diagram of phase calculation and median filtering. Optionally, in order to obtain the phase decoding of an object from the timestamp of the event stream, we can select a starting reference time based on the formation method of the time surface. According to Equation (1), under ideal conditions, within an effective scanning period T, all pixels have experienced a unique positive polarity event and a unique negative polarity event. The phase calculation method is shown in Equation 2:

[0230] P E (u, v) = ω t (t(u,v)-t ref )%(2π), P E (u, v)∈[0, 2π) (2)

[0231] Where (u, v) are pixel coordinates, t ref Let t(u, v) be the selected starting reference time, and t(u, v) be the timestamp (positive and negative polarity) of the event points within the period. Due to the characteristics of the event camera imaging principle, the effective data is accompanied by a large amount of random noise. Generally speaking, random fluctuations in event points can be resolved through simple filtering by performing median filtering on the timestamps of all event points of the same pixel within the period. That is, the filtered phase is as follows:

[0232] P E (u, v) = ω t (Med{t1(u,v),t2(u,v)...,t n (u, v)}-t ref )%(2π) (3)

[0233] Among them, t i (u, v), i∈{1,2,...,n} is the timestamp of the i-th event point on pixel (u, v) within the period. Median filtering can effectively remove random noise.

[0234] Thus, the event phase preprocessing module compresses the 3D event stream information into a two-dimensional image and obtains the wrapped phase decoding of the projection pattern through a simple and fast method. Compared with the decoding of traditional camera images, the embodiment of this application is faster and more efficient.

[0235] Because event cameras and traditional cameras have different fields of view and resolutions, the content they capture of the scene differs. Using a calibrated homography matrix H_(EC), the image obtained from the traditional camera is mapped to the corresponding position on the event camera, achieving pixel-level alignment.

[0236] In one possible implementation, frame space alignment can be performed on pixel images and event data. Since the event camera cannot obtain frame information, the key to alignment is obtaining valid frame images from a set of event points within a certain time period. Optionally, the checkerboard calibration board can be recovered using the classic E2VID algorithm. In image matching tasks, alignment of corresponding regions on the same plane can be achieved by calculating the homography matrix H through feature point matching. E-C The pixel correspondence between the two different planes is as follows:

[0237] q2∝H E-C q1 (4);

[0238] After normalization expansion, we can obtain the coordinates of point q1 (u1, v1) and point q2 (u2, v2):

[0239]

[0240] It can be seen that H E-C There are 8 unknowns. Each pair of matching points can provide 2 equations. Ideally, four pairs of matching points are sufficient to solve the matrix. However, due to errors in mismatched points and coordinate points, overdetermined equations formed by multiple pairs of matching points can effectively reduce errors. Alternatively, RANSAC (Random Sample Consensus) can be used to obtain a least-squares solution and the optimal matching homography matrix H. E-C The corresponding regions of the event camera and the APS camera are obtained based on the solved mapping matrix, thus achieving pixel-level mapping.

[0241] In one possible implementation, a first feature representation corresponding to the pixel image can be obtained through feature extraction, and a second feature representation corresponding to the event data can be obtained through feature extraction.

[0242] In one possible implementation, the first and second feature representations obtained through feature extraction can be feature representations of the same size. Here, the "size" of the feature representation can be understood as its width and height.

[0243] The following describes how to obtain the first feature representation of a pixel image through feature extraction, and how to obtain the second feature representation of event data through feature extraction:

[0244] In one possible implementation, features can be extracted from pixel images using a feature extraction network (e.g., an attention-based feature extraction network) to obtain a first feature representation of the pixel images, and features can be extracted from event data using a feature extraction network (e.g., an attention-based feature extraction network) to obtain a second feature representation of the event data.

[0245] In one possible implementation, the pixel image and event data can be processed separately by a feature extraction module such as SFE to obtain shallow feature representations of the pixel image and the event data. Then, the shallow feature representations of the pixel image and the event data can be processed by an attention network to obtain deep feature representations of the pixel image and the event data (that is, the first feature representation and the second feature representation in the embodiments of this application).

[0246] In one possible implementation, multiple cascaded attention modules can be used to process the feature representations corresponding to the pixel image and the event data to obtain the first feature representation and the second feature representation. The output of each attention module serves as the input to the next adjacent attention module for attention-based processing. In other words, each attention module can perform attention-based processing based on the feature representations of the pixel image and the event data output by the previous connected attention module (the first attention module obtains a shallow feature representation of the pixel image and a shallow feature representation of the event data, while the last attention module obtains the first and second feature representations).

[0247] In one possible implementation, since the pixel images captured by the camera have lower noise than the event data captured by the event camera, in order to achieve the fusion of the two modalities, a neural network can be used to extract target information from the feature representation of the pixel images. This target information can represent the weight of the noise contained in each location point of the feature representation. The target information and the fourth feature representation are then fused to achieve effective suppression of event noise.

[0248] In one possible implementation, the neural network can be an attention network, where the target information can be global relevance weights. The attention weights of the pixel image and event data are added together, resulting in low-noise Attenuation. FThe global relevance weights can effectively achieve Atten E The weighting ratio of noise is adjusted to effectively suppress event noise.

[0249] In one possible implementation, the neural network is a first attention network. Based on a third feature representation, target information is obtained through the first attention network, and the target information includes first attention information. The first attention information is the spatial correlation information of the pixel image. The third feature representation can be the shallow feature representation corresponding to the pixel image described in the above embodiments, or the feature representation corresponding to the pixel image obtained through a certain attention module. The first attention network can belong to the next adjacent attention module of that attention module. The spatial correlation information of the pixel image can represent the weight ratio of noise.

[0250] Due to the unique nature of event stream data and the resolution difference between event cameras and traditional cameras, a self-attention mechanism was added to the network to achieve modality fusion, and a high-resolution reconstruction module was used to obtain fast and high-quality results.

[0251] In one possible implementation, a second attention network can be used to process the fourth feature representation to obtain second attention information. The second attention information is the spatial correlation information of the event data. The third feature representation can be the shallow feature representation corresponding to the event data described in the above embodiments, or the feature representation corresponding to the event data obtained through a certain attention module. The second attention network can belong to the next adjacent attention module of the attention module. The second attention information is used to perform attention fusion with the first attention information to obtain third attention information. The third attention information is used to obtain the second feature representation.

[0252] In one possible implementation, the attention fusion is a summation operation at corresponding positions.

[0253] In one possible implementation, the first attention information, the second attention information, and the third attention information can be used to represent the correlation between the Q vector and the K vector; the third attention information can be used to fuse with the K vector obtained by processing the fourth feature representation through the second attention network to obtain the first feature representation.

[0254] The following example, using a neural network model based on transformer layers, illustrates how to obtain the correlation between the Q vector and the K vector:

[0255] In one possible implementation, refer to Figure 12 , Figure 12This is a schematic diagram of the structure of a neural network model in an embodiment of this application, such as... Figure 12 As illustrated, a transformer-based neural network model can include sequentially connected embedding layers and multiple transformer layers. As those skilled in the art will understand, transformer models are commonly used for performing natural language processing (NLP) tasks. It is important to understand that... Figure 12 The structure shown is merely an example; the number of transformer layers can be set as needed. The neural network model determines the feature vector corresponding to the current node based on the N output vectors obtained from each transformer layer.

[0256] In the embedding layer, the current input is embedded to obtain multiple feature vectors. The core feature of the transformer model lies in its unique attention mechanism. (See reference...) Figure 13 , Figure 13 This is a schematic diagram of a transformer layer structure. The transformer layers of various neural networks in the embodiments of this application can be referenced from this diagram. Figure 13 The structure shown in the figure, such as Figure 13 As shown, the transformer layer consists of a multi-head attention layer, an add & normalization layer, a feed forward layer, and another add & normalization layer, which are sequentially adjacent to each other.

[0257] The multi-head attention layer receives N input vectors X1 from the layer above, which can also be represented as matrix X. It employs a self-attention mechanism, transforming each vector based on the correlation between them to obtain N output vectors, which can also be represented as matrix Y. This multi-head attention layer is understood to be directly connected to the embedding layer, for example... Figure 13 In a transformer layer directly connected to the embedding layer, the input vector it receives is the embedding vector output by the embedding layer; when this multi-head attention layer is a multi-head attention layer included in subsequent transformer layers, for example... Figure 13 The transformer layer directly connected to the previous transformer layer includes a multi-head attention layer, whose input vector is the output vector of the previous transformer layer. In the multi-head attention layer, the MHA layer includes multiple attention heads (e.g., ...). Figure 14 The following are Head 1, Head 2, ..., Head N shown in the figure.

[0258] Figure 14This is a schematic diagram illustrating the operation of an attention head, showing how the attention head transforms an input matrix X into an output matrix Y. For example... Figure 14 As shown, the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V are applied to N input vectors respectively.<X1,X2,…,XN> The input vectors Xi are transformed to obtain the first intermediate vector (q vector), second intermediate vector (k vector), and third intermediate vector (v vector) corresponding to each input vector. Operationally, the input matrix X, composed of N input vectors, can be linearly transformed using the first transformation matrix Q, the second transformation matrix K, and the third transformation matrix V, respectively, to obtain the Q matrix, K matrix, and V matrix of the input matrix. These matrices are then split to obtain the q vector, k vector, and v vector corresponding to each input vector. For any i-th input vector Xi among the N input vectors, the correlation degree between the i-th input vector Xi and each input vector Xj is determined based on the dot product operation between the first intermediate vector (q vector, qi) corresponding to the i-th input vector and each second intermediate vector (k vector, kj) corresponding to each input vector Xj. Although the dot product result of qi and kj can be directly used to determine the correlation degree, a more classic approach is to first divide the dot product result by a constant, then perform a softmax operation, and use the result as the correlation degree between the input vector Xi and Xj (that is, the correlation degree between the Q vector and the K vector), i.e.:

[0259]

[0260] Therefore, the correlation degree α between the i-th input vector Xi and each of the input vectors Xj can be used as the basis for determining the correlation degree α between the i-th input vector Xi and each of the input vectors Xj. i，j As a weighting factor, the third intermediate vector (v vector, vj) corresponding to each input vector Xj is weighted and combined to obtain the i-th combined vector Ci corresponding to the i-th input vector Xi:

[0261]

[0262] Therefore, we can obtain a vector sequence of N combined vectors corresponding to N input vectors.<C1,C2,…,CN> Or matrix C. Based on this combined vector sequence, N output vectors can be obtained. Specifically, in one embodiment, the vector sequence of N combined vectors can be directly used as N output vectors, i.e., Yi = Ci. In this case, the output matrix Y is the combined vector matrix C, which can also be written as:

[0263]

[0264] The above describes the processing flow of an attention head. In the MHA architecture, the MHA layer maintains m sets of transformation matrices. Each set of transformation matrices includes the aforementioned first transformation matrix Q, second transformation matrix K, and third transformation matrix V, allowing the above operations to be performed in parallel to obtain m combined vector sequences (i.e., m matrices C). Each vector sequence includes N combined vectors obtained based on a set of transformation matrices. In this case, the MHA layer concatenates the m combined vector sequences to obtain a concatenated matrix; then, it transforms this concatenated matrix using the fourth transformation matrix W to obtain the final output matrix Y. This output matrix Y can be split into N output vectors.<Y1,Y2,…,YN> Through the above operations, the MHA layer performs transformation operations based on the correlation between the N input vectors to obtain N output vectors.

[0265] Compared to multi-layer convolutions for receptive field expansion, global relevance calculation of feature maps can be performed using two-dimensional matrix multiplication to obtain the global receptive field. However, this method only calculates global relevance weights, losing local relevance at specific pixel locations, and cannot effectively guarantee the unity of local and global self-attention mechanisms. To obtain a local receptive field, a corresponding feedforward network (FFN) can be added. Its basic components can include an input layer, hidden layers, activation layers, and an output layer. The shallow network in the hidden layers can realize the calculation of local relevance.

[0266] Reference Figure 15 , Figure 15 This is a framework diagram of an embodiment of this application.

[0267] In one possible implementation, the first feature representation can also be processed by a feedforward neural network (FFN) to obtain a processed first feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the processed first feature representation and the second feature representation; or, the second feature representation can be processed by a feedforward neural network (FFN) to obtain a processed second feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the first feature representation and the processed second feature representation.

[0268] It should be understood that the outputs of some or all of the attention modules can be fused to obtain the first feature representation and the second feature representation, ensuring that the event and image feature maps retain the proportion of their respective features.

[0269] In one possible implementation, the output of each attention module includes a feature representation corresponding to the pixel image and a feature representation corresponding to the event data; the first feature representation is obtained by fusing the feature representations corresponding to the pixel image obtained by some or all of the attention modules; or, the second feature representation is obtained by fusing the feature representations corresponding to the event data obtained by some or all of the attention modules.

[0270] 1002. The first feature representation and the second feature representation are fused to obtain the third feature representation.

[0271] In one possible implementation, the first feature representation and the second feature representation have the same size in the spatial dimension, and the features are fused into a splicing operation in the channel dimension.

[0272] 1003. Based on the fusion result, the scene is reconstructed in three dimensions using a three-dimensional reconstruction network to obtain the three-dimensional reconstruction result.

[0273] For example, to address the issue of different input resolutions, based on the high resolution of the image, high-resolution depth estimation results were obtained through the UP-Net network module.

[0274] In one possible implementation, the resolution of the 3D reconstruction result is consistent with the resolution of the pixel image.

[0275] In one possible implementation, the 3D reconstruction result includes multiple location points and the corresponding depth information for each location point. Since structured light imaging is a 3D image, to facilitate network training, the 3D image is mapped to pixel depth information, thereby converting it into a prediction of depth estimation.

[0276] The following describes a specific example of an embodiment of this application with reference to the accompanying drawings:

[0277] like Figure 15 As shown, projection decoding includes two parts: wrap-around phase decoding and phase unwrapping. During the event stream decoding process obtained from the event camera, only wrap-around phase decoding can be achieved, making it difficult to accurately obtain the unwrapped absolute phase. Furthermore, since the phase values ​​corresponding to the event stream data differ from the traditional image intensity values, effectively fusing different modal data poses a significant challenge to the network. Secondly, the median filtering used in preprocessing can only remove outlier event noise, and its effect on suppressing temporal fluctuations and some small event noise is poor. In this embodiment, the feature fusion module FF-Net effectively solves the above problems, with the noise suppression block (NSB) being the core part of the fusion module. Its basic framework is as follows: Figure 16A As shown.

[0278] It includes an RDB feature extraction component, a dual-channel self-attention mechanism (D-MHSA), linear normalization, and a feed-forward network (FFN). After extracting shallow features of the image through the SFE module, a multi-level noise suppression module enables the suppression of event phase noise and modality fusion using a single high-resolution image. The self-attention mechanism in the dual channels obtains the spatial correlation of the wrapped phase, thereby completing the spatial unfolding of the phase.

[0279] The basic implementation process of MHSA is as follows:

[0280]

[0281] In this model, the vectors Q, K, and V obtained through simple convolution represent the Query vector, Key vector, and Value vector, respectively. Q is the query vector, K represents the vector indicating the correlation between the queried information and other information, and V represents the vector of the queried information. The self-attention mechanism of the 2D feature map can effectively extract the correlation between spatial pixels. Especially in stereo imaging tasks, the correlation between pixels can effectively extract the baseline, enabling pixel-level disparity analysis and depth estimation.

[0282] Compared to multi-layer convolutions for receptive field expansion, MHSA calculates the global relevance of feature maps through two-dimensional matrix multiplication, thus obtaining the global receptive field. However, MHSA can only calculate global relevance weights, losing the local relevance at specific pixel locations, and cannot effectively guarantee the unity of local and global self-attention mechanisms. To obtain the local receptive field, a corresponding feedforward neural network (FFN) is added to MHSA, such as... Figure 16B This is a simple feedforward neural network. Its basic components include an input layer, hidden layers, activation layers, and an output layer. The shallow network in the hidden layers enables the calculation of local correlations.

[0283] This embodiment introduces a self-attention mechanism into the fusion network module FF-Net, achieving dual-channel self-attention extraction and fusion based on the traditional self-attention mechanism. After passing through the shallow feature extraction module SFE, the image and event preprocessing phase obtains shallow features, which can be represented as... The number of channels is C = 64, and the shallow feature map is H×W = (128, 128).

[0284] Because the self-attention mechanism achieves an ideal combination of local and global receptive fields, FFN cascades multiple layers (e.g., 6 layers) of noise suppression modules (NSBs). The specific implementation of each NSB module is as follows... Figure 16AAs shown in the left half, after extracting image and event features using the RDB module, a D-MHSA (Double Multi-Head Self-Attention) module is used to fuse the two different modalities of data. The specific fusion method is as follows: Figure 15 The middle section is shown. D-MHSA extracts Q, K, and V feature vectors separately from feature maps E and F, and uses self-attention to obtain Attenuation. E Atten F To reduce the amount of data caused by matrix multiplication and ensure the normal expansion of vector Q, this embodiment performs downsampling of the input feature map before calculating K and V, which effectively reduces the amount of data to be calculated.

[0285] Secondly, by Figure 16A As can be seen from the D-MHSA part, in order to achieve the fusion of different modalities, the attention weights of images and events are added together, resulting in low-noise Attenuation. F The global relevance weights can effectively achieve Atten E The weighting ratio of noise is adjusted to effectively suppress event noise. Simultaneously, data fusion between different modalities is achieved, and output feature maps are obtained.

[0286] After fusing the global feature maps, E, E′ and F, F′ are superimposed before inputting into the FFN, ensuring that the event and image feature maps retain the proportions of their respective features, and the features are normalized. Since the RDB (Residual DenseBlocks) module can implement the pixel local receptive field task, this embodiment simplifies the FFN network structure while maintaining the self-attention mechanism. Its specific structure is as follows: Figure 16B As shown, the input channels C=64 are expanded to C=64x4 in the hidden layer. The hidden layer consists of two convolutional layers Conv3×3 with padding=1, which ensures that the feature map size remains unchanged. The GELU activation function is used in the activation layer to reduce the number of channels to the original number of channels before output, thereby realizing the acquisition of the local receptive field.

[0287] To further integrate images and event phase frames, FF-Net, after completing the feature extraction of the multi-level noise suppression module (NSB), superimposes the multi-level feature map channels and uses a global feature fusion module (GFF) to achieve the fusion of features from different layers. The GFF module includes two convolutional layers: a channel-reducing convolutional layer Conv1×1 and a fusion convolutional layer Conv3×3. Multi-level feature fusion plays an important role in information integrity and feature extraction effect.

[0288] The resolution of the images acquired in this embodiment is H. F ×W F=640×640, the resolution of the event phase frame is H E ×W E =256×256. To address the issue of different input resolutions, based on the high resolution of the image, a high-resolution depth estimation result is obtained through the UP-Net network module. The specific implementation process of the UP-Net network is as follows: First, using a high-resolution reconstruction network, feature unfolding is performed on the feature map after channel stacking, and the pixel coordinates are normalized to the range (-1, 1) according to the output resolution. This is then concatenated with the unfolded feature map in terms of channel dimensions. Finally, after passing through multiple (e.g., 4) fully connected layers, the final depth estimation result is obtained.

[0289] The following section will introduce the beneficial effects of this application in conjunction with experiments:

[0290] On the given training and test sets, the embodiments of this application are compared with the benchmark algorithm, and the specific performance is shown in Table 1.

[0291] Based on the imaging target model type, the dataset was divided into a training set including simple texture models, light texture models, and complex texture models, with all training sets consisting of color models. The test set included all color models and some white models, as well as all datasets at scan cycles of 64ms, 32ms, and 16ms.

[0292] Table 1 Comparison Results of Objective Indicators

[0293]

[0294] The traditional algorithm NFPP, also based on dynamic surface structured light scanning projection, was chosen for comparison. Unlike the embodiments in this application, it obtains the event stream through sinusoidal surface structured light projection and obtains the event phase based on cross-correlation operations. While obtaining the phase through cross-correlation is more complex and redundant than the calculation in this paper, it has higher applicability. The phase of a traditional image can also be obtained using this method. The method of obtaining object depth through events is called NFPP(-E), while the method of obtaining object depth through a traditional image is called NFPP(-F). For effective comparison, data acquisition was completed within 128ms, and the frame rate of the traditional camera was set to 30FPS.

[0295] Quantitative results show that the proposed method achieves the best performance in both absolute and relative errors. It is evident that NFPP(-F) has the largest error; the low frame rate of traditional cameras results in fewer images acquired within 128ms, leading to an imbalance between the generated event phase and the traditional image phase. Figure 17 As shown, the phase obtained through cross-correlation has low spatial resolution, and the phase acquisition compresses effective information, making it impossible to obtain effective depth estimation results.

[0296] Secondly, as model complexity increases, the accuracy of both NFPP(-E) and MMFN decreases. However, in tests on color models, the accuracy of NFPP(-E) actually improves, while the accuracy of MMFN still declines. The results indicate that the event camera is unaffected by object color. While object color still has some influence on the MMFN algorithm due to the use of single-frame high-resolution images, it still maintains high accuracy and good robustness.

[0297] Figure 18 For a qualitative comparison of the algorithms, the results show that NFPP(-F), due to its low phase resolution, exhibits abruptly discontinuous depth estimations and fails to recover the details of object depth estimation. In contrast, the low latency of event cameras allows for phase retrieval of event streams during high-speed scanning, and the NFPP(-E) results show that it can recover object depth information more accurately. However, traditional methods cannot effectively suppress event noise, resulting in a large amount of random noise in the depth estimation results, which is more pronounced at phase transitions. This invention combines the advantages of network fusion, integrating the high temporal resolution of events, the low noise of images, and the edge clarity characteristics brought by high spatial resolution, to obtain high-resolution, high-quality depth estimation results.

[0298] To verify the fusion effect of events and images and its advantages compared to different input combinations, different comparative experiments were set up. These included depth estimation using only the event stream (Event); depth estimation using all image data within a 128ms period (F(-n)); depth estimation using phase maps generated from all images acquired over 128ms (F(-p)); depth estimation using a single frame of white light projection image (F(-s)); and depth estimation using a dual-channel approach of a single frame image and the event stream (EF), where the dual-channel approach did not use the fusion component, and features were extracted from the event phase and image separately and then cascaded at the end of the FF-Net module before being input into the high-resolution reconstruction network module. The results are shown in Table 2.

[0299] Table 2 compares the accuracy of different inputs and fusion methods. / indicates that the evaluation metric for this scan cycle is the same as that for 128ms.

[0300]

[0301] First, the event-and-image fusion approach performs better than using images or events alone. Specifically, since the networks are trained in 128ms, using sequential images (F(-n)) or image-generated phases (F(-p)) yields good results in 128ms tests, but completely loses its effectiveness as the scan speed increases. Using single-frame images (F(-s)) eliminates the need to consider scan cycle length, resulting in clear contours in the depth estimation map, but with poor accuracy. Using events alone yields higher accuracy, but this accuracy decreases as the scan cycle shortens. Figure 17 It is evident that the event phase introduces more noise and phase delay, leading to a sharp increase in depth estimation error during high-speed scanning. The fused EF and MMFN networks effectively address these issues by reducing errors through the fusion of the advantages of different modalities. In contrast, MMFN, using an attention mechanism, achieves effective depth estimation within a 16ms scan cycle. Secondly, Table 3 shows that the event-based network model (Event) is more robust to object color, while depth estimation using only images (F(-s)) is more susceptible to interference. The event-image fusion (MMFN) method exhibits the smallest error and has a wider applicability to objects.

[0302] Table 3. Experimental results on the robustness of different inputs and blending methods in terms of color.

[0303]

[0304] Figure 19 The qualitative results for different inputs and fusion methods are shown in the figure. It can be seen that the depth estimation of F(-n) and F(-p) fails under high-speed scanning, while F(-s) is not affected by the scanning cycle, and its depth estimation results are stable and the edges are clear in all scanning cycles.

[0305] However, it performs poorly on color images, consistent with the results in Table 3. The Event algorithm based on event phase can perform depth estimation well, but due to the low resolution and high noise characteristics of event phase, the depth estimation is more blurred at the edges and cannot obtain effective output in the case of 16ms.

[0306] For the results of different fusion methods of EF and MMFN, it is difficult to actively observe and compare the differences with the true values. Figure 20 The graph shows the errors between the predicted and actual values ​​for several main methods, with more prominent colors indicating larger errors. As can be seen, Event prediction performs poorly at edges, while F(-s) has a larger overall error due to prediction of a single frame. MMFN shows the best performance, exhibiting the smallest errors at different scan speeds.

[0307] Due to the presence of significant random noise in the event stream and the delay in event stream timestamps during high-speed scanning, this application's embodiments added different types and proportions of noise to the original event stream to verify its noise resistance performance. Spatial noise was added randomly as a percentage of the total number of pixels. Temporal perturbation added random fluctuations to the timestamps of all events; the experimental scan period was 128ms, and the fluctuation ratio was added as a percentage of the period.

[0308] The experimental results of the noise robustness are as follows Figure 21A As shown, the comparison methods include Event, EF, and MMFN. The graphs show that MMFN has the best suppression effect for both types of noise, while Event, lacking single-frame image fusion, loses stability under high noise levels. Secondly, since event phase depends on timestamp calculation, temporal perturbations have a greater impact on the network than random noise. However, due to the introduction of high-resolution single-frame images, the proposed MMFN can effectively suppress timestamp perturbation problems.

[0309] The following section introduces the 3D reconstruction method of this application from the perspective of software module architecture:

[0310] The system flow of this application embodiment is divided into a network training phase and a network inference phase, and the system architecture is as follows: Figure 21B As shown, it includes a dynamic projection module, a delay calculation and median filtering module, a frame alignment module, and a structured light depth estimation network based on multi-mode input. This invention receives an event stream and an image to be processed, and generates a high frame rate, high resolution depth estimation map.

[0311] The embodiments of this application have a wide range of applications and can be used as key technologies in biometrics, gesture control, smart logistics, 3D video production, etc. Multiple modules in the multimodal input-based structured light depth estimation network can be applied to other image processing and imaging tasks based on multimodal input, such as the multi-level event phase denoising and fusion module (FF-Net) and the feature upsampling module (UP-Net). Furthermore, the embodiments of this application exhibit high robustness to real-world open scenes and can be adapted to different sensor models.

[0312] The following describes a three-dimensional reconstruction device provided in the embodiments of this application from the perspective of the device itself, referring to... Figure 22 , Figure 22 This is a schematic diagram of the structure of a three-dimensional reconstruction device provided in an embodiment of this application, such as... Figure 22 As shown, a three-dimensional reconstruction device 2200 provided in this application embodiment includes:

[0313] The acquisition module 2201 is used to acquire a first feature representation and a second feature representation, wherein the first feature representation is the feature representation corresponding to the pixel image acquired by the RGB camera, and the second feature representation is the feature representation corresponding to the event data acquired by the event camera; the pixel image and the event data are acquired for the same scene;

[0314] The specific description of the acquisition module 2201 can be found in the description of step 1001 in the above embodiments, and will not be repeated here.

[0315] The fusion module 2202 is used to fuse the first feature representation and the second feature representation to obtain a third feature representation;

[0316] The specific description of the fusion module 2202 can be found in the description of step 1002 in the above embodiments, and will not be repeated here.

[0317] The 3D reconstruction module 2203 is used to perform 3D reconstruction of the scene through a 3D reconstruction network based on the third feature representation to obtain the 3D reconstruction result.

[0318] The specific description of the three-dimensional reconstruction module 2203 can be found in the description of step 1003 in the above embodiment, and will not be repeated here.

[0319] In one possible implementation, the event data is obtained from multiple frames of event data collected by the event camera within a target time period, and the pixel image is obtained from an image collected by the RGB camera within the target time period.

[0320] In one possible implementation, the first feature representation and the second feature representation have the same size in the spatial dimension, and the features are fused into a splicing operation in the channel dimension.

[0321] In one possible implementation, the acquisition module is specifically used for:

[0322] Based on the third feature representation, target information is obtained through a neural network. The third feature representation is the feature representation corresponding to the pixel image. The target information represents the weight of noise contained in the position point in the fourth feature representation. The fourth feature representation is the feature representation corresponding to the event data.

[0323] The second feature representation is obtained based on the fourth feature representation and the target information.

[0324] In one possible implementation, the neural network is a first attention network, and the acquisition module is specifically used for:

[0325] Based on the third feature representation, target information is obtained through a first attention network, wherein the target information includes first attention information; the first attention information is the spatial correlation information of the pixel image.

[0326] The fourth feature representation is processed through a second attention network to obtain second attention information. This second attention information represents the spatial correlation of the event data. It is then fused with the first attention information to obtain third attention information.

[0327] The third attention information is used to obtain the second feature representation.

[0328] In one possible implementation, the attention fusion is a summation operation at corresponding positions.

[0329] In one possible implementation, the first attention information, the second attention information, and the third attention information are used to represent the correlation between the Q vector and the K vector;

[0330] The third attention information is used to obtain the first feature representation, including:

[0331] The third attention information is used to fuse with the K vector obtained by processing the fourth feature representation through the second attention network to obtain the first feature representation.

[0332] In one possible implementation, the device further includes:

[0333] The FFN module is used to process the first feature representation through a feedforward neural network (FFN) to obtain a processed first feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the processed first feature representation and the second feature representation; or,

[0334] The second feature representation is processed by a feedforward neural network (FFN) to obtain a processed second feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the first feature representation and the processed second feature representation.

[0335] In one possible implementation, the acquisition module is specifically used for:

[0336] Multiple attention modules are connected in series to process the feature representations corresponding to the pixel images and the event data to obtain the first feature representation and the second feature representation; wherein, the output of each attention module is used as the input of the next adjacent attention module and is processed based on the attention mechanism.

[0337] In one possible implementation, the output of each attention module includes a feature representation corresponding to the pixel image and a feature representation corresponding to the event data;

[0338] The first feature is obtained by fusing the feature representations corresponding to the pixel image obtained from some or all of the attention modules; or,

[0339] The second feature is obtained by fusing the feature representations corresponding to the event data obtained from some or all of the attention modules.

[0340] In one possible implementation, the resolution of the 3D reconstruction result is consistent with the resolution of the pixel image.

[0341] In one possible implementation, the 3D reconstruction result includes multiple location points and depth information corresponding to each location point.

[0342] In one possible implementation, the device further includes:

[0343] The phase calculation module is used to perform phase calculation on the event data acquired by the event camera to obtain the event phase;

[0344] The second feature is represented as the feature representation corresponding to the event phase.

[0345] In one possible implementation, the event phase includes multiple phase values ​​corresponding to the target location point, wherein different phase values ​​correspond to different timestamps; the device further includes:

[0346] A filtering module is used to perform median filtering on the event phase to obtain a processed event phase; the second feature representation is the feature representation corresponding to the processed event phase; wherein...

[0347] The processed event phase includes the target phase value corresponding to the target location point, which is obtained by median filtering of multiple phase values ​​corresponding to the target location point.

[0348] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 23 , Figure 23 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 2300 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The execution device 2300 implements... Figure 10The corresponding embodiment describes the function of the 3D reconstruction method. Specifically, the execution device 2300 includes: a receiver 2301, a transmitter 2302, a processor 2303, and a memory 2304 (wherein the execution device 2300 may have one or more processors 2303), wherein the processor 2303 may include an application processor 23031 and a communication processor 23032. In some embodiments of this application, the receiver 2301, transmitter 2302, processor 2303, and memory 2304 may be connected via a bus or other means.

[0349] Memory 2304 may include read-only memory and random access memory, and provides instructions and data to processor 2303. A portion of memory 2304 may also include non-volatile random access memory (NVRAM). Memory 2304 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0350] The processor 2303 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0351] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 2303. The processor 2303 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by the integrated logic circuitry in the hardware of the processor 2303 or by instructions in software form. The processor 2303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or microcontroller, a vision processing unit (VPU), a tensor processing unit (TPU), or other processors suitable for AI computation. It may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 2303 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 2304. Processor 2303 reads information from memory 2304 and, in conjunction with its hardware, completes steps 801 to 804 in the above embodiments.

[0352] Receiver 2301 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 2302 can be used to output digital or character information through the first interface; transmitter 2302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 2302 may also include a display device such as a display screen.

[0353] This application also provides a training device; please refer to [link / reference]. Figure 24 , Figure 24This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 2400 is implemented by one or more servers. The training device 2400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 2424 (e.g., one or more processors) and memory 2432, and one or more storage media 2430 (e.g., one or more mass storage devices) for storing application programs 2442 or data 2444. The memory 2432 and storage media 2430 can be temporary or persistent storage. The program stored in the storage media 2430 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 2424 may be configured to communicate with the storage media 2430 and execute the series of instruction operations in the storage media 2430 on the training device 2400.

[0354] The training device 2400 may also include one or more power supplies 2426, one or more wired or wireless network interfaces 2450, one or more input / output interfaces 2458; or, one or more operating systems 2441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0355] This application also provides a computer program product including computer-readable instructions, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0356] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0357] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the model training method described in the above embodiments, or to cause the chip within the training device to execute the steps related to model training in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0358] For details, please refer to Figure 25 , Figure 25 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 2500. The NPU 2500 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 2503, which is controlled by the controller 2504 to retrieve matrix data from the memory and perform multiplication operations.

[0359] In some implementations, the arithmetic circuit 2503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 2503 is a two-dimensional pulsating array. The arithmetic circuit 2503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2503 is a general-purpose matrix processor.

[0360] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2508.

[0361] Unified memory 2506 is used to store input and output data. Weight data is directly transferred to weight memory 2502 via Direct Memory Access Controller (DMAC) 2505. Input data is also transferred to unified memory 2506 via DMAC.

[0362] BIU stands for Bus Interface Unit 2510, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 2509.

[0363] The Bus Interface Unit (BIU) 2510 is used by the instruction fetch memory 2509 to fetch instructions from external memory, and also by the memory access controller 2505 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0364] The DMAC is mainly used to move input data from external memory DDR to unified memory 2506, or to weight data to weight memory 2502, or to input data to input memory 2501.

[0365] The vector computation unit 2507 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0366] In some implementations, the vector computation unit 2507 can store the processed output vector in the unified memory 2506. For example, the vector computation unit 2507 can apply a linear function, or a nonlinear function, to the output of the computation circuit 2503, such as linear interpolation of feature planes extracted by a convolutional layer, or, for example, a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 2507 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 2503, for example, for use in subsequent layers of the neural network.

[0367] The instruction fetch buffer 2509 connected to the controller 2504 is used to store the instructions used by the controller 2504.

[0368] Unified memory 2506, input memory 2501, weighted memory 2502, and instruction fetch memory 2509 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0369] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0370] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0371] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0372] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0373] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: A first feature representation and a second feature representation are obtained. The first feature representation is the feature representation corresponding to a pixel image captured by an RGB camera, and the second feature representation is the feature representation corresponding to event data captured by an event camera. The pixel image and the event data are captured for the same scene. The second feature representation is obtained based on a fourth feature representation and target information. The fourth feature representation is the feature representation corresponding to the event data. The target information is obtained by using a neural network based on the feature representation corresponding to the pixel image. The target information represents the weight of noise contained in the position points in the fourth feature representation. The neural network is a first attention network, and the target information includes first attention information. The first attention information is the spatial correlation information of the pixel image. The second feature representation is obtained based on third attention information, which is obtained by attention fusion of the first attention information and the second attention information. The second attention information is obtained by processing the fourth feature representation through a second attention network. The second attention information is the spatial correlation information of the event data. The first feature representation and the second feature representation are fused to obtain the third feature representation; Based on the third feature representation, the scene is reconstructed in three dimensions using a three-dimensional reconstruction network to obtain the three-dimensional reconstruction result.

2. The method according to claim 1, characterized in that, The event data is obtained from multiple frames of event data collected by the event camera within the target time period, and the pixel image is obtained from the image collected by the RGB camera within the target time period.

3. The method according to claim 1 or 2, characterized in that, The first feature representation and the second feature representation have the same size in the spatial dimension, and the features are fused into a splicing operation in the channel dimension.

4. The method according to claim 1 or 2, characterized in that, The attention fusion is an addition operation at corresponding positions.

5. The method according to claim 1 or 2, characterized in that, The first attention information, the second attention information, and the third attention information are used to represent the correlation between the Q vector and the K vector; The third attention information is used to obtain the first feature representation, including: The third attention information is used to fuse with the K vector obtained by processing the fourth feature representation through the second attention network to obtain the first feature representation.

6. The method according to claim 1 or 2, characterized in that, The method further includes: The first feature representation is processed by a feedforward neural network (FFN) to obtain a processed first feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the processed first feature representation and the second feature representation; or... The second feature representation is processed by a feedforward neural network (FFN) to obtain a processed second feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the first feature representation and the processed second feature representation.

7. The method according to claim 1 or 2, characterized in that, The acquisition of the first feature representation and the second feature representation includes: Multiple attention modules are connected in series to process the feature representations corresponding to the pixel images and the event data to obtain the first feature representation and the second feature representation; wherein, the output of each attention module is used as the input of the next adjacent attention module and is processed based on the attention mechanism.

8. The method according to claim 7, characterized in that, The output of each attention module includes the feature representation corresponding to the pixel image and the feature representation corresponding to the event data; The first feature is obtained by fusing the feature representations corresponding to the pixel image obtained from some or all of the attention modules; or, The second feature is obtained by fusing the feature representations corresponding to the event data obtained from some or all of the attention modules.

9. The method according to claim 1 or 2, characterized in that, The resolution of the 3D reconstruction result is consistent with the resolution of the pixel image.

10. The method according to claim 1 or 2, characterized in that, The 3D reconstruction results include multiple location points and the depth information corresponding to each location point.

11. The method according to claim 1 or 2, characterized in that, The method further includes: Phase calculation is performed on the event data collected by the event camera to obtain the event phase; The second feature representation is specifically the feature representation corresponding to the event phase.

12. The method according to claim 11, characterized in that, The event phase includes multiple phase values ​​corresponding to the target location point, and different phase values ​​correspond to different timestamps; the method further includes: The event phase is subjected to median filtering to obtain the processed event phase; the second feature representation is the feature representation corresponding to the processed event phase; wherein... The processed event phase includes the target phase value corresponding to the target location point, which is obtained by median filtering of multiple phase values ​​corresponding to the target location point.

13. A three-dimensional reconstruction device, characterized in that, The device includes: An acquisition module is used to acquire a first feature representation and a second feature representation. The first feature representation is the feature representation corresponding to a pixel image acquired by an RGB camera, and the second feature representation is the feature representation corresponding to event data acquired by an event camera. The pixel image and the event data are acquired for the same scene. The second feature representation is obtained based on a fourth feature representation and target information. The fourth feature representation is the feature representation corresponding to the event data. The target information is obtained by a neural network based on the feature representation corresponding to the pixel image, and the target information represents the weight of noise contained in the position points in the fourth feature representation. The neural network is a first attention network, and the target information includes first attention information. The first attention information is the spatial correlation information of the pixel image. The second feature representation is obtained based on third attention information, which is obtained by attention fusion of the first attention information and the second attention information. The second attention information is obtained by processing the fourth feature representation through a second attention network, and the second attention information is the spatial correlation information of the event data. The fusion module is used to fuse the first feature representation and the second feature representation to obtain a third feature representation; The 3D reconstruction module is used to perform 3D reconstruction of the scene based on the third feature representation through a 3D reconstruction network to obtain the 3D reconstruction result.

14. The apparatus according to claim 13, characterized in that, The event data is obtained from multiple frames of event data collected by the event camera within the target time period, and the pixel image is obtained from the image collected by the RGB camera within the target time period.

15. The apparatus according to claim 13 or 14, characterized in that, The first feature representation and the second feature representation have the same size in the spatial dimension, and the features are fused into a splicing operation in the channel dimension.

16. The apparatus according to claim 13 or 14, characterized in that, The attention fusion is an addition operation at corresponding positions.

17. The apparatus according to claim 16, characterized in that, The first attention information, the second attention information, and the third attention information are used to represent the correlation between the Q vector and the K vector; The third attention information is used to obtain the first feature representation, including: The third attention information is used to fuse with the K vector obtained by processing the fourth feature representation through the second attention network to obtain the first feature representation.

18. The apparatus according to claim 13 or 14, characterized in that, The device further includes: The FFN module is used to process the first feature representation through a feedforward neural network (FFN) to obtain a processed first feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the processed first feature representation and the second feature representation; or, The second feature representation is processed by a feedforward neural network (FFN) to obtain a processed second feature representation; the feature fusion of the first feature representation and the second feature representation includes: fusing the first feature representation and the processed second feature representation.

19. The apparatus according to claim 13 or 14, characterized in that, The acquisition module is specifically used for: Multiple attention modules are connected in series to process the feature representations corresponding to the pixel images and the event data to obtain the first feature representation and the second feature representation; wherein, the output of each attention module is used as the input of the next adjacent attention module and is processed based on the attention mechanism.

20. The apparatus according to claim 19, characterized in that, The output of each attention module includes the feature representation corresponding to the pixel image and the feature representation corresponding to the event data; The first feature is obtained by fusing the feature representations corresponding to the pixel image obtained from some or all of the attention modules; or, The second feature is obtained by fusing the feature representations corresponding to the event data obtained from some or all of the attention modules.

21. The apparatus according to claim 13 or 14, characterized in that, The resolution of the 3D reconstruction result is consistent with the resolution of the pixel image.

22. The apparatus according to claim 13 or 14, characterized in that, The 3D reconstruction results include multiple location points and the depth information corresponding to each location point.

23. The apparatus according to claim 13 or 14, characterized in that, The device further includes: The phase calculation module is used to perform phase calculation on the event data acquired by the event camera to obtain the event phase; The second feature is represented as the feature representation corresponding to the event phase.

24. The apparatus according to claim 23, characterized in that, The event phase includes multiple phase values ​​corresponding to the target location point, and different phase values ​​correspond to different timestamps; the device further includes: A filtering module is used to perform median filtering on the event phase to obtain a processed event phase; the second feature representation is the feature representation corresponding to the processed event phase; wherein... The processed event phase includes the target phase value corresponding to the target location point, which is obtained by median filtering of multiple phase values ​​corresponding to the target location point.

25. A system, characterized in that, Includes RGB cameras, event cameras, and devices as described in any one of claims 13 to 24.

26. A computing device, characterized in that, The computing device includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code and execute the method as described in any one of claims 1 to 12.

27. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 12.

28. A computer program product, comprising code, characterized in that, When the code is executed, it is used to implement the method as described in any one of claims 1 to 12.

Images