A target detection method and electronic device

Abstract

The application discloses a target detection method and an electronic device. The target detection method comprises the following steps: obtaining a first feature set according to input data; obtaining a second feature set of the first feature set in a BEV space according to the first feature set; obtaining target feature data according to the second feature set; and obtaining a target detection result according to the target feature data, so that the target detection based on the BEV does not need to add additional information to assist in solving and is not sensitive to camera internal parameters.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a target detection method and electronic device. Background Technology

[0002] Bird's Eye View (BEV) representation, a popular perception paradigm in the field of autonomous driving, maps sensor data captured by multiple onboard cameras from image space to a unified BEV space. Compared to the independent perspective view (PV) spaces of individual sensors, this unified BEV space facilitates cross-camera, multimodal, and temporal fusion. Unlike post-fusion and pre-fusion techniques, model training in the BEV space can achieve end-to-end optimization. However, BEV-based object detection typically requires additional information to aid in the solution and is sensitive to camera intrinsic parameters. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a target detection method and an electronic device, which enable BEV-based target detection to be solved without adding additional information and is insensitive to camera intrinsic parameters.

[0004] In a first aspect, embodiments of the present invention provide a target detection method, the method comprising:

[0005] Based on the input data, the first feature set is obtained;

[0006] Based on the first feature set, a second feature set of the first feature set in the BEV space is obtained;

[0007] The target feature data is obtained based on the second feature set;

[0008] Based on the target feature data, the target detection result is obtained. In the process of BEV-based target detection, no additional information is needed to assist in the solution. Moreover, the training and inference are independent of the camera intrinsic parameters, are not sensitive to the camera intrinsic parameters, and the model can be transferred to different vehicle models.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, before obtaining the first feature set based on the input data, the method further includes: receiving the input data.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the input data includes multiple images captured by multiple cameras of a mobile device.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the first feature set includes the first feature data of each image in the input data.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the first feature set based on the input data includes:

[0013] The first feature set is obtained by extracting the first feature data from the input data through the backbone network.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, obtaining a second feature set of the first feature set in the BEV space based on the first feature set includes:

[0015] The second feature set is obtained by extracting the second feature data of the first feature data in the BEV space from the first feature set.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the target detection result based on the target feature data includes:

[0017] The target is extracted from the target feature data to obtain the target detection result.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the step of extracting the second feature data of the first feature data in the BEV space from the first feature set includes:

[0019] Align historical BEV query vectors with the current image's BEV query vector based on the motion trajectory of the mobile device;

[0020] By fusing the aligned historical BEV query vector with the current image's BEV query vector, a query vector for BEV features is obtained.

[0021] Based on the query vector of the BEV features, the predicted height of the grid in the BEV space is obtained;

[0022] A reference point is obtained by projecting the predicted height of the grid in the BEV space and the spatial position of the grid onto the current image using the camera's intrinsic and extrinsic parameters;

[0023] The second feature data of the image is obtained based on the reference point, the query vector of the BEV feature, and the first feature data corresponding to the current image in the first feature set. This embodiment of the invention provides explicit modeling of height, improving detection accuracy.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the target feature data based on the second feature set includes:

[0025] The target feature data is obtained by passing the second feature set through multiple cascaded encoder layers.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the target feature data by passing the second feature set through multiple cascaded encoder layers includes:

[0027] The second feature set is passed through the first encoder layer in the plurality of cascaded encoder layers to obtain the first output data;

[0028] The first output data is passed through multiple cascaded encoder layers, excluding the first encoder layer, to obtain the target feature data.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the first output data by passing the second feature set through the first encoder layer of the plurality of cascaded encoder layers includes:

[0030] The second feature data in the second feature set are fused to obtain the third feature data;

[0031] The fourth feature data is obtained by filtering the third feature data using background features.

[0032] The fifth feature data is obtained by passing the sum of the fourth feature data and the query vector of the BEV feature through the LayerNorm layer;

[0033] The sixth feature data is obtained by passing the fifth feature data through multiple MLP network layers;

[0034] The first output data is obtained by passing the sum of the fifth and sixth feature data through the LayerNorm layer. This embodiment of the invention models high uncertainty and filters BEV features from the background during height prediction, thereby improving detection accuracy.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, the mobile device includes a vehicle.

[0036] In conjunction with the first aspect, in some implementations of the first aspect, the target includes one or any combination of a person, a vehicle, and an obstacle.

[0037] Secondly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and when the processor executes the program instructions, the electronic device performs the steps of the method described above.

[0038] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when the program requests to be run by a computer, cause the computer to perform the method described above.

[0039] Fourthly, embodiments of the present invention provide a computer program product comprising instructions that, when the computer program product is run on a computer or any at least one processor, cause the computer to perform the functions / steps as described above.

[0040] In the target detection method and electronic device provided in the embodiments of the present invention, the target detection method includes: obtaining a first feature set based on input data; obtaining a second feature set of the first feature set in the BEV space based on the first feature set; obtaining target feature data based on the second feature set; and obtaining a target detection result based on the target feature data, so that BEV-based target detection does not require adding additional information to assist in the solution and is insensitive to camera intrinsic parameters. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0042] Figure 2 This is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention;

[0043] Figure 3 The flowchart for the LSS method;

[0044] Figure 4 A flowchart of the BEVFormer method;

[0045] Figure 5 An architecture diagram of a target detection system provided in an embodiment of the present invention;

[0046] Figure 6 for Figure 5 Schematic diagram of the medium-altitude predictor unit;

[0047] Figure 7 A flowchart of a target detection method provided in an embodiment of the present invention;

[0048] Figure 8 This is a flowchart illustrating the process of extracting second feature data from the first feature set within the BEV space in an embodiment of the present invention.

[0049] Figure 9 for Figure 7 The flowchart for obtaining target feature data based on the second feature set;

[0050] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0052] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0053] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0055] Figure 1 A schematic diagram of the structure of the electronic device 100 is shown.

[0056] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0057] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0058] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0059] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0060] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0061] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0062] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0063] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0064] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0065] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0066] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0067] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0068] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0069] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0071] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0072] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0073] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0074] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0075] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0076] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0077] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0078] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0079] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0080] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0081] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0082] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0083] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0084] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0085] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0086] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0087] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0088] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0089] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0090] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0091] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0092] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to illustrate the software structure of electronic device 100.

[0093] Figure 2 This is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0094] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0095] The application layer can include a series of application packages.

[0096] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0097] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0098] like Figure 2 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0099] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0100] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0101] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0102] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0103] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0104] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0105] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0106] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0107] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0108] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0109] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0110] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0111] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0112] A 2D graphics engine is a graphics engine for 2D drawing.

[0113] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0114] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 100.

[0115] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0116] BEV-based target detection methods typically include two categories: Category I target detection methods and Category II target detection methods.

[0117] The first type of object detection method models the depth information in the image space, projects image features into 3D space through pseudo-point clouds, and then performs pooling operations on the height dimension to transform it into BEV space. Examples include the LSS method (from the paper: Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (ECCV 2020)) and the BEVDepth method (from the paper: BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection (AAAI2023)). Figure 3 The flowchart of the LSS method is as follows: Figure 3 As shown, the LSS method includes:

[0118] Step 21: Use a CNN network to extract the F1 curve of the input features from each camera image. i .

[0119] Step 22, based on F1 i Predict the depth distribution D of each pixel i .

[0120] Step 23, set F1 i With D i Multiplication expands each pixel from a C-dimensional vector of image features into a C×D two-dimensional matrix, which is equivalent to assigning different weights to the image features according to the depth distribution, resulting in a pseudo-point cloud.

[0121] Step 24: Based on camera intrinsic parameters, image coordinates, and depth information, convert the 2.5D pseudo-point cloud to 3D to obtain F1. i 3D .

[0122] Step 25: Based on the camera's external parameters, adjust the F1 value of each camera. i 3D All were converted to the vehicle's coordinate system, and then the features from different cameras were fused.

[0123] Step 26: Obtain the BEV feature by summing and removing the height dimension.

[0124] However, the LSS method implicitly models depth, meaning there is no supervision of depth information, and therefore cannot guarantee the accuracy of the learning.

[0125] The BEVDepth method is similar to the LSS method, except that in step 22 above, the BEVDepth method adds a prediction depth subnetwork to the network to explicitly learn depth information. The BEVDepth method supervises the learning of depth information and has high accuracy, but its training depends on the depth information provided by LiDAR. Furthermore, because depth learning is related to camera intrinsics, learning depth features can be difficult if the intrinsics of different cameras are different.

[0126] The second type of object detection method models the height, mapping it from 3D space to 2D and extracting features at the corresponding location. For example, the BEVFormer method (from the paper: BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (ECCV 2022)). Figure 4 The flowchart for the BEVFormer method is as follows: Figure 4 As shown, the BEVFormer methods include:

[0127] Step 31: Use a CNN network to extract the input features F2 of each camera image. i .

[0128] Step 32: Assuming each cell in the BEV space has a fixed height of N reference points, for the BEV query Q (BEV query vector Q) and all F2... i Perform a deformable cross attention operation and sum the features extracted from different reference points to obtain the feature sum.

[0129] Among them, the Deformable cross attention operation involves mapping 3D reference points to 2D image space through camera intrinsic and extrinsic parameters.

[0130] Step 33: Combine the features with transformer layer operations such as Add, Norm, and Feed Forward to obtain BEV features.

[0131] Here, Add stands for Residual Connection, which is used to prevent network degradation.

[0132] Norm stands for Layer Normalization, which is used to normalize the activation values ​​of each layer.

[0133] Here, Feed Forward represents a Multi-Layer Perceptron (MLP) network, and its result can be understood as a projection transformation of the MLP input.

[0134] In step 32 of the BEVFormer method, the reference height is assumed to be fixed, and then different weights are assigned to the height through the attention mechanism. This is equivalent to implicitly modeling the height, which cannot guarantee the accuracy of the learning.

[0135] To address the aforementioned technical problems with BEV-based target detection, this invention provides a target detection system. Figure 5 This is an architecture diagram of a target detection system provided in an embodiment of the present invention.

[0136] like Figure 5 As shown, the target detection system 400 includes a backbone network unit 410, an encoder unit 420, and a decoder unit 430. The target detection system 400 can be configured in an electronic device. The hardware and software structures of the electronic device provided in this embodiment of the invention can be found in [reference needed]. Figure 1 and Figure 2 The relevant description of electronic device 100 in the text.

[0137] The backbone network unit 410 is used to receive input data.

[0138] For example, the input data includes multiple images captured by multiple cameras on a mobile device.

[0139] For example, mobile devices include vehicles.

[0140] The backbone network unit 410 is also used to obtain a first feature set based on the input data and send the first feature set to the encoder unit 420.

[0141] For example, the first feature set includes first feature data for each image in the input data. The backbone network unit 410 is specifically used to: extract the first feature data from the input data through the backbone network to obtain the first feature set. For example, the backbone network unit 410 extracts the first feature data for each image I in the input data. i First feature data F3 i .

[0142] For example, the specific form of the backbone network can be various mainstream backbone networks, such as ResNet (residual network), VoVNet or ViT, etc., and this invention does not make any specific limitation.

[0143] The encoder unit 420 is used to obtain a second feature set of the first feature set in the BEV space based on the first feature set; obtain target feature data based on the second feature set; and send the target feature data to the decoder unit 430.

[0144] like Figure 5 As shown, the encoder unit 420 includes N encoder layers connected in series. Each encoder layer includes: a temporal fusion unit 421, a height predictor unit 422, and a spatial cross-attention unit 423.

[0145] In some possible embodiments, the encoder unit 420 is specifically configured to: align the historical BEV query vector with the current image's BEV query vector based on the motion trajectory of the mobile device; fuse the aligned historical BEV query vector with the current image's BEV query vector to obtain a query vector for BEV features; obtain the predicted height of the grid in the BEV space based on the query vector of the BEV features; project the predicted height of the grid in the BEV space and the grid's spatial position onto the current image using camera intrinsic and extrinsic parameters to obtain a reference point; and obtain the second feature data of the image based on the reference point, the query vector of the BEV features, and the first feature data corresponding to the current image in the first feature set.

[0146] Specifically, the temporal fusion unit 421 aligns the historical BEV query vector with the current image's BEV query vector based on the mobile device's motion trajectory. By fusing the aligned historical BEV query vector with the current image's BEV query vector, a query vector for BEV features is obtained, and this query vector is sent to the height predictor unit 422, the spatial cross-attention unit 423, the first residual structure, and the layer normalization unit 425. The present invention does not specifically limit the fusion method; it can be a deformable attention method.

[0147] Specifically, the height predictor unit 422 is used to obtain the predicted height of the grid in the BEV space based on the query vector of the BEV feature, and send the predicted height of the grid in the BEV space to the spatial cross-attention unit 423.

[0148] For example, Figure 6 for Figure 5 A schematic diagram of the medium-altitude predictor unit, as shown below. Figure 6As shown, the height predictor unit 422 combines the query vector of the BEV features from the temporal fusion unit 421 with the height features (height embedding) H from the previous encoder layer. i-1 Perform concatenation, then pass through an MLP layer and combine with the height feature H. i-1 Adding them together yields the height feature H. i The high-resolution feature H is sent to the next encoder layer. i It is also used to predict the height, uncertainty, and background / foreground classification of each grid in the BEV space. These three prediction results are used to calculate the loss function for network training.

[0149] Specifically, the spatial cross-attention unit 423 is used to receive the predicted height of the grid in the BEV space from the height predictor unit 422. The predicted height of the grid in the BEV space and the spatial position of the grid are projected onto the current image through the camera's intrinsic and extrinsic parameters to obtain a reference point. Based on the reference point, the query vector of the BEV feature, and the first feature data corresponding to the current image in the first feature set, the second feature data of the image is obtained.

[0150] For example, the spatial cross-attention unit 423 obtains the second feature data by performing deformable attention operations on the reference point, the query vector of the BEV feature, and the first feature data.

[0151] The spatial cross-attention unit 423 uses the predicted height of the grid in the BEV space and its spatial position within the grid as reference points. These are projected onto the image space based on camera intrinsic and extrinsic parameters. Then, based on the Deformable Attention method, features are extracted near the projection points of the reference points, which are used as the second feature data. Here, the reference points are coordinates in the 3D world, and each grid has 2D coordinates in the BEV plane, which, combined with the predicted height, form 3D coordinates. The projection point is the point on the image projected from the aforementioned 3D world reference points through camera intrinsic and extrinsic parameters. The location of the extracted features on the image depends on the projection point of the deformable attention operation and the sampling offset (obtained by transforming the query vector of the BEV features). That is, the projection point plus the sampling offset yields the image feature location to be processed.

[0152] like Figure 5 As shown, the encoder layer also includes a query mask unit 424, a first residual structure and a layer normalization (add&norm) unit 425, a feed forward unit 426, and a second residual structure and a layer normalization (add&norm) unit 427.

[0153] In some possible embodiments, encoder unit 420 is further configured to obtain target feature data by passing the second feature set through multiple cascaded encoder layers (N cascaded encoder layers). Specifically, encoder unit 420 is configured to pass the second feature set through the first encoder layer of the multiple cascaded encoder layers to obtain first output data; and pass the first output data through multiple cascaded encoder layers other than the first encoder layer to obtain target feature data.

[0154] In some possible embodiments, the encoder unit 420 is specifically used to fuse the second feature data in the second feature set to obtain the third feature data; to obtain the fourth feature data by performing background feature filtering on the third feature data; to obtain the fifth feature data by passing the sum of the fourth feature data and the query vector of the BEV feature through the LayerNorm layer; to obtain the sixth feature data by passing the fifth feature data through multiple MLP network layers; and to obtain the first output data by passing the sum of the fifth feature data and the sixth feature data through the LayerNorm layer.

[0155] Specifically, the spatial cross-attention unit 423 is also used to fuse the second feature data in the second feature set to obtain the third feature data, and send the third feature data to the query mask unit 424. The present invention does not specifically limit the fusion method; it can be a deformable attention method.

[0156] Specifically, the query mask unit 424 is used to obtain the fourth feature data by performing background feature filtering on the third feature data, and then send the fourth feature data to the first residual structure and the layer normalization unit 425.

[0157] In this embodiment of the invention, considering that the background region (the part without the target) has no supervision information and the predicted height result is not interpretable, the query mask unit 424 filters out the background features, that is, the features of the background region are not updated in the first residual structure and the layer normalization unit 425.

[0158] In this embodiment of the invention, the height uncertainty is modeled by the height predictor unit 422, and the BEV features of the background are filtered by the query mask unit 424 during the height prediction process to improve the detection accuracy.

[0159] Specifically, the first residual structure and layer normalization unit 425 are used to obtain the fifth feature data by passing the sum of the query vectors of the fourth feature data and the BEV feature through the LayerNorm layer, and send the fifth feature data to the feedforward unit 426 and the second residual structure and layer normalization unit 427.

[0160] The first residual structure and the layer normalization unit 425 add the query vectors of the fourth feature data and the BEV feature, and then pass the result through the LayerNorm layer to obtain the fifth feature data.

[0161] Specifically, the feedforward unit 426 is used to obtain the sixth feature data by passing the fifth feature data through multiple MLP network layers, and send the sixth feature data to the second residual structure and the layer normalization unit 427.

[0162] The feedforward unit 426 consists of several MLP network layers.

[0163] Specifically, the second residual structure and the layer normalization unit 427 are used to obtain the first output data by passing the sum of the fifth feature data and the sixth feature data through the LayerNorm layer.

[0164] The second residual structure and the layer normalization unit 427 add the fifth feature data and the sixth feature data, and then pass the result through the LayerNorm layer to obtain the first output data.

[0165] The decoder unit 430 is used to obtain the target detection result based on the target feature data.

[0166] In some possible embodiments, the decoder unit 430 is used to obtain the target detection result by extracting the target from the target feature data.

[0167] The target feature data output by encoder unit 420 includes image features from the BEV perspective. This can be understood as the target and background having different representations, resulting in strong distinguishability.

[0168] For example, the decoder unit 430 extracts the target to be detected by performing a cross-attention operation on the target feature data and the learnable query vector.

[0169] For example, the target may include one or any combination of people, vehicles, and obstacles.

[0170] The target detection system provided in this embodiment of the invention can extract "targets" such as vehicles, people, and obstacles from images input from multiple cameras.

[0171] In summary, the target detection system provided in this embodiment of the invention does not require additional information to assist in the solution process during BEV-based target detection. Furthermore, its training and inference are independent of camera intrinsic parameters, making it insensitive to camera intrinsic parameters and allowing for model transfer to different vehicle types. Moreover, by modeling height uncertainty and filtering BEV features from the background during height prediction, it improves detection accuracy.

[0172] To verify the beneficial effects of the target detection system 400 provided in this embodiment of the invention, the method of BEV-based target detection using the target detection system 400 will be referred to as BEVHeight below.

[0173] BEVHeight replaces the predicted height with the true value. As shown in Table 1, on the nuScenes validation set, the mAP of the BEVHeight method is 37.3% higher than that of BEVFormer, indicating that the accuracy of BEVHeight's height prediction has a significant impact on the improvement of accuracy.

[0174] Table 1. Height Prediction Results for BEVHeight and BEVFormer

[0175] Model condition DNS↑ mAP↑ BEVFormer - 0.517 0.416 BEVHeight high 0.725 0.789

[0176] BEVHeight, which explicitly predicts height, as shown in Table 2, improves both NDS and mAP metrics on the nuScenes validation set by 0.5% compared to BEVFormer, which implicitly predicts height.

[0177] Table 2. Height Prediction Results for BEVHeight and BEVFormer

[0178] Model high DNS↑ mAP↑ BEVFormer implicit 0.517 0.416 BEVHeight Explicit 0.522 0.421

[0179] Network structure ablation experiments were conducted using BEVHeight (nuScenesval dataset, SR: self-recursive way of predicting heights, Unc.M: uncertainty-based mask, Seg.M: segmentation-based mask). As shown in Table 3, the three techniques of autoregressive prediction height, uncertainty mask, and segmentation mask all improved the mAP index.

[0180] Table 3 shows the results of network structure ablation experiments using BEVHeight.

[0181] SR Unc.M Seg.M DNS↑ mAP↑ 0.522 0.421 √ 0.525 0.422 √ √ 0.528 0.424 √ √ 0.527 0.427

[0182] BEVHeight exhibits high robustness in modeling. Rearview cameras have different camera intrinsics, as shown in Table 4. When using rearview cameras for detection, height modeling is more accurate and robust to camera intrinsics than depth modeling, which is beneficial for transferring to vehicle models with different sensor configurations.

[0183] Table 4 shows the results of height modeling and depth modeling using the rear-view camera.

[0184]

[0185] As shown in Table 5, the nuScenes val set results show that BEVHeight has a 1.3% higher mAP than BEVFormer.

[0186] Table 5 Results of nuScenes val set

[0187] Model backbone network DNS↑ mAP↑ BEVFormer R101-DCN 0.517 0.416 BEVHeight R101-DCN 0.532 0.429

[0188] Based on the target detection system 400 described above, this embodiment of the invention provides a target detection method.

[0189] Figure 7 This is a flowchart illustrating a target detection method provided in an embodiment of the present invention. Figure 7 As shown, the target detection method includes:

[0190] Step 502: Obtain the first feature set based on the input data.

[0191] For example, the input data includes multiple images captured by multiple cameras on a mobile device.

[0192] For example, the first feature set includes first feature data for each image in the input data.

[0193] For example, mobile devices include vehicles.

[0194] In some possible embodiments, step 502 specifically includes: extracting first feature data from the input data through the backbone network to obtain a first feature set.

[0195] For example, the specific form of the backbone network can be various mainstream backbone networks, such as ResNet (Residual Network), VoVNet, or ViT, etc., and this invention does not impose a specific limitation. For example, the backbone network can be used to extract each image I from the input data. i First feature data F3 i .

[0196] Optionally, before step 502, the target detection method further includes step 501.

[0197] Step 501: Receive input data.

[0198] Step 504: Based on the first feature set, obtain the second feature set of the first feature set in the BEV space.

[0199] In some possible embodiments, step 504 specifically includes: obtaining the second feature set by extracting the second feature data of the first feature data in the BEV space from the first feature set.

[0200] In some possible embodiments, such as Figure 8 As shown, by extracting the second feature data of the first feature data in the first feature set within the BEV space, specifically including:

[0201] Step 504a: Align the historical BEV query vector with the current image's BEV query vector based on the motion trajectory of the mobile device.

[0202] Step 504b: By fusing the aligned historical BEV query vector with the current image's BEV query vector, a query vector for BEV features is obtained.

[0203] In this invention, the fusion method is not specifically limited and can be a deformable attention method.

[0204] Step 504c: Based on the query vector of the BEV features, obtain the predicted height of the grid in the BEV space.

[0205] For example, the query vector for BEV features is combined with the height features (H) from the previous encoder layer. i-1 Perform concatenation, then pass through a Multi-Layer Perceptron (MLP) layer and combine with the height feature H. i-1 Adding them together yields the height feature H. i The high-resolution feature H is sent to the next encoder layer. i It is also used to predict the height, uncertainty, and background / foreground classification of each grid in the BEV space. These three prediction results are used to calculate the loss function for network training.

[0206] Step 504d: The predicted height and spatial position of the grid in the BEV space are projected onto the current image using the camera's intrinsic and extrinsic parameters to obtain the reference point.

[0207] Step 504e: Based on the reference point, the query vector of the BEV feature, and the first feature data corresponding to the current image in the first feature set, obtain the second feature data of the current image.

[0208] For example, the second feature data is obtained by performing deformable attention on the reference point, the query vector of the BEV feature, and the first feature data.

[0209] For example, the predicted height of a grid in the BEV space and its spatial position within that grid are used as reference points. These are projected onto the image space based on camera intrinsic and extrinsic parameters. Then, features are extracted near the projection point of the reference point using the Deformable Attention method, and these features are used as the second feature data. Here, the reference point represents the coordinates in the 3D world, and each grid has 2D coordinates in the BEV plane. Adding the predicted height forms the 3D coordinates. The projection point is the point on the image projected from the aforementioned 3D world reference point using camera intrinsic and extrinsic parameters. The location of the extracted features on the image depends on the projection point of the deformable attention operation and the sampling offset (obtained by transforming the query vector of the BEV features). That is, the projection point plus the sampling offset yields the image feature location to be processed.

[0210] Step 506: Obtain the target feature data based on the second feature set.

[0211] In some possible embodiments, step 506 specifically includes: obtaining target feature data by passing the second feature set through multiple cascaded encoder layers. Further, step 506 specifically includes: passing the second feature set through the first encoder layer of the multiple cascaded encoder layers to obtain first output data; and passing the first output data through multiple cascaded encoder layers other than the first encoder layer to obtain target feature data.

[0212] In some possible embodiments, such as Figure 9 As shown, obtaining the first output data by passing the second feature set through the first encoder layer in a series of encoder layers specifically includes:

[0213] Step 506a: Fuse the second feature data in the second feature set to obtain the third feature data.

[0214] In this invention, the fusion method is not specifically limited and can be a deformable attention method.

[0215] Step 506b: Obtain the fourth feature data by performing background feature filtering on the third feature data.

[0216] In this embodiment of the invention, considering that the background region (the part without the target) has no supervision information and the predicted height result is not interpretable, the background features are filtered out, that is, the features of the background region are not updated in step 506c.

[0217] In this embodiment of the invention, by modeling the high degree of uncertainty, the BEV features of the background are filtered out during the height prediction process, thereby improving the detection accuracy.

[0218] Step 506c: Obtain the fifth feature data by passing the sum of the query vectors of the fourth feature data and the BEV feature through the LayerNorm layer;

[0219] Step 506d: Obtain the sixth feature data by passing the fifth feature data through multiple MLP network layers;

[0220] Step 506e: Obtain the first output data by passing the sum of the fifth feature data and the sixth feature data through the LayerNorm layer.

[0221] Step 508: Obtain the target detection result based on the target feature data.

[0222] In some possible embodiments, step 508 specifically includes: extracting the target from the target feature data to obtain the target detection result.

[0223] The target includes one or any combination of people, vehicles, and obstacles.

[0224] The target detection method provided in this invention can extract "targets" such as vehicles, people, and obstacles from images input from multiple cameras.

[0225] The target detection method provided in this embodiment of the invention includes: obtaining a first feature set based on input data; obtaining a second feature set of the first feature set in the BEV space based on the first feature set; obtaining target feature data based on the second feature set; and obtaining target detection results based on the target feature data, so that BEV-based target detection does not require adding additional information to assist in the solution and is insensitive to camera intrinsic parameters.

[0226] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. It should be understood that the electronic device 600 is capable of performing each step in the above-described target detection method. To avoid repetition, details are not provided here. The electronic device 600 includes a processing unit 601 and a receiving unit 602.

[0227] The processing unit 601 is configured to obtain a first feature set based on input data; obtain a second feature set of the first feature set in the BEV space based on the first feature set; obtain target feature data based on the second feature set; and obtain target detection results based on the target feature data.

[0228] Optionally, before the processing unit 601 obtains the first feature set based on the input data, the receiving unit 602 is used to receive the input data.

[0229] Optionally, the input data includes multiple images captured by multiple cameras on a mobile device.

[0230] Optionally, the first feature set includes the first feature data of each image in the input data.

[0231] Optionally, the processing unit 601 is specifically used to extract the first feature data of the input data through the backbone network to obtain the first feature set.

[0232] Optionally, the processing unit 601 is specifically used to obtain the second feature set by extracting the second feature data of the first feature data in the BEV space from the first feature data in the first feature set.

[0233] Alternatively, ...

[0234] Optionally, the processing unit 601 is specifically used to obtain the target detection result by extracting the target from the target feature data.

[0235] Optionally, the processing unit 601 is specifically configured to: align the historical BEV query vector with the current image's BEV query vector based on the motion trajectory of the mobile device; fuse the aligned historical BEV query vector with the current image's BEV query vector to obtain a query vector for BEV features; obtain the predicted height of the grid in the BEV space based on the query vector of the BEV features; project the predicted height of the grid in the BEV space and the spatial position of the grid onto the current image using camera intrinsic and extrinsic parameters to obtain a reference point; and obtain the second feature data of the image based on the reference point, the query vector of the BEV features, and the first feature data corresponding to the current image in the first feature set.

[0236] Optionally, the processing unit 601 is specifically used to obtain the target feature data by passing the second feature set through multiple cascaded encoder layers.

[0237] Optionally, the processing unit 601 is specifically configured to pass the second feature set through the first encoder layer of the plurality of serial encoder layers to obtain first output data; and pass the first output data through a plurality of serial encoder layers other than the first encoder layer to obtain the target feature data.

[0238] Optionally, the processing unit 601 is specifically used to fuse the second feature data in the second feature set to obtain the third feature data; to obtain the fourth feature data by performing background feature filtering on the third feature data; to obtain the fifth feature data by passing the sum of the fourth feature data and the query vector of the BEV feature through a LayerNorm layer; to obtain the sixth feature data by passing the fifth feature data through multiple MLP network layers; and to obtain the first output data by passing the sum of the fifth feature data and the sixth feature data through the LayerNorm layer.

[0239] Optionally, the mobile device includes a vehicle.

[0240] Optionally, the target may include one or any combination of people, vehicles, and obstacles.

[0241] It should be understood that the electronic device 600 described herein is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, without specific limitation. For example, a "unit" can be a software program, hardware circuitry, or a combination of both that implements the functions described above. The hardware circuitry may include application-specific integrated circuits (ASICs), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components supporting the described functions.

[0242] Therefore, the units of the various examples described in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the present invention.

[0243] This application provides an electronic device, which can be a terminal device or a circuit device built into the terminal device. This electronic device can be used to perform the functions / steps described in the method embodiments above.

[0244] This application provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the functions / steps described in the above method embodiments.

[0245] This application also provides a computer program product containing instructions that, when run on a computer or any at least one processor, cause the computer to perform the functions / steps described in the above method embodiments.

[0246] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0247] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0248] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0249] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0250] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A target detection method, characterized in that, The method includes: Based on the input data, the first feature set is obtained; Based on the first feature set, a second feature set of the first feature set in the BEV space is obtained; The target feature data is obtained based on the second feature set; Based on the target feature data, the target detection result is obtained; The step of obtaining a second feature set of the first feature set in the BEV space based on the first feature set includes: Align historical BEV query vectors with the current image's BEV query vector based on the motion trajectory of the mobile device; By fusing the aligned historical BEV query vector with the current image's BEV query vector, a query vector for BEV features is obtained. Based on the query vector of the BEV features, uncertainty modeling is performed on the height of the grid in the BEV space, and the BEV features of the background are filtered out during the height prediction process to obtain the predicted height of the grid in the BEV space. A reference point is obtained by projecting the predicted height of the grid in the BEV space and the spatial position of the grid onto the current image using the camera's intrinsic and extrinsic parameters; The second feature data of the image is obtained based on the reference point, the query vector of the BEV feature, and the first feature data corresponding to the current image in the first feature set.

2. The method according to claim 1, characterized in that, Before obtaining the first feature set based on the input data, the method further includes: receiving the input data.

3. The method according to claim 1 or 2, characterized in that, The input data includes multiple images captured by multiple cameras on a mobile device.

4. The method according to claim 3, characterized in that, The first feature set includes the first feature data of each image in the input data.

5. The method according to any one of claims 1-2 and 4, characterized in that, The first feature set obtained based on the input data includes: The first feature set is obtained by extracting the first feature data from the input data through the backbone network.

6. The method according to claim 3, characterized in that, The first feature set obtained based on the input data includes: The first feature set is obtained by extracting the first feature data from the input data through the backbone network.

7. The method according to any one of claims 1-2, 4, and 6, characterized in that, The step of obtaining the target detection result based on the target feature data includes: The target is extracted from the target feature data to obtain the target detection result.

8. The method according to claim 3, characterized in that, The step of obtaining the target detection result based on the target feature data includes: The target is extracted from the target feature data to obtain the target detection result.

9. The method according to claim 5, characterized in that, The step of obtaining the target detection result based on the target feature data includes: The target is extracted from the target feature data to obtain the target detection result.

10. The method according to claim 1, characterized in that, The step of obtaining target feature data based on the second feature set includes: The target feature data is obtained by passing the second feature set through multiple cascaded encoder layers.

11. The method according to claim 10, characterized in that, The step of passing the second feature set through multiple cascaded encoder layers to obtain the target feature data includes: The second feature set is passed through the first encoder layer in the plurality of cascaded encoder layers to obtain the first output data; The first output data is passed through multiple cascaded encoder layers, excluding the first encoder layer, to obtain the target feature data.

12. The method according to claim 11, characterized in that, The step of passing the second feature set through the first encoder layer of the plurality of cascaded encoder layers to obtain the first output data includes: The second feature data in the second feature set are fused to obtain the third feature data; The fourth feature data is obtained by filtering the third feature data using background features. The fifth feature data is obtained by passing the sum of the fourth feature data and the query vector of the BEV feature through the LayerNorm layer; The sixth feature data is obtained by passing the fifth feature data through multiple MLP network layers; The first output data is obtained by passing the sum of the fifth feature data and the sixth feature data through the LayerNorm layer.

13. The method according to claim 1, characterized in that, The mobile device includes a vehicle.

14. The method according to claim 3, characterized in that, The mobile device includes a vehicle.

15. The method according to any one of claims 10-14, characterized in that, The target includes one or any combination of people, vehicles, and obstacles.

16. An electronic device, characterized in that, The device includes a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions that, when the processor executes the program instructions, cause the electronic device to perform the steps of the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when the program is requested to be run by a computer, cause the computer to perform the method as described in any one of claims 1-15.

Images