Target detection method and device, vehicle and computing device cluster

Abstract

The embodiment of the invention discloses a target detection method, and the method comprises the steps: obtaining first input data, which is data of a first mode; obtaining a first feature of the first input data through a first feature extraction network, and extracting data of an interested part from the first input data as first segmentation data; according to the first feature and the first segmentation data, a second feature is obtained through a generative network, and the second feature comprises an enhanced feature relative to the first segmentation data; and obtaining a target detection result about the first input data through a detection network according to the first feature and the second feature. In the method, the second feature is the feature which is larger in information amount and richer in feature relative to the first segmentation data, and the feature information of the target in the first input data is fully enhanced, so that a detection network can obtain a more accurate target detection result according to the rich feature information in the first feature and the second feature, and the detection accuracy is improved. And the target detection precision is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a target detection method, device, vehicle, and computing device cluster. Background Art

[0002] Object detection is widely used in many application fields. At present, a commonly used object detection method is to collect optical images through visual sensors (such as cameras deployed at multiple angles) to perform object detection based on the optical images through neural networks. To this end, before object detection, a large amount of manually annotated training data is needed to train the neural network so that the neural network has the ability to detect various objects.

[0003] However, many of the currently collected training data sets present a long-tail distribution, that is, in the training data set, a small number of categories (also called head categories) correspond to a large number of samples, while most categories (also called tail categories) correspond to only a small number of samples. When the amount of training data for a certain category is small, the neural network's ability to detect data of the corresponding category will be significantly reduced, resulting in low detection accuracy of the neural network. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, apparatus, vehicle, and computing device cluster, which can enhance the detection capability of a neural network when performing target detection tasks, thereby improving target detection accuracy.

[0005] In a first aspect, the present application provides a target detection method, which includes: obtaining first input data, the first input data being data of a first modality; obtaining a first feature of the first input data through a first feature extraction network, and extracting data of a portion of interest from the first input data as first segmentation data; obtaining a second feature through a generation network based on the first feature and the first segmentation data, the second feature including an enhanced feature relative to the first segmentation data; and obtaining a target detection result about the first input data through a detection network based on the first feature and the second feature.

[0006] In the first aspect, the second feature is an enhanced feature relative to the first segmented data, that is, the second feature is a feature with a larger amount of information and richer features relative to the first segmented data, which fully complements and strengthens the feature information of the target in the first input data. It can be seen that by generating a network, more feature information about the target to be detected can be mined from the first data and the first segmented data, so that the second feature can enhance the features in the first segmented data and provide richer features for target detection.

[0007] In this way, if the first feature and the second feature are combined, it is possible to fully utilize the existing data information on the basis of maintaining the feature information of the original first input data, enhance the features of possible targets, and obtain richer, more complete and more detailed features, so that the detection network can obtain more accurate target detection results based on the rich feature information in the first feature and the second feature, thereby improving the detection accuracy of targets in scenarios such as long-tail data category detection, thereby improving the target detection accuracy.

[0008] In a possible implementation of the first aspect, obtaining a second feature through a generative network based on the first feature and the first segmentation data includes: performing super-resolution processing on the first segmentation data to obtain the first segmentation data after super-resolution processing; obtaining the second feature through a generative network based on the first feature and the first segmentation data after super-resolution processing.

[0009] In this possible implementation, the first segmentation data after super-resolution processing can be used as a conditional input of the generation network, thereby guiding the generation network to reconstruct the enhanced features corresponding to the first segmentation data from the first features.

[0010] In a possible implementation manner of the first aspect, extracting data of a portion of interest from first input data as first segmentation data includes: determining the portion of interest from a first feature as a first sub-feature; and determining data of the portion of interest in the first input data as the first segmentation data based on the first sub-feature.

[0011] In this possible implementation, the first segmented data is obtained by extracting data of an interested portion from the first input data. Therefore, the first segmented data may include the target in the first input data.

[0012] The second feature generated based on the first segmented data and the first feature is an enhanced feature relative to the first segmented data, that is, the second feature is a feature with a larger amount of information and richer features relative to the first segmented data, which fully completes and strengthens the feature information of the target in the first input data. It can be seen that by generating a network, more feature information about the target to be detected can be mined from the first data and the first segmented data, so that the second feature can enhance the features in the first segmented data, and can provide richer features for target detection, thereby providing a better data foundation for target detection, which is conducive to improving the accuracy of target detection.

[0013] In a possible implementation of the first aspect, obtaining a target detection result about the first input data through a detection network based on the first feature and the second feature, including: fusing the first sub-feature and the second feature to obtain a third feature; and obtaining a target detection result about the first input data through a detection network based on the third feature.

[0014] Exemplarily, the first sub-feature and the second feature can be reduced in dimension by relatively simple convolution or other methods, thereby effectively enhancing the features of the original mode (such as the first mode in this example) without significantly increasing the data processing delay.

[0015] The third feature obtained by fusing the first sub-feature and the second feature can carry more complete feature information about the target in the first input data, while the original feature information of the first sub-feature is not lost. In this way, after obtaining the third feature, the target detection result about the first input data can be obtained more accurately through the detection network based on the third feature.

[0016] In a possible implementation of the first aspect, obtaining a target detection result about the first input data through a detection network according to the third feature, including: replacing the first sub-feature in the first feature with the third feature to obtain the replaced first feature; and obtaining a target detection result about the first input data through a detection network according to the replaced first feature.

[0017] In this possible implementation, when the first sub-feature and the second feature are fused, the size of the fused third feature can be made the same as the size of the first sub-feature.

[0018] Since the third feature can carry more complete feature information about the target in the first input data without losing the original feature information of the first sub-feature, the third feature replaces the first sub-feature, and the obtained first feature after replacement can include more complete and more high-resolution feature information about the target in the first input data and the original feature information of the first input data. In this way, the detection network can more accurately identify the target based on the replaced first feature, thereby significantly improving the detection capability of the detection network and obtaining more accurate target detection results.

[0019] In a possible implementation of the first aspect, the number of first features is multiple; based on the replaced first features, a target detection result about the first input data is obtained through a detection network, including: after obtaining the replaced first features corresponding to each first feature, based on each replaced first feature, a target detection result about the first input data is obtained through a detection network.

[0020] In this possible implementation, in some examples, different first features are obtained by extracting features based on different scales. In this way, first features at different scales can be extracted from the first input data, so that feature information of different sizes in the first input data can be more fully extracted through different receptive fields.

[0021] In a possible implementation manner of the first aspect, the method also includes: acquiring second input data, the first input data and the second input data include the same part of interest; obtaining a fourth feature of the second input data through a second feature extraction network; obtaining a second feature through a generation network based on the first feature and the first segmentation data, including: obtaining the second feature through a generation network based on the first feature, the first segmentation data and the fourth feature.

[0022] In this possible implementation, based on the first feature, the first segmentation data and the fourth feature, the second feature obtained by generating a network not only includes the feature information of the input data itself (i.e., the first input data), but also includes the feature information of other input data (i.e., the second input data), and the second input data can be data of the second modality, or data of the first modality but different from the first input data. It can be seen that through this possible implementation, the richness and information content of the features in the second feature can be improved, providing a better data foundation for subsequent data processing, so that the subsequent detection network can have a more stable detection capability and improve the accuracy of target detection.

[0023] In a possible implementation of the first aspect, the method also includes: extracting data of the part of interest from the second input data as second segmentation data; obtaining a fifth feature through a generation network based on the fourth feature and the second segmentation data, the fifth feature including an enhanced feature relative to the second segmentation data; and obtaining a target detection result about the second input data through a detection network based on the fourth feature and the fifth feature.

[0024] In a possible implementation manner of the first aspect, the method further includes: obtaining a target detection result of the input data according to the target detection result of the first input data and / or the target detection result of the second input data.

[0025] In this possible implementation, exemplarily, the first input data and the second input data each have corresponding weights, that is, the target detection result regarding the first input data and the target detection result regarding the second input data each have corresponding weights, then, the input data target detection result can be obtained based on the target detection result regarding the first input data and the corresponding weight, and the target detection result regarding the second input data and the corresponding weight.

[0026] The second aspect of the present application provides a target detection device, which has the function of implementing the method of the first aspect or any possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as an acquisition module, a processing module, a generation module, and a detection module.

[0027] The third aspect of the present application provides a chip, which includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0028] A fourth aspect of the present application provides a vehicle, which includes the device of the second aspect or the chip of the third aspect to implement a method as described in the first aspect or any possible implementation method of the first aspect.

[0029] In a fifth aspect, the present application provides a computing device cluster, which includes at least one computing device, and the at least one computing device includes at least one processor, a memory, and computer execution instructions stored in the memory and executable on the processor. When the computer execution instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0030] In a sixth aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation of the first aspect.

[0031] The seventh aspect of the present application provides a computer program product, which includes one or more computer execution instructions. When the computer execution instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation method of the first aspect.

[0032] Among them, the technical effects brought about by the second to seventh aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of an exemplary scenario for target detection in the field of autonomous driving provided by an embodiment of the present application;

[0034] Figure 2 It is a flowchart of a target detection method provided in an embodiment of the present application;

[0035] Figure 3 It is a flowchart of a target detection method provided in an embodiment of the present application;

[0036] Figure 4 is a schematic diagram of extracting a first feature from first input data through a first feature extraction network provided in an embodiment of the present application;

[0037] Figure 5 is a schematic diagram of a diffusion model provided in an embodiment of the present application;

[0038] Figure 6 It is a flow chart of a generation network provided in an embodiment of the present application as a diffusion model;

[0039] Figure 7 It is a flow chart of a case where a generative network is a generative adversarial network provided in an embodiment of the present application;

[0040] Figure 8 This is a data processing flow diagram provided by an embodiment of the present application;

[0041] Figure 9a is a schematic diagram of a fusion of a first sub-feature and a second feature provided in an embodiment of the present application;

[0042] Figure 9b is a schematic diagram of another fusion of the first sub-feature and the second feature provided in an embodiment of the present application;

[0043] Fig.10 It is a schematic diagram of a process for obtaining a target detection result provided by an embodiment of the present application;

[0044] Fig.11 It is a schematic diagram of a process of performing target detection based on multimodal data provided by an embodiment of the present application;

[0045] Fig.12 is another schematic diagram of a process for performing target detection based on multimodal data provided in an embodiment of the present application;

[0046] Fig.13 is a schematic diagram of a target detection device provided in an embodiment of the present application;

[0047] Fig.14 is a schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0048] Fig.15 is a structural schematic diagram of a vehicle provided in an embodiment of the present application;

[0049] Fig.16 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of singular or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, and this is merely a description of the way in which objects with the same attributes are described in the embodiments of the present application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0053] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0054] (1) Neural Network

[0055] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0056]

[0057] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0058] (2) Loss Function

[0059] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0060] (3) Back propagation algorithm

[0061] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0062] Object detection is widely used in many application fields. For example, in the field of autonomous driving, it is often necessary to detect objects in the surrounding environment of the vehicle, such as lane line detection, obstacle detection, red light detection, road traffic sign detection, etc. through the relevant algorithms of autonomous driving.

[0063] At present, a commonly used target detection method is to collect optical images through visual sensors (such as cameras deployed at multiple angles) to perform target detection based on the optical images through neural networks, etc. To this end, before performing target detection, a large amount of manually annotated training data is required to train the neural network so that the neural network has the ability to detect various objects.

[0064] However, many of the currently collected training data sets present a long-tail distribution, that is, in the training data set, a small number of categories (also called head categories) correspond to a large number of samples, while most categories (also called tail categories) correspond to only a small number of samples. Such training data sets will make the neural network have a higher detection accuracy for the head category, but a weaker detection ability for the tail category, resulting in a lower detection accuracy for the tail category, resulting in a lower target detection accuracy of the neural network.

[0065] To this end, an embodiment of the present application provides a target detection method that can fully mine the feature information of input data when performing target detection tasks through a neural network, thereby improving the neural network's ability to detect data and improving the accuracy of target detection.

[0066] The target detection method of the embodiment of the present application can be applied to electronic devices.

[0067] The specific type of the electronic device is not limited here. Exemplarily, the electronic device can be a terminal device, a cloud server, a single server or a server cluster, or a virtual machine (VM) or a container.

[0068] The type of terminal device is not limited here. Exemplarily, the terminal device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a terminal in industrial control, a terminal in self-driving (such as a car computer, a vehicle computing platform, a domain controller, a sensor, etc.), a terminal in remote medical, a terminal in smart grid, a terminal in transportation safety, a terminal in smart city, a terminal in smart home, a terminal in Internet of things (IoT), a wearable device, a robot, etc., or a combination of one or more thereof.

[0069] The target detection method of the present application may be applied in a variety of fields. For example, the target detection method may be applied in one or more of the fields of autonomous driving, smart home, security, etc.

[0070] For example, in one example, the above-mentioned electronic device can be a chip. Specifically, the target detection method of the embodiment of the present application can be applied to a chip deployed in a vehicle, so that the vehicle has a target detection function, which facilitates better execution of intelligent driving tasks.

[0071] In another example, the electronic device may be a computing device cluster including at least one computing device, and the computing device cluster may provide cloud resources. Specifically, the target detection method of the embodiment of the present application may be applied to the computing device cluster, so that the computing device cluster may interact with the vehicle and / or roadside equipment to obtain input data and perform target detection tasks, and then transmit the target detection results to the vehicle, so that the vehicle performs intelligent driving tasks according to the target detection results.

[0072] For the convenience of description, in the subsequent introduction, the target detection method used in the field of autonomous driving is used as an example.

[0073] like Figure 1 The figure shows an exemplary scenario diagram of target detection in the field of autonomous driving.

[0074] exist Figure 1 In the scenario shown, roadside equipment (such as Figure 1The roadside device 1 and the roadside device 2 in the embodiment of the present application are used to collect input data through the roadside device and transmit the input data to the cloud, so as to execute the target detection method in any embodiment of the present application through the cloud, thereby detecting obstacles such as goods, tires, gravel, etc. on the road. At this time, the electronic device in the embodiment of the present application can be one or more servers in the cloud. Alternatively, it can also be Figure 1 The vehicle on the road in the vehicle collects input data, and then the target detection method in any embodiment of the present application can be executed through the vehicle-mounted terminal in the vehicle; or the vehicle can also upload the input data to the cloud to execute the target detection method in any embodiment of the present application through the cloud. Alternatively, the vehicle can obtain input data through vehicle-to-everything (V2X), for example, the vehicle obtains input data through roadside equipment or from other vehicles, and then the target detection method in any embodiment of the present application can be executed through the vehicle-mounted terminal in the vehicle; or the vehicle uploads the input data to the cloud to execute the target detection method in any embodiment of the present application through a computing device cluster in the cloud.

[0075] Based on the above electronic device, the overall flow chart of a target detection method in an embodiment of the present application is as follows: Figure 2 Among them, a target detection method of an embodiment of the present application includes steps 301-305, step 301 is used to obtain data, step 302 is used to implement feature extraction, steps 303-304 are used to implement feature generation, and step 305 is used to implement feature fusion to obtain a target detection result.

[0076] Combine the following Figure 3 , specifically introduces steps 301-305 included in a target detection method in an embodiment of the present application.

[0077] Step 301, obtaining first input data.

[0078] The first input data is data of a first modality.

[0079] There may be multiple sources of the first input data. For example, the first input data may be stored locally in the electronic device executing the embodiment of the present application, or may be transmitted to the electronic device by other devices, or may be collected by the electronic device through a web page or other applications, or may be collected by the electronic device through a sensor (such as an image sensor, millimeter wave radar, laser radar, ultrasonic radar, etc.), or may be obtained by the electronic device after processing image data, voice data, point cloud data, etc.

[0080] In the embodiment of the present application, there is no restriction on the modality of the first input data. For example, in the field of autonomous driving, the first input data may be data in an image modality such as an image or a video, or data in a point cloud modality such as radar point cloud data or laser point cloud data.

[0081] Step 302: Obtain a first feature of first input data through a first feature extraction network.

[0082] In the embodiment of the present application, the first feature extraction network can extract features from the data of the first modality. The first feature extraction network can be a currently existing or subsequently developed neural network, which is not limited in the embodiment of the present application.

[0083] Exemplarily, the first feature extraction network may include convolutional neural networks (CNN), transformer models and / or feature pyramid networks (FPN), etc.

[0084] In the embodiment of the present application, the number of first features extracted from the first input data by the first feature extraction network may be one or more.

[0085] In some examples, considering that the size of the target in the first input data may vary in actual application scenarios, the first feature extraction network can extract first features corresponding to different scales from the first input data to fully extract the feature information contained in the first input data.

[0086] For example, if Figure 4 As shown, it is an exemplary schematic diagram of extracting a first feature from first input data through a first feature extraction network provided in an embodiment of the present application.

[0087] exist Figure 4 In , the first feature extraction network includes a backbone network and an FPN module, so as to obtain the first features at four scales through a series of convolution or transformer processing, for example Figure 4 The first feature at the first scale, the first feature at the second scale, the first feature at the third scale, and the first feature at the fourth scale shown in .

[0088] Exemplarily, the backbone network may adopt a CNN, transformer or a combination thereof, for example, one or more networks such as ResNet, CSPresnet, InceptionNet, DenseNet, EfficientNet, and Transformer series may be adopted. Figure 4 In the figure, three layers of the backbone network are shown, namely C3, C4, and C5. The FPN module adopts a multi-scale feature fusion method, which fuses the features of different layers in the vertical direction of the FPN module (for example, Figure 4 The feature fusion between the P4 layer and the P3 layer in the FPN module, and the feature fusion between the layers in the FPN module and the corresponding layers in the backbone network (e.g. Figure 4 The feature fusion between the P4 layer of FPN and the corresponding C4 layer in the backbone network, etc., can generate the first feature with better feature expression and provide a better data foundation for subsequent data processing.

[0089] Step 303: extracting data of the portion of interest from the first input data as first segmented data.

[0090] In the embodiment of the present application, the portion of interest may be determined from the first input data, thereby extracting data of the portion of interest from the first input data as the first segmented data.

[0091] Exemplarily, if the first input data is an image, the portion of interest in the first input data may be a region of interest (ROI) in the first input data. The region of interest in the image may be an area to be processed that is outlined from the processed image in the form of a box, circle, ellipse, irregular polygon, etc. In the target detection task, the region of interest may be considered as an area that may include the target. If the first input data is point cloud data, the portion of interest in the first input data may be considered as a portion of the point cloud information that may include the target in the point cloud data.

[0092] There are many ways to determine the portion of interest from the first input data. In one example, the first input data can be directly processed to extract a preselected box from the first input data, and the portion framed by the preselected box is used as the portion of interest in the first input data.

[0093] In another example, the interesting portion in the first input data may be determined according to the first feature.

[0094] For example, in some examples, the above step 303 includes: determining the part of interest from the first feature; based on the mapping relationship between the first input data and the first feature, determining the data of the part of interest in the first input data according to the part of interest in the first feature as the first segmentation data.

[0095] In an embodiment of the present application, considering that the first feature is a feature extracted from the first input data and can reflect the information of the first input data, the extracted first feature can be fully utilized to determine the part of interest through methods such as a pretrained region proposal network (PRPN).

[0096] Taking the first input data as an image as an example, through the anchor technology in PRPN, one or more candidate regions can be determined from the first feature, and the one or more candidate regions may contain the target to be detected. Each candidate region may correspond to a score, which may represent the probability that the corresponding candidate region may contain the target. Then, the part of interest can be determined from the one or more candidate regions based on the score of each candidate region and a preset threshold, etc. For example, the candidate region with a score greater than the preset threshold can be used as the part of interest in the first feature, and the n candidate regions with a score greater than the preset threshold and the highest score can also be used as the part of interest in the first feature. In this way, the part of interest in the first feature can be determined.

[0097] Then, based on the mapping relationship between the first input data and the first feature, the part of interest in the first feature can be mapped to the first input data. For example, the corner point coordinates of the part of interest in the first feature can be mapped to the first input data, so that the part of interest in the first input data can be determined.

[0098] After determining the portion of interest in the first input data, the portion of interest in the first input data may be cropped, and the cropped image may be used as the first segmented data.

[0099] It can be understood that in the embodiment of the present application, the execution order of step 302 and step 303 can be in various situations, which is not limited here.

[0100] For example, referring to the above example, if the first input data is directly processed to extract a pre-selection box from the first input data, and the part framed by the pre-selection box is used as the part of interest in the first input data, step 303 can be executed before step 302, or step 302 and step 303 can be executed in parallel.

[0101] If the interested portion is determined from the first feature as the first sub-feature, and the interested portion in the first input data is determined according to the first sub-feature to obtain data of the interested portion in the first input data as the first segmented data, then step 303 is performed after step 302 .

[0102] Step 304: Obtain a second feature by generating a network based on the first feature and the first segmentation data.

[0103] The second features include enhanced features relative to the first segmented data.

[0104] In the embodiment of the present application, the generative network can be an existing generative network such as a diffusion model, a variational auto encoder (VAE), a generative adversarial network (GAN), a flow-based generative model, etc., or it can be other generative networks developed subsequently, and the embodiment of the present application does not limit this.

[0105] It can be understood that the first segmented data is obtained by extracting the data of the interested part from the first input data, so the first segmented data may include the target in the first input data. In many scenarios, for example, in the field of autonomous driving, the target in the first input data is a distant obstacle, so the data volume and size of the first segmented data are usually small, which may be unfavorable for detecting the target in the first input data.

[0106] The second feature is an enhanced feature relative to the first segmented data, that is, the second feature is a feature with a larger amount of information and richer features relative to the first segmented data, which fully complements and strengthens the feature information of the target in the first input data. It can be seen that by generating a network, more feature information about the target to be detected can be mined from the first data and the first segmented data, so that the second feature can enhance the features in the first segmented data, and can provide richer features for target detection, thereby providing a better data basis for target detection, which is conducive to improving the accuracy of target detection.

[0107] In some embodiments, step 304 includes: performing super-resolution processing on the first segmented data to obtain first segmented data after super-resolution processing; and obtaining a second feature by generating a network based on the first feature and the first segmented data after super-resolution processing.

[0108] In an embodiment of the present application, there may be multiple ways to perform super resolution (SR) processing on the first segmented data. For example, the first segmented data may be super-resolved using, but not limited to, a bicubic interpolation algorithm. In this way, data with a higher resolution and a larger data volume than the first segmented data may be obtained. For example, if the first segmented data is image data, the size of the first segmented data after super resolution processing may be 2 times, 4 times, etc., of the first segmented data.

[0109] After obtaining the first segmented data after super-resolution processing, the second feature can be obtained through the generative network according to the first feature and the first segmented data after super-resolution processing. The first segmented data after super-resolution processing can be used as a conditional input of the generative network, thereby guiding the generative network to reconstruct the enhanced feature corresponding to the first segmented data from the first feature.

[0110] Below, taking the generation networks as diffusion models and generative adversarial networks as examples, the specific processing process of obtaining the second feature is exemplarily introduced.

[0111] 1. In this example, the generated network is a diffusion model.

[0112] Diffusion models usually include two processes: forward process and reverse process. Both the forward process and the reverse process are parameterized Markov chains, where the reverse process can be used to generate data, and variational inference is generally used for modeling and then solving.

[0113] like Figure 5 As shown in the dashed part from right to left in the figure, the forward process is the process of gradually adding Gaussian noise to the original feature data until the data completely becomes random noise. Among them, the original feature data X0 represents the first feature V rgb For the original feature data X0~q(X0), the forward process can include a total of T steps of diffusion process, where q(X0) refers to the Gaussian noise added to the original feature data X0. Each step of the diffusion process is a step of the feature data X0 obtained by the previous diffusion process. t-1 Add Gaussian noise as follows:

[0114]

[0115] It means that the variance used in each step gradually increases as the number of steps increases, and the variance also gradually increases, satisfying the inequality β1<β2<…<β T .X t is the characteristic data of this diffusion process. If the number of diffusion steps T is large enough, then the final X T That is, the normal distribution noise completely loses the original feature data. Each diffusion process in the forward process will generate a noisy feature data X t , based on q(X t |X t-1 ) is known, according to the properties of the Markov process, we can recursively get q(X t |X0):

[0116]

[0117] like Figure 5 As shown in the solid line from left to right, the reverse process can be understood as a feature generation process. The reverse process is an inverse diffusion process because it is impossible to directly obtain q(X t-1 |X t ) distribution, and in order to achieve the purpose of obtaining a newly generated super-resolution feature with a distribution close to the original feature distribution from any random noise distribution, p is used θ (X t-1 |X t ) to approximate this distribution. According to the forward process, we can get:

[0118]

[0119] The loss function L of the reverse diffusion process with the number of steps t in the reverse process can be set t It is expressed as:

[0120] L t =D KL (q(X t |X t+1 X0)||p θ (X t |X t+1 )),1≤t≤T

[0121] D KL (·) is the Kullback-Leibler (KL) divergence loss. The diffusion model is optimized to reduce the value of the loss function, that is, to minimize q(X t |X t+1 X0) and p θ (X t |X t+1 ). During the reverse diffusion process, the generated features will gradually tend to the original feature distribution.

[0122] In this example, the input data of the diffusion model may include the input of the forward process and the conditional input. The conditional input is used to guide the feature generation process of the diffusion model.

[0123] In this example, the first feature can be used as the input of the forward process of the diffusion model.

[0124] The conditional input of the diffusion model may include but is not limited to the first segmentation data. The first segmentation data is used as the conditional input of the diffusion model to guide the diffusion model to generate enhanced features for the feature part corresponding to the first segmentation data.

[0125] In addition, in some other examples, the conditional input of the diffusion model may also include feature data of other input data different from the first input data, for example, it may also include feature data of one or more other modes different from the first mode, and / or feature data of other input data belonging to the first mode but different from the first input data. The specific implementation of this example can refer to Figure 11-12 And the description of any related embodiment is not limited here.

[0126] The following takes the case where the conditional input of the diffusion model includes the first segmented data as an example to exemplarily introduce the processing procedure of the diffusion model in this example.

[0127] The traditional diffusion model can be considered as an equally weighted sequence of denoising autoencoders ∈ (Z t ,t); t=1…T. The goal is to t The prediction corresponds to the denoised variant, or prediction noise, where Z t The first feature V rgb The noisy version obtained by the forward process.

[0128] In this example, the diffusion model can be obtained by conditional denoising autoencoder ∈ θ (Z t ,t,y) to achieve, t=1…T, the goal is to t and y predictions correspond to the denoised variants, or prediction noise, where Z t The first feature V rgb The noise version obtained by the forward process, y, is the conditional input of the diffusion model to guide and control the feature generation process of the diffusion model. In this example, y may include the first segmentation data after super-resolution processing, and in some other examples, y may also include feature data of one or more other modalities (e.g., point cloud data) different from the first modality.

[0129] like Figure 6 In the example shown, the first input feature V of the diffusion model is rgb After the forward process, the feature Z of the noise version is obtained T In addition, the conditional input y of the diffusion model is introduced, and the encoder τ θ Map y to an intermediate representation τ θ (y), the encoder τ θ It can also be understood as a translator so that the intermediate representation τ after mapping θ (y) and ZT Located in the same domain space, so that the intermediate representation τ can be subsequently θ (y) and Z T Processing to eliminate y and Z T The difference in expression between them. Among them, in obtaining the intermediate representation τ θ (y) After that, τ θ (y) and Z T Splice and obtain the intermediate features after splicing. The intermediate features and Z T Can be used as input to the inverse process of the diffusion model.

[0130] like Figure 6 In the example shown, the reverse process of the diffusion model is implemented by a denoising network and an attention mechanism. For example, the denoising network can use a UNet backbone network to learn the potential representation of the diffusion process, and an attention mechanism is added to UNet to specifically implement ∈ θ (Z T ,t,y).

[0131] The attention mechanism can be implemented by the following formula:

[0132]

[0133] Where Q is the query matrix, K is the bond matrix, V is the value matrix, i is the i-th step in the reverse process, is the query weight matrix, is the query key matrix, is the query value matrix, and It can be determined through training, and d is the dimension of the key matrix.

[0134] In the denoising network, there can be one or more layers of the network combined with the attention mechanism. For example, Figure 6 In the example shown, it is shown that in the denoising network, at least 4 layers are combined with the attention mechanism, and the input of the layer combined with the attention mechanism can include the output of the previous layer of the layer and τ θ (y) and Z T . Figure 6 In the example shown, the user can use a switch to configure whether to use the attention mechanism in the denoising network, as well as the layers that combine the attention mechanism.

[0135] In addition, in some examples, techniques such as skip connection can also be used in the denoising network.

[0136] If the diffusion model is passed To represent, the above-mentioned association processing operation of the forward process noise Noise and the condition input Condition is represented by the function F, then, the generation process of the second feature can be described by the following formula:

[0137]

[0138] In this way, based on the denoising network and attention mechanism, after T-step denoising, we can obtain an enhanced feature V with richer details. Gen rgb , that is to say, obtaining the second feature.

[0139] 2. In this example, the generative network is a generative adversarial network.

[0140] The generative adversarial network GAN may include a generator and a discriminator. Through learning through the mutual game between the generator and the discriminator, a second feature with richer features and clearer details can be generated.

[0141] like Figure 7 As shown, in this example, the input of GAN may include one or more of the following two methods: (1) using only noise of the same size as input data; (2) using the first feature V rgb + noise Noise as input, the first feature V can be concatenated in a way that is not limited to rgb and noise Noise to form new input data. After the generator receives the input data, it outputs the corresponding second feature V Gen rgb .

[0142] In addition, in some examples, in order to make it easier to fit the generative adversarial network, the second feature V output Gen rgb It will then go through a simple pre-trained conversion network (including but not limited to convolutional neural networks (CNN) structure), the main purpose of which is to convert the input second feature V Gen rgb Reduce the dimension and expand the size to output data consistent with the size of the first segmented data after super-resolution processing. For example, if the first input data is an RGB image, the pre-trained conversion network can also output an RGB image Img consistent with the size of the first segmented data after super-resolution processing. Gen rgb .Img Gen rgbThe first segmentation data after the corresponding super-resolution processing is used as the input of the discriminator network. The discriminator network is mainly used to calculate Img Gen rgb The similarity score of the first segmented data after super-resolution processing in the image space. The higher the similarity score of the two, the more similar the image obtained by the generated network is to the original image, and the discriminator cannot distinguish the authenticity of the two. This method also further ensures that the generated second feature V Gen rgb The first feature V of the original input rgb The distribution is relatively close, but at the same time, the second feature V of super resolution with richer information and clearer details can be obtained. Gen rgb .

[0143] Step 305: Obtain a target detection result about the first input data through a detection network according to the first feature and the second feature.

[0144] The detection network may be an existing or subsequently developed detection network, which is not limited here. For example, the detection network may be a small neural network based on CNN, or a large neural network based on a transformer model.

[0145] Since the second feature has more information and richer features than the first segmented data, it fully completes and strengthens the feature information of the target in the first input data. Therefore, the detection network can accurately detect the target in the first input data based on the first feature and the second feature, which significantly improves the accuracy of target detection.

[0146] For example, Figure 8 The figure is a schematic diagram of an exemplary data processing flow provided in an embodiment of the present application.

[0147] exist Figure 8 In the example shown, the first input data is input data in image mode as an example for introduction.

[0148] exist Figure 8 In the example shown, the first segmented data can be segmented from the first input data according to the portion of interest in the first feature (i.e., the first sub-feature), and then the first segmented data is subjected to super-resolution processing to obtain the first segmented data after super-resolution processing. Then, the second feature can be obtained by generating a network based on the first segmented data after super-resolution processing and the first feature.

[0149] Among them, since the second feature is an enhanced feature corresponding to the interesting part of the first feature (that is, the first sub-feature), in some examples, the second feature can be fused with the first sub-feature in the first feature to enhance the feature information of the interesting part in the first feature, and then the fused feature is input into the detection network to facilitate the detection network to perform feature recognition and detection and obtain accurate target detection results.

[0150] Specifically, in some embodiments, based on the first feature and the second feature, a target detection result about the first input data is obtained through a detection network, including: fusing the first sub-feature and the second feature to obtain a third feature; based on the third feature, obtaining the target detection result about the first input data through a detection network.

[0151] There are many methods for fusing the first sub-feature with the second feature, which are not limited here. For example, during the fusion process, the dimensions of the first sub-feature and the second feature can be adjusted to the same size, and the first sub-feature after the dimension adjustment is combined with the second feature after the dimension adjustment to obtain the third feature.

[0152] Two fusion methods are provided as examples below:

[0153] 1. Fusion method 1

[0154] The first sub-feature can be processed by a convolution operation to obtain a processed first sub-feature, and the second feature can be processed by multiple layers of dilated convolution and convolution operations to obtain a processed second feature. The processed second feature has the same size as the processed first sub-feature, so that the processed second feature and the processed first sub-feature can be matrix-added, that is, the processed second feature and the processed first sub-feature can be element-added to obtain a third feature, and the size of the fused third feature is the same as the size of the first sub-feature.

[0155] In addition, in some examples, there may be multiple ways to implement the specific process of processing the second feature to obtain the processed second feature, which is not limited here.

[0156] For example, Figure 9a In the example shown, multiple paths (such as Figure 9a The second feature is processed respectively by the paths 1, 2 and 3 shown in the figure, so that the second feature passes through multiple paths (for example Figure 9aThe convolution kernel size and / or step size of the dilated convolution in different paths can be different, and the convolution kernel size and / or step size of the convolution operation can be different, so that the second feature can be reduced in dimension with different receptive fields in different paths to obtain the reduced-dimensional features corresponding to the three paths. Then, the reduced-dimensional features corresponding to the three paths can be matrix-added, that is, the corresponding elements can be added to obtain the processed second feature.

[0157] In this example, the second feature is dimensionally reduced through multiple paths respectively, so that the second feature is dimensionally reduced with different receptive fields, so that the feature information of different scales in the second feature can be better retained in the second feature after dimension reduction, thereby providing more sufficient and complete feature information for subsequent data processing.

[0158] 2. Fusion method 2

[0159] The first sub-feature may be processed by a convolution operation to obtain a processed first sub-feature. Furthermore, the second feature may be processed by multiple layers of dilated convolution and convolution operations to obtain a processed second feature. The processed second feature has the same size as the processed first sub-feature.

[0160] For example, in Figure 9b In the example shown, multiple paths (such as Figure 9b The second feature is processed respectively by the paths 1, 2 and 3 shown in the figure, so that the second feature passes through multiple paths (for example Figure 9b Then, the reduced-dimensional features corresponding to the three paths can be matrix-added, that is, the corresponding elements can be added to obtain the processed second features. Then, the reduced-dimensional second features and the processed first sub-features can be spliced ​​along the channel dimension to obtain the spliced ​​features, and then the spliced ​​features can be convolved through a convolution layer to obtain the third feature, and the size of the fused third feature is the same as that of the first sub-feature.

[0161] In any of the above fusion methods, relatively simple methods such as convolution are used for data processing, which can effectively enhance the features of the original mode (such as the first mode in this example) without significantly increasing the data processing delay.

[0162] The third feature obtained by fusing the first sub-feature and the second feature can carry more complete feature information about the target in the first input data, while the original feature information of the first sub-feature is not lost. In this way, after obtaining the third feature, the target detection result about the first input data can be obtained more accurately through the detection network based on the third feature.

[0163] There are multiple specific ways to process the third feature through the detection network. For example, the third feature can be used as an input of the detection network, or the detection network can also combine the third feature with multiple features such as the first feature to perform target detection.

[0164] In one embodiment, according to the third feature, a target detection result about the first input data is obtained through a detection network, including: replacing the first sub-feature in the first feature with the third feature to obtain the replaced first feature; according to the replaced first feature, obtaining the target detection result about the first input data through a detection network.

[0165] like Fig.10 As shown, it is a schematic diagram of a process for obtaining target detection results provided in an embodiment of the present application.

[0166] exist Fig.10 In the example shown, the number of interesting parts in the first feature is two (that is, Fig.10 In this way, referring to the above embodiment, the third feature 1 corresponding to the first sub-feature 1 and the third feature 2 corresponding to the first sub-feature 2 can be obtained. Then, the third feature 1 can replace the first sub-feature 1 in the first feature, and the third feature 2 can replace the first sub-feature 2 in the first feature to obtain the replaced first feature, so as to obtain the target detection result about the first input data through the detection network according to the replaced first feature.

[0167] Since the third feature can carry more complete feature information about the target in the first input data without losing the original feature information of the first sub-feature, the third feature replaces the first sub-feature, and the obtained first feature after replacement can include more complete and more high-resolution feature information about the target in the first input data and the original feature information of the first input data. In this way, the detection network can more accurately identify the target based on the replaced first feature, thereby significantly improving the detection capability of the detection network and obtaining more accurate target detection results.

[0168] In addition, in some embodiments, there are multiple first features, and different first features are obtained by performing feature extraction based on different scales; according to the replaced first features, a target detection result about the first input data is obtained through a detection network, including: after obtaining the replaced first features corresponding to each first feature, according to each replaced first feature, a target detection result about the first input data is obtained through a detection network.

[0169] In an embodiment of the present application, different first features may be obtained by performing feature extraction based on different scales. For example, they may be obtained by performing feature extraction using convolution kernels of different sizes and / or different step sizes, thereby more fully extracting feature information of different sizes in the first input data through different receptive fields.

[0170] After obtaining a plurality of first features, for each first feature, the interested portion in the first input data can be correspondingly determined according to the interested portion in the first feature, so as to obtain data of the interested portion corresponding to the first feature in the first input data as the first segmentation data corresponding to the first feature. In this way, the first segmentation data corresponding to each first feature can be obtained from the first input data.

[0171] Then, according to any first feature and the first segmentation data corresponding to the first feature, a second feature corresponding to the first feature is obtained by generating a network, and the second feature includes an enhanced feature relative to the first segmentation data corresponding to the corresponding first feature. In this way, the second feature corresponding to each first feature can be obtained.

[0172] The third feature corresponding to the first feature can be obtained by fusing the first sub-feature in any first feature with the second feature corresponding to the first sub-feature in the first feature. The third feature corresponding to the first feature replaces the first sub-feature corresponding to the first feature to obtain the corresponding replaced first feature.

[0173] In this way, the replaced first features corresponding to each first feature can be obtained. After obtaining the replaced first features corresponding to each first feature, the target detection result about the first input data is obtained through the detection network according to each replaced first feature.

[0174] Exemplarily, the detection network can output information of one or more candidate boxes at the first feature scale (e.g., candidate box coordinates, categories corresponding to candidate boxes, etc.) and scores of each candidate box, which can reflect the probability that the corresponding candidate box includes the target. In this way, a candidate box with high confidence (e.g., a score higher than the corresponding score threshold) can be determined from the one or more candidate boxes based on a preset score threshold and the score of each candidate box. Then, based on the mapping relationship between the first input data and the first feature, the candidate box with high confidence in the first feature can be mapped to the first input data to determine the location and category of the target in the first input data, so as to obtain target detection results for the first input data.

[0175] In this way, the input data input into the detection network may include feature information at different scales of the first input data, and may include super-resolution feature information of possible targets at different scales, so that the detection network can fully perform feature recognition and detection and obtain more accurate target detection results.

[0176] It can be seen that in the embodiment of the present application, the data of the part of interest in the first input data (i.e., the first segmented data) and the first feature can be used to generate a second feature that is similar to the original feature distribution of the first input data but has a higher resolution and richer details, so that the detection network can combine the first feature containing the complete feature information of the first input data and the second feature after feature enhancement based on the part of interest in the first input data, and fully mine the feature information, thereby being able to obtain a higher detection capability and improve the accuracy of target detection. For example, in detection scenarios such as long-tail distribution, the feature potential of the tail category data with relatively sparse data can be fully mined without adding additional data of the tail category, thereby significantly improving the target detection accuracy of the long-tail category data.

[0177] For example, in some examples, the target detection results of the tail category data using an embodiment of the present application and the existing model are shown in Table 1.

[0178] Table 1: Target detection results of tail category data using an embodiment of the present application and the existing model

[0179]

[0180] In the example of Table 1, the number of data whose ground truth (GT) is the tail category is 92.

[0181] In the prior art, when only a CNN-based model or a transformer-based model is used for target detection, the input data can be subjected to feature extraction and then target detection can be performed directly through the detection network.

[0182] When a CNN-based model is combined with an embodiment of the present application, or a transformer-based model is combined with an embodiment of the present application to perform a target detection task, the CNN-based model or the transformer-based model can be used as a detection network. When performing a target detection task, an embodiment of the present application can be used, such as Figure 2 The process shown processes the input data to obtain the corresponding target detection results.

[0183] It can be seen that when any of the above embodiments is used for experiments, no matter whether the detection network adopts a small neural network based on CNN or a large neural network based on the transformer model, performance indicators such as the recall rate of target detection for tail category data are significantly improved compared with the solution that does not adopt the embodiments of the present application.

[0184] In the above embodiment, the target detection method is introduced by taking the first input data as an example, wherein the input data may include but is not limited to the first input data. For example, in some examples, the target detection method of the embodiment of the present application can also be used to perform target detection based on multiple sets of input data.

[0185] In the following, in combination with any of the above embodiments, an object detection method based on multiple sets of input data is exemplarily introduced.

[0186] In some embodiments, the target detection method also includes: obtaining second input data, the first input data and the second input data include the same part of interest; obtaining a fourth feature of the second input data through a second feature extraction network; obtaining a second feature through a generation network based on the first feature and the first segmentation data, including: obtaining the second feature through a generation network based on the first feature, the first segmentation data and the fourth feature.

[0187] In some examples, the second input data is data of a second modality, and the second modality is different from the first modality. The modalities of the first input data and the second input data are not limited here.

[0188] For example, in the field of autonomous driving, the first input data may be image modality data, and the second input data may be laser point cloud modality data or radar point cloud modality data. During vehicle driving, the camera and laser radar in the vehicle may collect data on the same target on the road (such as obstacles, signs or surrounding vehicles on the road), thereby obtaining the first input data and the second input data.

[0189] In other examples, the second input data may also be data of the same modality as the first input data, that is, the second input data is also data of the first modality. The second input data may have a different data source and / or a different acquisition time from the first input data.

[0190] For example, the first modality is an image modality.

[0191] The first input data is image data collected by photographing the environment in front of the vehicle with a first posture through a first camera of the vehicle; and the second input data is image data collected by photographing the environment in front of the vehicle with a second posture through a second camera of the vehicle.

[0192] Alternatively, the first input data is image data collected at a first moment by photographing the environment in front of the vehicle with a first posture through the first camera of the vehicle; and the second input data is image data collected at a second moment by photographing the environment in front of the vehicle with a third posture through the first camera of the vehicle.

[0193] The first input data and the second input data include the same portion of interest. In this case, the first input data and the second input data may be data about the same possible target in the scene.

[0194] For example, the first input data and the second input data are data obtained by photographing the environment in front of the vehicle. In one scenario, the first input data is image data obtained by photographing the environment in front of the vehicle by the first camera, and the second input data is point cloud data obtained by collecting information about the environment in front of the vehicle by the laser radar. In another scenario, the first input data is image data obtained by photographing the environment in front of the vehicle by the first camera in the first posture, and the second input data is image data obtained by photographing the environment in front of the vehicle by the second camera in the second posture.

[0195] In the above examples, the first input data and the second input data have different data sources, different corresponding postures and / or different modalities, but it can be considered that the first input data and the second input data include part or all of the interested part in the environment in front of the vehicle.

[0196] like Fig.11 In the exemplary process diagram shown, after obtaining the second input data, the second input data can be feature extracted by the second feature extraction network. The second feature extraction network can be the same as the first feature extraction network, or there can be differences, which is not limited in the embodiment of the present application.

[0197] After the second input data is subjected to feature extraction by the second feature extraction network to obtain a fourth feature of the second input data, a portion of the input data of the generation network can be obtained according to the fourth feature.

[0198] In some examples, the fourth feature may be directly input into the generative network.

[0199] In other examples, such as Fig.11 As shown, the interesting part in the second input data can also be determined according to the interesting part in the fourth feature, so as to obtain the input data of the generating network corresponding to the first input data according to the interesting part in the second input data, the first feature and the first segmentation data.

[0200] For example, the part of interest can be determined from the fourth feature as the second sub-feature, and then the part of interest in the second input data can be determined according to the second sub-feature to obtain the data of the part of interest in the second input data as the second segmented data. Then, the second segmented data can be super-resolution processed to obtain the second segmented data after super-resolution processing. In this example, the second segmented data after super-resolution processing, the first feature and the first segmented data can be used as input data of the generation network corresponding to the first input data, and the second sub-feature, the first feature and the first segmented data can also be used as input data of the generation network corresponding to the first input data.

[0201] It can be seen that there are many ways to obtain the input data of the generated network according to the fourth feature. Figure 6 In the example shown, the fourth feature can be directly used as a condition input of the diffusion model, or the second segmentation data or the second sub-feature obtained after super-resolution processing according to the fourth feature can also be used as a condition input of the diffusion model.

[0202] In addition, when the second input data is data of the second modality, the fourth feature or the second segmented data after super-resolution processing can output feature information aligned with the first feature through an alignment module (including but not limited to a neural network based on a CNN framework, a Transformer framework, etc.), thereby solving the problem of a gap in the representation space of multimodal data, so that the diffusion model can align the feature information of multiple modalities so that it can be jointly analyzed and processed in the diffusion model to generate features with richer information.

[0203] It can be seen that in the embodiment of the present application, the input data for generating the network may include feature data of multiple input data (i.e., feature information of the first feature corresponding to the first input data and feature information of the fourth feature corresponding to the second input data), and may also include data of an interesting part of the first input data (i.e., first segmentation data) as conditional information or guiding information for generating the network.

[0204] In this way, according to the first feature, the first segmentation data and the fourth feature, the second feature obtained by generating the network not only contains the feature information of the input data itself (i.e., the first input data), but also contains the feature information of other input data (i.e., the second input data). For example, the second input data can be data of the second modality, then the second feature integrates the feature information of the multimodal data; or the second input data is data of the first modality but different from the first input data, then the second feature integrates the feature information of multiple input data. Therefore, the second feature generated based on the guidance of the first segmentation data and the fourth feature is a feature with a larger amount of information and richer features than the first segmentation data, which fully completes and strengthens the feature information of possible targets in the first input data. It can be seen that through the embodiments of the present application, the richness and information content of the features in the second feature can be improved, providing a better data basis for subsequent data processing, so that the subsequent detection network can have a more stable detection capability and improve the accuracy of target detection.

[0205] It should be noted that in the embodiment of the present application, the second input data may be one or more, and in the embodiment of the present application, the number and type of the second modality are not limited herein. In other words, the second feature can be obtained by generating a network based on the first feature, the first segmentation data and one or more fourth features. The modality of the input data corresponding to each fourth feature may be the same or different. For example, in one example, the fourth feature 1 can be obtained by the second input data 1 under the millimeter wave radar point cloud modality (second modality 1), and the fourth feature 2 can be obtained by the second input data 2 under the lidar point cloud modality (second modality 2). Then, the fourth feature 1 and the fourth feature 2 can both be used as conditional inputs of the generation network to obtain the second feature.

[0206] In some embodiments, the target detection result regarding the first input data may be used as the target detection result of the content, and the fourth feature is not further processed.

[0207] Or, in some embodiments, Fig.11 As shown, the target detection result corresponding to the second input data can also be obtained according to the fourth feature.

[0208] Specifically, in some embodiments, the method also includes: extracting data of the part of interest from the second input data as second segmentation data; obtaining a fifth feature through a generation network based on the fourth feature and the second segmentation data, the fifth feature including an enhanced feature relative to the second segmentation data; obtaining a target detection result about the second input data through a detection network based on the fourth feature and the fifth feature.

[0209] In the embodiment of the present application, the method for obtaining the second segmentation data may be the same as or similar to the first segmentation data, and specific reference may be made to the relevant description of the first segmentation data, which will not be repeated here. The method for obtaining the fifth feature may be the same as or similar to the second feature, and specific reference may be made to the relevant description of the second feature. For example, the fifth feature may be obtained by generating a network based on the fourth feature, the second segmentation data, and the second feature of a different modality corresponding to the fourth feature, so that the fifth feature may be generated by combining multi-modal feature information, etc. In addition, the specific method for obtaining the target detection result about the second input data through the detection network may also be the same as or similar to obtaining the target detection result about the first input data, and will not be repeated here.

[0210] After obtaining the target detection result about the first input data and the target detection result about the second input data, the target detection result of the input data can be obtained based on the target detection result about the first input data and / or the target detection result about the second input data. The specific method for determining the target detection result of the input data is not limited in the embodiment of the present application.

[0211] Exemplarily, the first modality and the second modality each have corresponding weights, that is, the target detection result regarding the first input data and the target detection result regarding the second input data each have corresponding weights. Then, the target detection result of the input data can be obtained based on the target detection result regarding the first input data and the corresponding weight, and the target detection result regarding the second input data and the corresponding weight.

[0212] like Fig.12 As shown, it is a schematic diagram of an exemplary process of performing target detection based on multiple sets of input data provided in an embodiment of the present application.

[0213] The first input data may be image data, and the second input data may be point cloud data.

[0214] The image data is extracted through the image feature extraction network (i.e., the first feature extraction network) to obtain the image spatial feature V rgb As the first feature, the point cloud data is extracted through the point cloud feature extraction network (i.e., the second feature extraction network) to obtain the point cloud spatial feature V Point Cloud Then, we can combine the features of multiple sets of input data (that is, the image space feature V rgb And point cloud spatial features V Point Cloud ), through the generative network, the super-resolution image spatial feature V is obtained Gen rgb As the second feature, the super-resolution point cloud spatial feature V is obtained Gen PointCloud As the fifth feature. Taking the second feature as an example, we can use the point cloud spatial feature V Point Cloud The part of interest in the point cloud data is determined as the second segmentation data; then, the second segmentation data after super-resolution processing can be obtained, so that according to the second segmentation data after super-resolution processing, the image spatial feature V rgb And the first segmentation data after super-resolution processing, the super-resolution image spatial feature V is obtained by generating a network Gen rgb The generation process of the fifth feature can refer to the second feature.

[0215] In this way, the super-resolution image spatial feature V Gen rgb and image spatial features V rgb Perform feature fusion to obtain image space fusion feature V Merge rgb As the third feature, the target detection result about the image data is obtained through the detection network according to the third feature; and the super-resolution point cloud spatial feature V Gen Point Cloud And point cloud spatial features V Point Cloud Perform feature fusion to obtain the point cloud spatial fusion feature V Merge Point Cloud , based on the point cloud spatial fusion feature V Merge Point Cloud , the target detection result about the point cloud data is obtained through the detection network. In this way, the final target detection result can be obtained according to the target detection result about the image data and the target detection result about the point cloud data.

[0216] It can be seen that the scheme of the embodiment of the present application can be flexibly applied to target detection scenarios of various modes (which can be single mode or multi-mode), and is highly practical. In addition, the scheme of the embodiment of the present application can make full use of the existing data information on the basis of maintaining the feature distribution of the original input data, enhance the features of possible targets, obtain richer, more complete and more detailed feature information, and improve the richness of the features, thereby improving the detection accuracy of targets in long-tail data in scenarios such as scarce long-tail data without the need for additional data collection and without the need for additional collection of a large amount of data.

[0217] The target detection method provided in the embodiments of the present application is described above from multiple aspects. The target detection device provided in the embodiments of the present application is described below in conjunction with the accompanying drawings.

[0218] like Fig.13As shown, the embodiment of the present application provides a target detection device 130, and the target detection device 130 includes:

[0219] An acquisition module 1301 is used to acquire first input data, where the first input data is data of a first modality;

[0220] A processing module 1302 is used to obtain a first feature of the first input data through a first feature extraction network, and extract data of an interested portion from the first input data as first segmented data;

[0221] A generating module 1303 is used to obtain a second feature by generating a network according to the first feature and the first segmentation data, where the second feature includes an enhanced feature relative to the first segmentation data;

[0222] The detection module 1304 is used to obtain a target detection result about the first input data through a detection network according to the first feature and the second feature.

[0223] Optionally, the generating module 1303 is used to: perform super-resolution processing on the first segmented data to obtain the first segmented data after the super-resolution processing; and obtain the second feature through a generating network according to the first feature and the first segmented data after the super-resolution processing.

[0224] Optionally, the processing module 1302 is used to: determine an interesting portion from the first feature as a first sub-feature; based on a mapping relationship between the first input data and the first feature, determine data of the interesting portion in the first input data according to the first sub-feature as first segmented data.

[0225] Optionally, the detection module 1304 is used to: fuse the first sub-feature and the second feature to obtain a third feature; and obtain a target detection result about the first input data through a detection network based on the third feature.

[0226] Optionally, the detection module 1304 is used to: replace the first sub-feature in the first feature with the third feature to obtain the replaced first feature; and obtain a target detection result about the first input data through a detection network based on the replaced first feature.

[0227] Optionally, there are multiple first features; the detection module 1304 is used to: after obtaining the replaced first features corresponding to each first feature, obtain the target detection result about the first input data through the detection network according to each replaced first feature.

[0228] Optionally, the acquisition module 1301 is further used to: acquire second input data, the first input data and the second input data include the same portion of interest;

[0229] The processing module 1302 is further used to: obtain a fourth feature of the second input data through a second feature extraction network;

[0230] The generation module 1303 is used to obtain the second feature by generating a network according to the first feature, the first segmentation data and the fourth feature.

[0231] Optionally, the processing module 1302 is used to: extract data of the portion of interest from the second input data as second segmented data;

[0232] The generating module 1303 is used to: obtain a fifth feature by generating a network according to the fourth feature and the second segmentation data, wherein the fifth feature includes an enhanced feature relative to the second segmentation data;

[0233] The detection module 1304 is used to obtain a target detection result about the second input data through a detection network according to the fourth feature and the fifth feature.

[0234] Optionally, the detection module 1304 is used to obtain a target detection result of the input data according to a target detection result of the first input data and / or a target detection result of the second input data.

[0235] Fig.14 1 is a possible logical structure diagram of a chip 140 provided in an embodiment of the present application. The chip 140 is used to implement the functions of the electronic device involved in any of the above embodiments. The chip 140 includes: a memory 1401, a processor 1402, a communication interface 1403 and a bus 1404. Among them, the memory 1401, the processor 1402, and the communication interface 1403 are connected to each other through the bus 1404.

[0236] The memory 1401 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1401 may store a program. When the program stored in the memory 1401 is executed by the processor 1402, the processor 1402 and the communication interface 1403 are used to execute one or more steps in the above-mentioned target detection method embodiment.

[0237] The processor 1402 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, for executing relevant programs to implement the functions required to be executed by the acquisition module, processing module, generation module and detection module in the target detection device in the above embodiment, or to execute one or more steps in the method embodiment of the present application. The steps of the method disclosed in the embodiment of the present application can be executed by a hardware decoding processor, or by hardware and / or software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1401, and the processor 1402 reads the information in the memory 1401, and executes one or more steps in the above target detection method embodiment in combination with its hardware.

[0238] The communication interface 1403 uses a transceiver device such as but not limited to a transceiver to implement communication between the chip 140 and other devices or communication networks.

[0239] The bus 1404 can realize the path for transmitting information between the various components of the chip 140 (for example, the memory 1401, the processor 1402 and the communication interface 1403). The bus 1404 can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0240] like Fig.15 As shown, in another embodiment of the present application, a vehicle 150 is also provided, the vehicle 150 comprising Fig.13 The target detection device shown or Fig.14 The chip shown.

[0241] Fig.16 , which is a possible logical structure diagram of a computing device cluster 170 provided in an embodiment of the present application. The computing device cluster 170 is used to implement the functions of the electronic device involved in any of the above embodiments. The computing device cluster 170 includes at least one computing device 160, and the computing device 160 includes: a memory 1601, a processor 1602, a communication interface 1603, and a bus 1604. Among them, the memory 1601, the processor 1602, and the communication interface 1603 are connected to each other through the bus 1604.

[0242] The memory 1601 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1601 may store a program. When the program stored in the memory 1601 is executed by the processor 1602, the processor 1602 and the communication interface 1603 are used to execute one or more steps in the above-mentioned target detection method embodiment.

[0243] The processor 1602 can be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components or any combination thereof, for executing relevant programs to implement the functions required to be executed by the acquisition module, processing module, generation module and detection module in the target detection device in the above embodiment, or to execute one or more steps in the method embodiment of the present application. The steps of the method disclosed in the embodiment of the present application can be executed by a hardware decoding processor, or by hardware and / or software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1601, and the processor 1602 reads the information in the memory 1601, and executes one or more steps in the above target detection method embodiment in combination with its hardware.

[0244] The communication interface 1603 uses a transceiver such as, but not limited to, a transceiver to implement communication between the computing device 160 and other devices or a communication network.

[0245] The bus 1604 can realize the path for transmitting information between the various components of the computing device 160 (for example, the memory 1601, the processor 1602, and the communication interface 1603). The bus 1604 can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Fig.16 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0246] In another embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the above Fig.14 or Fig.16 The steps performed by the processor in .

[0247] In another embodiment of the present application, a computer program product is also provided. The computer program product includes a computer-executable instruction. When the processor executes the computer-executable instruction, the above-mentioned Fig.14 or Fig.16 The steps performed by the processor in .

[0248] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0249] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0250] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0251] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0252] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire first input data, where the first input data is data of a first modality; Obtaining a first feature of the first input data through a first feature extraction network, and extracting data of an interested portion from the first input data as first segmentation data; Obtaining, by generating a network, a second feature according to the first feature and the first segmentation data, wherein the second feature includes an enhanced feature relative to the first segmentation data; According to the first feature and the second feature, a target detection result on the first input data is obtained through a detection network.

2. The method according to claim 1, characterized in that The step of obtaining the second feature by generating a network according to the first feature and the first segmentation data includes: Performing super-resolution processing on the first segmented data to obtain first segmented data after super-resolution processing; A second feature is obtained through the generation network according to the first feature and the first segmentation data after the super-resolution processing.

3. The method according to claim 1 or 2, characterized in that: The step of extracting the data of the portion of interest from the first input data as the first segmented data comprises: determining an interesting portion from the first feature as a first sub-feature; Data of an interested portion in the first input data is determined according to the first sub-feature as first segmented data.

4. The method according to claim 3, characterized in that The obtaining, by means of a detection network, a target detection result on the first input data according to the first feature and the second feature, comprises: fusing the first sub-feature and the second feature to obtain a third feature; According to the third feature, a target detection result regarding the first input data is obtained through a detection network.

5. The method according to claim 4, characterized in that According to the third feature, obtaining a target detection result about the first input data through a detection network includes: Replacing the first sub-feature in the first feature with the third feature to obtain the replaced first feature; According to the replaced first feature, a target detection result regarding the first input data is obtained through the detection network.

6. The method according to claim 5, characterized in that The number of the first features is multiple; The step of obtaining, by means of the detection network, a target detection result on the first input data according to the replaced first feature comprises: After obtaining the replaced first features respectively corresponding to each of the first features, a target detection result on the first input data is obtained through the detection network according to each of the replaced first features.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire second input data, wherein the first input data and the second input data include the same portion of interest; Obtaining a fourth feature of the second input data through a second feature extraction network; The step of obtaining the second feature by generating a network according to the first feature and the first segmentation data includes: The second feature is obtained through the generative network according to the first feature, the first segmentation data and the fourth feature.

8. The method according to claim 7, characterized in that The method further comprises: Extracting data of a portion of interest from the second input data as second segmented data; According to the fourth feature and the second segmentation data, a fifth feature is obtained through the generation network, wherein the fifth feature includes an enhanced feature relative to the second segmentation data; According to the fourth feature and the fifth feature, a target detection result regarding the second input data is obtained through the detection network.

9. The method according to claim 8, characterized in that The method further comprises: An object detection result of the input data is obtained according to the object detection result on the first input data and / or the object detection result on the second input data.

10. A target detection device, characterized in that: include: An acquisition module, used to acquire first input data, where the first input data is data of a first modality; A processing module, configured to obtain a first feature of the first input data through a first feature extraction network, and extract data of an interested portion from the first input data as first segmented data; A generating module, configured to obtain a second feature by generating a network according to the first feature and the first segmentation data, wherein the second feature includes an enhanced feature relative to the first segmentation data; A detection module is used to obtain a target detection result about the first input data through a detection network according to the first feature and the second feature.

11. The device according to claim 10, characterized in that The generating module is used to: perform super-resolution processing on the first segmented data to obtain first segmented data after super-resolution processing; A second feature is obtained through the generation network according to the first feature and the first segmentation data after the super-resolution processing.

12. The device according to claim 10 or 11, characterized in that The processing module is used for: determining an interesting portion from the first feature as a first sub-feature; Data of an interested portion in the first input data is determined according to the first sub-feature as first segmented data.

13. The device according to claim 12, characterized in that The detection module is used to: fuse the first sub-feature and the second feature to obtain a third feature; According to the third feature, a target detection result regarding the first input data is obtained through a detection network.

14. The device according to claim 13, characterized in that The detection module is used to: replace the first sub-feature in the first feature with the third feature to obtain the replaced first feature; According to the replaced first feature, a target detection result regarding the first input data is obtained through the detection network.

15. The device according to any one of claims 10 to 14, characterized in that: The acquisition module is further used to: acquire second input data, wherein the first input data and the second input data include the same portion of interest; The processing module is further used to: obtain a fourth feature of the second input data through a second feature extraction network; The generating module is further used to obtain the second feature through the generating network according to the first feature, the first segmentation data and the fourth feature.

16. A chip, characterized in that: The chip includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method described in any one of claims 1-9.

17. A vehicle, characterized in that: The vehicle comprises the device according to any one of claims 10 to 15 or the chip according to claim 16.

18. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, and the at least one processor executes the instructions to implement the steps of the method described in any one of claims 1-9.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images