Object detection methods, model training methods, and autonomous vehicles

Abstract

The present disclosure provides a target detection method, a training method for a deep learning model, an apparatus, an electronic device, a storage medium, a program product, and an autonomous driving vehicle, which relate to the field of artificial intelligence technology, and in particular to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, and can be applied to scenes such as autonomous driving. The specific implementation scheme is: extracting features from the image to be detected in the image sequence to obtain an image feature sequence; performing visual interaction processing on the image feature sequence and the query matrix to obtain a bird's-eye view feature; performing target detection on the bird's-eye view feature to obtain an initial target detection result; based on the initial target detection result, deleting the query element in the query matrix to obtain a lightweight query matrix; and obtaining a target detection result based on the lightweight query matrix and the image feature sequence.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large models and other technical fields, which can be applied to scenarios such as autonomous driving, and especially to target detection methods, deep learning model training methods, devices, electronic devices, storage media, program products and autonomous driving vehicles. Background Art

[0002] Computer vision technology offers great potential for improving image processing capabilities. Computer vision is a science that studies how to use electronic devices to "see", that is, using cameras and computers to replace human eyes to identify, track and measure targets. Computer vision technology provides great help for the development of applications in public security, information security, financial security and driving safety. Summary of the invention

[0003] The present disclosure provides a target detection method, a deep learning model training method, a device, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of the present disclosure, a target detection method is provided, comprising: extracting features from an image to be detected in a sequence of images to be detected to obtain an image feature sequence, wherein the sequence of images to be detected includes a plurality of images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes a plurality of image features arranged in a predetermined orientation order; performing visual interaction processing on the image feature sequence and a query matrix to obtain a bird's-eye view feature, wherein the query matrix includes a plurality of query elements, and the query elements are used to represent grid space features of the predetermined space; performing target detection on the bird's-eye view feature to obtain an initial target detection result; based on the initial target detection result, deleting the query elements in the query matrix to obtain a lightweight query matrix; and obtaining a target detection result based on the lightweight query matrix and the image feature sequence.

[0005] According to another aspect of the present disclosure, a training method for a deep learning model is provided, comprising: extracting features from sample images in a sample image sequence to obtain a sample image feature sequence, wherein the sample image sequence includes a plurality of sample images for representing different orientation information of a predetermined space, and the sample image feature sequence includes a plurality of sample image features arranged in a predetermined orientation order; performing visual interaction processing on the sample image feature sequence and a sample query matrix to obtain sample bird's-eye view features, wherein the sample query matrix includes a plurality of sample query elements, and the sample query elements are used to represent grid space features of the predetermined space; performing target detection on the sample bird's-eye view features to obtain sample initial target detection results; based on the sample initial target detection results, deleting query elements in the sample query matrix to obtain a lightweight sample query matrix; based on the lightweight sample query matrix and the sample image feature sequence, obtaining a sample target detection result; and training a deep learning model based on the sample target detection result and the sample label to obtain a trained deep learning model.

[0006] According to an embodiment of the present disclosure, a target detection device includes: a feature extraction module, which is used to extract features from an image to be detected in an image sequence to obtain an image feature sequence, wherein the image sequence to be detected includes multiple images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes multiple image features arranged in a predetermined orientation order; a visual interaction module, which is used to perform visual interaction processing on the image feature sequence and a query matrix to obtain a bird's-eye view feature, wherein the query matrix includes multiple query elements, and the query elements are used to represent the grid space features of the predetermined space; an initial detection module, which is used to perform target detection on the bird's-eye view feature to obtain an initial target detection result; a deletion module, which is used to delete the query elements in the query matrix based on the initial target detection result to obtain a lightweight query matrix; and a target detection module, which is used to obtain a target detection result based on the lightweight query matrix and the image feature sequence.

[0007] According to another aspect of the present disclosure, a training device for a deep learning model is provided, comprising: a sample feature extraction module, used to extract features from sample images in a sample image sequence to obtain a sample image feature sequence, wherein the sample image sequence includes a plurality of sample images for representing different orientation information of a predetermined space, and the sample image feature sequence includes a plurality of sample image features arranged in a predetermined orientation order; a sample visual interaction module, used to perform visual interaction processing on the sample image feature sequence and a sample query matrix to obtain a sample bird's-eye view feature, wherein the sample query matrix includes a plurality of sample query elements, and the sample query elements are used to represent the grid space features of the predetermined space; a sample initial detection module, used to perform target detection on the sample bird's-eye view feature to obtain a sample initial target detection result; a sample deletion module, used to delete the query elements in the sample query matrix based on the sample initial target detection result to obtain a lightweight sample query matrix; a sample target detection module, used to obtain a sample target detection result based on the lightweight sample query matrix and the sample image feature sequence; and a model training module, used to train a deep learning model based on the sample target detection result and the sample label to obtain a trained deep learning model.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as disclosed in the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0011] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: an electronic device.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1An exemplary system architecture to which the target detection method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0015] Figure 2 A flowchart of a target detection method according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 3 A flowchart of acquiring a sequence of images to be detected according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 4 A schematic diagram of determining a bird's-eye view sub-feature according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 5 A schematic diagram schematically shows a flow chart of a target detection method according to another embodiment of the present disclosure;

[0019] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 7 A block diagram of a target detection device according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 8 A block diagram schematically shows a training device for a deep learning model according to an embodiment of the present disclosure; and

[0022] Fig. 9 A block diagram of an electronic device suitable for implementing a target detection method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] With the rapid development and widespread application of autonomous driving technology, the demand for key technologies such as precise positioning and environmental perception of autonomous driving vehicles is becoming more and more urgent. In the application scenarios of autonomous driving, autonomous driving vehicles need to accurately detect and locate surrounding vehicles, pedestrians, and road traffic equipment such as guardrails and cones. On autonomous driving vehicles, multiple information acquisition devices, such as cameras, are often installed to obtain multiple images from different perspectives. Obstacle detection with three-dimensional surround view is achieved through multiple images from different perspectives. Specifically, obstacle detection with three-dimensional surround view can also be called multi-view 3D object detection, or BEV obstacle detection (bird's-eye-view detection).

[0025] The models used by most of the currently available target detection methods have a large number of model parameters, high computational complexity, and slow latency, making them difficult to be directly put into practical applications.

[0026] The present disclosure provides a target detection method, a training method for a deep learning model, a device, an electronic device, a storage medium, and a program product. The target detection method provided by the present disclosure can solve the above technical problems.

[0027] According to an embodiment of the present disclosure, a target detection method may include: extracting features from an image to be detected in a sequence of images to be detected to obtain an image feature sequence. The sequence of images to be detected includes a plurality of images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes a plurality of image features arranged in a predetermined orientation order. The image feature sequence and a query matrix are visually interactively processed to obtain a bird's-eye view feature. The query matrix includes a plurality of query elements, and the query elements are used to represent the grid space features of the predetermined space. Target detection is performed on the bird's-eye view feature to obtain an initial target detection result. Based on the initial target detection result, the query elements in the query matrix are deleted to obtain a lightweight query matrix. Based on the lightweight query matrix and the image feature sequence, a target detection result is obtained.

[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0029] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] Figure 1 An exemplary system architecture to which the target detection method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0031] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the target detection method and apparatus can be applied may include a terminal device, but the terminal device may implement the target detection method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include an autonomous driving vehicle 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the autonomous driving vehicle 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, and the like.

[0033] The autonomous driving vehicle 101 can be a truck, a car, a motorcycle, a tricycle, etc. As long as it is an autonomous driving vehicle equipped with a vehicle terminal, an information collection device, a positioning device, a ranging radar, an acceleration sensor, etc. The vehicle terminal can be, for example, an ECU (Electronic Control Unit) electronic controller unit. The vehicle terminal can establish a communication connection with the information collection device and the positioning device through CAN (Controller Area Network) to control the information collection device, the positioning device, the ranging radar, and the acceleration sensor, etc.

[0034] The server 103 may be a server that provides various services, such as a background management server that provides support for the autonomous driving vehicle (for example only). The background management server may be connected to a positioning device for the autonomous driving vehicle to obtain location information and map information of the autonomous driving vehicle 101.

[0035] It should be noted that the target detection method provided in the embodiment of the present disclosure can generally be performed by an autonomous driving vehicle. Accordingly, the target detection device provided in the embodiment of the present disclosure can also be set in the autonomous driving vehicle 101.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0038] Figure 2 The flowchart of the target detection method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method includes operations S210 to S250.

[0040] In operation S210, feature extraction is performed on the image to be detected in the image sequence to obtain an image feature sequence.

[0041] According to an embodiment of the present disclosure, the sequence of images to be detected may include a plurality of images to be detected for representing different orientations of a predetermined space.

[0042] For example, the sequence of images to be detected includes images P1, P2, P3, and P4 to be detected. Image P1 to be detected may be an image to be detected that collects spatial information about the due north direction of a predetermined space. Image P2 to be detected is an image to be detected that collects spatial information about the due east direction of a predetermined space. Image P3 to be detected may be an image to be detected that collects spatial information about the due south direction of a predetermined space. Image P4 to be detected is an image to be detected that collects spatial information about the due west direction of a predetermined space.

[0043] According to an embodiment of the present disclosure, a plurality of images to be detected may be arranged in a predetermined orientation sequence to form a sequence of images to be detected.

[0044] For example, the images are arranged in the order of north, east, south and west, and the obtained sequence of images to be detected includes images to be detected P1, P2, P3 and P4.

[0045] According to an embodiment of the present disclosure, a sequence of images to be detected arranged in a predetermined orientation order can be input into an image feature extraction module to obtain an image feature sequence. The image feature sequence includes multiple image features arranged in a predetermined orientation order. The multiple image features in the image feature sequence correspond one-to-one to the multiple images to be detected in the sequence of images to be detected.

[0046] According to an embodiment of the present disclosure, the image feature extraction module (Backbone Network) may include: a lightweight network model, such as Vit-Tiny (Vision Transformers Tiny, a lightweight visual codec) or Resnet-18 (residual network), etc.

[0047] According to the embodiments of the present disclosure, a lightweight network model is used to extract image features of an image sequence to be detected, which can reduce hardware energy consumption and improve operating efficiency while ensuring extraction accuracy.

[0048] In operation S220, visual interaction processing is performed on the image feature sequence and the query matrix to obtain a visual feature.

[0049] According to an embodiment of the present disclosure, a query matrix (grid-shaped bird's-eye-view detection queries) may include a plurality of query elements, each of which is used to characterize a grid space feature of a predetermined space.

[0050] According to an embodiment of the present disclosure, a predetermined space may be rasterized to obtain a plurality of raster spaces. Each query element may represent a raster space feature. The dimension of the query matrix may be set to be the same as the dimension of the image feature sequence by rasterization. Thus, the query matrix is ​​applied to the visual decoding process.

[0051] According to an embodiment of the present disclosure, each query element in the query matrix can be used to characterize grid space features, but it is not limited thereto. Each query element in the query matrix can also be used to characterize object features of each object.

[0052] According to an embodiment of the present disclosure, visual interaction processing of the query matrix and the image feature sequence can help highlight the differences between multiple target objects of different categories in the image feature sequence, thereby improving the richness and pertinence of the bird's-eye view features.

[0053] In operation S230, target detection is performed on the bird's-eye view feature to obtain an initial target detection result.

[0054] According to an embodiment of the present disclosure, performing target detection on the bird's-eye view feature to obtain an initial target detection result may include: performing multi-target detection and tracking on the bird's-eye view feature to obtain an initial target detection result. Specifically, the bird's-eye view feature may be input into a classification branch to obtain an initial category confidence. The bird's-eye view feature is input into a regression branch to obtain an initial position detection result. The initial target detection result may include at least one of the following: an initial category confidence and an initial position detection result.

[0055] In operation S240 , based on the initial target detection result, query elements in the query matrix are deleted to obtain a lightweight query matrix.

[0056] According to an embodiment of the present disclosure, based on the initial target detection result, deleting the query elements in the query matrix to obtain a lightweight query matrix may include: determining the initial target detection result matching the background information from a predetermined space based on the initial target detection result. Based on the initial target detection result matching the background information, deleting the query elements in the query matrix used to characterize the background position information or the grid space features used to characterize the background to obtain a lightweight query matrix.

[0057] In operation S250, a target detection result is obtained based on the lightweight query matrix and the image feature sequence.

[0058] According to an embodiment of the present disclosure, obtaining a target detection result based on a lightweight query matrix and an image feature sequence may include: performing target detection on the lightweight query matrix and the image feature sequence to obtain a target detection result. However, it is not limited thereto. It may also include: performing multiple visual interaction processes on the lightweight query matrix and the image feature sequence to obtain a deep bird's-eye view feature. Performing target detection on the deep bird's-eye view feature to obtain a target detection result.

[0059] According to other embodiments of the present disclosure, obtaining a target detection result based on a lightweight query matrix and an image feature sequence may include: performing target detection on the lightweight query matrix and the bird's-eye view feature to obtain a target detection result. However, it is not limited thereto. It may also include: performing multiple visual interaction processes on the lightweight query matrix and the bird's-eye view feature to obtain a deep bird's-eye view feature. Performing target detection on the deep bird's-eye view feature to obtain a target detection result.

[0060] According to the embodiments of the present disclosure, target detection is performed on a lightweight query matrix and a bird's-eye view feature to obtain a target detection result, and the deep semantic features in the bird's-eye view feature can be used to improve the accuracy of the target detection result.

[0061] According to other embodiments of the present disclosure, the initial target detection result may be used as the target detection result. Compared with the target detection method that uses the initial target detection result as the target detection result, the target detection method that obtains the target detection result based on the lightweight query matrix and the image feature sequence can interact the deep semantic features through multiple processing between the query matrix and the image feature sequence, so that the target detection result is accurate and effective.

[0062] According to an embodiment of the present disclosure, the image feature sequence and the query matrix are subjected to visual interaction processing to obtain a bird's-eye view feature, which is then processed using only the decoded visual interaction processing operation. While simplifying the processing operation, the query elements in the query matrix are deleted based on the initial target detection result, and the query matrix is ​​lightweight processed. In the subsequent process of performing deep feature processing based on the lightweight query matrix and image feature sequence, the amount of calculation is reduced while the richness of the features retained in the image feature sequence is improved, thereby reducing the hardware performance requirements while improving the accuracy and efficiency of target detection, and providing support for the direct application of the target detection method to terminal devices.

[0063] According to the relevant example, target detection can be performed based on the depth prediction method, for example, the depth information of each pixel in the image to be detected is predicted, and then each image to be detected is individually "Lit (extracted)" to obtain a feature frustum. For each camera, all feature frustums are "Splat (mapped)" into the rasterized bird's-eye view grid to form the BEV feature.

[0064] According to another related example, the depth information may not be predicted, and the image features may be directly converted into BEV space features based on the mapping relationship between the pixels and the 3D space.

[0065] According to another related example, a target detection method based on Transformer (encoder-decoder) can also be provided. The attention mechanism in Transformer is used to automatically extract BEV features. Specifically, a processing model of backbone (backbone network) + encoder (encoder) + decoder (decoder) can be constructed to process the image sequence to be detected to obtain the target detection result. Backbone is used to extract image features of multiple images to be detected from different perspectives, the encoder uses the transformer structure to extract BEV features, and the decoder uses the transformer structure to predict the target detection results based on the BEV features.

[0066] According to the embodiments of the present disclosure, the above-mentioned related examples have the technical problems of large amount of calculation, high hardware performance requirements, and difficulty in applying to terminal devices. Compared with the above-mentioned related examples, the target detection method provided by the embodiments of the present disclosure reduces the amount of calculation and the energy consumption of hardware while improving the accuracy and efficiency of target detection.

[0067] According to the embodiments of the present disclosure, the target detection method provided by the embodiments of the present disclosure can be applied to the field of autonomous driving technology. For example, in an autonomous driving vehicle, a multi-view Figure 3The three-dimensional target detection method helps autonomous vehicles accurately identify and locate targets such as pedestrians, vehicles, obstacles on the road, and provide them to the autonomous vehicles so that they can make driving decisions and control.

[0068] According to the embodiments of the present disclosure, the target detection method provided by the embodiments of the present disclosure can also be applied to the field of industrial automation technology. Figure 3 3D object detection methods are used for precise positioning and control of robots, as well as for quality inspection and classification of products on production lines.

[0069] According to the embodiments of the present disclosure, the target detection method provided by the embodiments of the present disclosure can also be applied to the field of security monitoring technology. Figure 3 D target detection methods are used to monitor the flow of people and vehicles in public places, airports, railway stations, etc., and provide real-time security alerts and warnings.

[0070] According to the embodiments of the present disclosure, the target detection method provided by the embodiments of the present disclosure can also be applied to the field of virtual reality technology. Figure 3 3D object detection methods are used to realize virtual scenes, allowing users to have more natural human-computer interaction with objects in the virtual environment.

[0071] According to an embodiment of the present disclosure, when executing Figure 2 In the operation S210 shown, before extracting features from the image to be detected in the image sequence to obtain the image feature sequence, the target detection method may further include an operation of acquiring the image sequence to be detected using a plurality of information acquisition devices with different viewing angles arranged in a predetermined space.

[0072] According to an embodiment of the present disclosure, the predetermined space may include: a cylindrical space centered on the execution body.

[0073] Figure 3 The flowchart of acquiring a sequence of images to be detected according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 3 As shown, taking the autonomous driving vehicle 310 as an example, the execution subject can be equipped with 6 cameras, and the installation positions include the front, left front, left rear, rear, right rear, and right front positions of the autonomous driving vehicle. These cameras are used to collect position information of different perspectives of a predetermined space of the autonomous driving vehicle, such as the front, left front, left rear, rear, right rear, and right front positions, to obtain image sequences P1, P2, P3, P4, P5, and P6 to be detected.

[0075] According to the embodiments of the present disclosure, Figure 2 Operation S220 shown, performing visual interaction processing on the image feature sequence and the query matrix to obtain a bird's-eye view feature, may include the following operations.

[0076] For example, based on the orientation information represented by the image to be detected in the image sequence to be detected, the target image feature and the target query sub-matrix matching the target image feature are determined from the image feature sequence. The target image feature and the target query sub-matrix are visually interactively processed to obtain a bird's-eye view sub-feature. Based on multiple bird's-eye view sub-features, a bird's-eye view feature is obtained.

[0077] According to an embodiment of the present disclosure, the orientation information represented by each image to be detected in the sequence of images to be detected can be understood as visual information about the orientation collected by the information collection device. The query elements in the query matrix can be used to represent grid space features, such as grid space position features. The image feature information associated with the grid space features can be learned through the query elements in the query matrix during the process of visual interaction processing with the target image features. The query matrix can be split into multiple query sub-matrices according to the orientation of the predetermined space. For example, if the image to be detected represents the orientation information of the due south orientation of the predetermined space, the target image features obtained based on the image to be detected can be visually interactively processed with the target query sub-matrix used to represent the due south orientation to obtain the south orientation.

[0078] According to an embodiment of the present disclosure, visual interaction processing may include: processing by a decoder in a codec, but is not limited thereto, and may also be processing by a cross attention module in a decoder in the codec. Any means may be used to perform visual interaction processing on the target image features and the target query submatrix.

[0079] According to an embodiment of the present disclosure, obtaining a bird's-eye view feature based on multiple bird's-eye view sub-features may include: using the multiple bird's-eye view sub-features as the bird's-eye view feature. However, it is not limited thereto. It may also include: splicing or fusing the multiple bird's-eye view sub-features to obtain the bird's-eye view feature. It may also include: encoding the multiple bird's-eye view sub-features to obtain the bird's-eye view feature.

[0080] According to an embodiment of the present disclosure, encoding a plurality of bird's-eye view sub-features to obtain a bird's-eye view feature may include: inputting the plurality of bird's-eye view sub-features into a FFN (feed-forward network) module to obtain a bird's-eye view feature.

[0081] According to the embodiments of the present disclosure, by using the query matrix and the target image features for visual interaction processing, the query matrix can be used to perform deep feature interaction on the target image features, thereby obtaining more accurate and distinctive bird's-eye view features, thereby improving the accuracy of the target detection results.

[0082] According to an exemplary embodiment of the present disclosure, performing visual interaction processing on target image features and target query submatrix to obtain bird's-eye view subfeatures may include: using the target image features as key data and value data, respectively, using the target query submatrix as query data, performing cross-attention processing on decoding, and obtaining bird's-eye view subfeatures.

[0083] According to an embodiment of the present disclosure, the target image features are used as key data and value data, and the target query sub-matrix is ​​used as query data, and decoded cross-attention processing is performed to obtain bird's-eye view sub-features. The decoded cross-attention processing can improve the deep-level feature interaction while simplifying the calculation process, thereby improving the processing efficiency.

[0084] Figure 4 The figure schematically shows a schematic diagram of determining a bird's-eye view sub-feature according to an embodiment of the present disclosure.

[0085] like Figure 4 As shown, taking the execution subject as an autonomous driving vehicle as an example, three cameras are respectively set in front of the autonomous driving vehicle 410, and three cameras are respectively set behind the autonomous driving vehicle. A predetermined space is set with the autonomous driving vehicle as the center. Six cameras are used to collect spatial visual information of the front, left front, left rear, rear, right rear and right front positions of the predetermined space of the autonomous driving vehicle, and the image sequences P1, P2, P3, P4, P5, and P6 to be detected about the front, left front, left rear, rear, right rear and right front positions are obtained. Feature extraction is performed on the image sequence to be detected, and image feature sequences F1, F2, F3, F4, F5, and F6 about the front, left front, left rear, rear, right rear and right front positions are obtained.

[0086] like Figure 4 As shown, the query matrix can be split into query sub-matrices Q1, Q2, Q3, Q4 for left front, left rear, right rear and right front positions in the order of front, left front, left rear, rear, right rear and right front positions.

[0087] like Figure 4 As shown, the query submatrix Q1 of the left front position is used as the target query submatrix, the image features F1 of the front position, F2 of the left front position and F3 of the left rear position are used as target image features, and the target query submatrix and the target image features are visually interactively processed to obtain the bird's-eye view subfeatures.

[0088] Similarly, the query submatrix of the left rear position is used as the target query submatrix, the image features of the left front position, the image features of the rear position and the image features of the left rear position are used as the target image features, and the target query submatrix and the target image features are visually interactively processed to obtain the bird's-eye view subfeatures. Similarly, the query submatrix of the right front position, the image features of the front position, the image features of the right front position and the image features of the right rear position can also be used to obtain the bird's-eye view subfeatures. The query submatrix of the right rear position, the image features of the rear position, the image features of the right front position and the image features of the right rear position can also be used to obtain the bird's-eye view subfeatures. Based on multiple bird's-eye view subfeatures, a bird's-eye view feature is obtained.

[0089] According to an embodiment of the present disclosure, an association relationship between the orientation of the query sub-matrix and the orientation information of the image to be detected may be established in advance to generate an association relationship table as shown in Table 1 below.

[0090]

[0091] Table 1

[0092] According to an embodiment of the present disclosure, based on the orientation information represented by each image to be detected in the image sequence to be detected and the association relationship table, the target image features and the target query submatrix matching the target image features can be determined from the image feature sequence.

[0093] According to the embodiments of the present disclosure, each query submatrix is ​​used to represent a grid space feature of a predetermined space, so the query submatrix is ​​not associated with all images to be detected from all viewing angles. For example, the format space feature represented by the left front query submatrix has a large correlation with the images to be detected in the front, left front, and left rear directions, and a small correlation with the images to be detected in the right front, right rear, and rear directions.

[0094] According to an embodiment of the present disclosure, the query matrix is ​​split according to the orientation, and then visual interaction processing is performed with the target image features corresponding to the orientation, thereby reducing the amount of calculation while ensuring processing accuracy.

[0095] According to another exemplary embodiment of the present disclosure, Figure 2 The operation S220 shown, performing visual interaction processing on the image feature sequence and the query matrix to obtain the bird's-eye view feature, may include: using the target image feature as key data and value data, respectively, and using the query matrix as query data, performing cross-attention processing of decoding to obtain the bird's-eye view sub-feature. The bird's-eye view feature is obtained based on the bird's-eye view sub-feature sequence corresponding to the image feature sequence one by one.

[0096] According to another embodiment of the present disclosure, the target image features may be used as key data and value data respectively, and the entire query matrix may be used as query data to perform visual interaction processing to obtain bird's-eye view sub-features.

[0097] According to an embodiment of the present disclosure, the query submatrix is ​​smaller in matrix dimension than the query matrix, thereby reducing the amount of data processed by visual interaction. In addition, the query matrix is ​​processed in blocks according to the orientation, which can ensure the processing accuracy while reducing the amount of data processing.

[0098] According to an embodiment of the present disclosure, when executing Figure 2 Before the operation S220 shown, the target detection method may further include: training multiple learnable query elements to obtain an initial query matrix. However, it is not limited thereto. It may further include: performing self-attention processing on the initial query matrix to obtain a query matrix.

[0099] According to an embodiment of the present disclosure, performing self-attention processing on an initial query matrix to obtain a query matrix may include: using the initial query matrix as query data, key data, and value data, respectively, performing self-attention processing for decoding to obtain a query matrix.

[0100] According to an embodiment of the present disclosure, the initial query matrix includes a plurality of learnable query elements, and the learnable query elements can be obtained by training through a predetermined sample image sequence and a sample label matching the predetermined sample image sequence. By adjusting the parameters of the plurality of learnable query elements, the trained learnable query elements have spatial position feature information, thereby allowing the matrix with the learnable query elements to perform visual interaction processing with the image feature sequence, and fully obtaining a bird's-eye view feature map from a bird's-eye view perspective.

[0101] According to an optional embodiment of the present disclosure, the initial query matrix may be used as a query matrix to directly perform visual interaction processing with the image feature sequence to obtain a bird's-eye view feature.

[0102] Compared with the method of directly using the initial query matrix as the query matrix, the initial query matrix is ​​processed by self-attention to obtain the query matrix, which can reduce the amount of data while retaining the grid space characteristics of the learnable query elements, thereby improving the efficiency of subsequent processing.

[0103] According to the embodiments of the present disclosure, Figure 2 Operation S230 shown, performing target detection on the bird's-eye view feature to obtain an initial target detection result, may include the following operations.

[0104] For example, target category detection is performed on the bird's-eye view feature to obtain an initial category confidence. Target position detection is performed on the bird's-eye view feature to obtain an initial position detection result. Based on the initial category confidence and the initial position detection result, an initial target detection result is obtained.

[0105] According to the embodiments of the present disclosure, target category detection and target position detection may be performed on the bird's-eye view features respectively to obtain initial category confidence and initial position detection results.

[0106] According to an embodiment of the present disclosure, the initial category confidence can be used to characterize the confidence of the target object identified in the image to be detected. The initial category confidence can be represented by a numerical value. The larger the numerical value of the initial category confidence is, the more likely it is that the target object is an obstacle target object; otherwise, it is a background object.

[0107] According to an embodiment of the present disclosure, the initial position detection result can be used to characterize the position of the target object identified in the image to be detected. The initial position detection result can be represented by a detection frame. For example, a rectangular frame or a contour frame of the target object. The position information of the target object in the image to be detected can be located by the initial position detection result.

[0108] According to an embodiment of the present disclosure, in a target detection method, target detection is performed on a bird's-eye view feature to obtain an operation of obtaining an initial target detection result. The initial target detection result can be used to perform lightweight processing on a query matrix, thereby performing a dimensionality reduction operation on the query matrix to achieve a reduction in processing volume, simplification of processing, and improvement of processing efficiency while ensuring processing accuracy.

[0109] According to an embodiment of the present disclosure, the initial target detection result may include multiple initial category confidences and multiple initial position detection results corresponding one-to-one to the multiple initial category confidences, and the multiple initial position detection results correspond one-to-one to multiple query elements in the query matrix.

[0110] According to an embodiment of the present disclosure, for example Figure 2 The operation S240 shown, based on the initial target detection result, deleting the query elements in the query matrix to obtain a lightweight query matrix, may include: for each initial position detection result, determining the query elements to be deleted that match the initial position detection result from the query matrix. Based on the initial category confidence that matches the initial position detection result and a predetermined confidence threshold, determining the screening results of the query elements to be deleted. Based on multiple screening results, deleting the query elements in the query matrix to obtain a lightweight query matrix.

[0111] According to an embodiment of the present disclosure, determining the screening result of the query element to be deleted based on the initial category confidence and the predetermined confidence threshold that match the initial position detection result may include: comparing the initial category confidence with the predetermined confidence threshold to determine the screening result of the query element to be deleted. For example, the initial category confidence is subtracted from the predetermined confidence threshold, and the subtraction result is used as the screening result.

[0112] According to an embodiment of the present disclosure, based on multiple screening results, query elements in a query matrix are deleted to obtain a lightweight query matrix, which may include: for each screening result, when the screening result is used to characterize that the initial category confidence is greater than or equal to a predetermined confidence threshold, retaining the query element to be deleted. When the screening result is used to characterize that the initial category confidence is less than a predetermined confidence threshold, deleting the query element to be deleted, thereby obtaining a lightweight query matrix.

[0113] According to other embodiments of the present disclosure, multiple initial category confidences may be sorted in descending order to obtain a sorting result. Based on the sorting result, K target initial category confidences are selected. The query elements matching the K target initial category confidences are retained, and the query elements other than the query elements matching the K target initial category confidences in the query matrix are deleted to obtain a lightweight query matrix.

[0114] According to an embodiment of the present disclosure, the query element related to the background target object in the query matrix can be determined by the initial category confidence and the predetermined confidence threshold, and the query element can be deleted, which can improve the subsequent processing efficiency while ensuring the processing accuracy.

[0115] According to another embodiment of the present disclosure, Figure 2 The operation S240 shown, based on the initial target detection result, deletes the query elements in the query matrix to obtain a lightweight query matrix, and also includes: determining a query submatrix to be deleted that matches a predetermined orientation space in a predetermined space from the query matrix. Based on multiple initial position detection results, determine a predetermined number of target initial category confidences that match the predetermined orientation space from multiple initial category confidences. Based on the initial position detection results of each of the predetermined number of target initial category confidences, delete the query elements in the query submatrix to be deleted to obtain a lightweight query submatrix. Based on multiple lightweight query submatrices, a lightweight query matrix is ​​obtained.

[0116] According to an embodiment of the present disclosure, a predetermined space can be divided to obtain a plurality of predetermined orientation spaces. Each predetermined orientation space corresponds to a query submatrix to be deleted. Based on a plurality of initial position detection results, a plurality of initial category confidences matching the predetermined orientation space are determined from a plurality of initial category confidences. The plurality of initial category confidences are sorted to obtain a sorting result. Based on the sorting result, a predetermined number of target initial category confidences are determined from the plurality of initial category confidences.

[0117] For example, the execution subject is an autonomous driving vehicle, and the predetermined orientation space is the predetermined orientation space of the left front orientation. The query submatrix of the left front orientation can be used as the query submatrix to be deleted Q = {q i Based on the multiple initial position detection results, multiple initial category confidences Y={y i}. The query element q to be deleted in the query submatrix to be deleted i and the initial category confidence y i Corresponding.

[0118] For example, multiple initial category confidences of the predetermined orientation space are sorted in descending order to obtain a sorting result. Based on the sorting result, a screening operation (top-k selection) is performed to filter out the query elements to be deleted for characterizing the background objects from the query submatrix to be deleted, thereby achieving a lightweight query submatrix that retains a predetermined number of, for example, K query elements.

[0119] According to an embodiment of the present disclosure, a lightweight query matrix can be obtained based on multiple lightweight query sub-matrices. sel The processing method can be as shown in formula (1).

[0120] Q sel = {q i |q i ∈Top_k(Q)}; Formula (1)

[0121] For example, the query matrix may include 300 query elements. After lightweight processing, 4 lightweight query sub-matrices are obtained, and each lightweight query sub-matrix includes 300 / 4=75 query elements.

[0122] According to an embodiment of the present disclosure, a query matrix is ​​split using a predetermined orientation space to obtain multiple query sub-matrices to be deleted. Multiple query sub-matrices to be deleted can be processed in parallel to obtain multiple lightweight query sub-matrices, thereby ensuring that each sub-matrix to be queried is lightweight and uniform while improving processing efficiency.

[0123] According to the embodiments of the present disclosure, Figure 2 The operation S250 shown, obtaining the target detection result based on the lightweight query matrix and the image feature sequence, may include: obtaining the target bird's-eye view feature based on the lightweight query matrix and the image feature sequence. Performing target detection on the target bird's-eye view feature to obtain the target detection result.

[0124] According to an embodiment of the present disclosure, obtaining a bird's-eye view feature of a target based on a lightweight query matrix and an image feature sequence may include: performing visual interaction processing on the lightweight query matrix and the image feature sequence to obtain the bird's-eye view feature of the target.

[0125] For example, based on the orientation information represented by each image to be detected in the image sequence to be detected, a target image feature and a lightweight target query sub-matrix matching the target image feature are determined from the image feature sequence. The target image feature and the lightweight target query sub-matrix are visually interactively processed to obtain a target bird's-eye view sub-feature. Based on multiple target bird's-eye view sub-features corresponding one-to-one to the image feature sequence, a target bird's-eye view feature is obtained.

[0126] According to an embodiment of the present disclosure, the target image features in the image feature sequence can be used as key data and value data respectively, and the lightweight target query sub-matrix matching the target image features can be used as query data to perform cross-attention processing of decoding to obtain the target bird's-eye view sub-features. Based on multiple target bird's-eye view sub-feature sequences, the target bird's-eye view features are obtained.

[0127] According to the embodiments of the present disclosure, a lightweight query matrix is ​​used to perform visual interaction processing with an image feature sequence. The lightweight query matrix can be used to perform deep feature interaction on the image feature sequence, thereby retaining rich feature information in the image feature sequence while filtering out background information through the lightweight query matrix, thereby reducing the amount of data processing while improving the accuracy of target detection results.

[0128] According to other embodiments of the present disclosure, the target bird's-eye view features may be obtained based on the lightweight query matrix and the bird's-eye view features.

[0129] According to an embodiment of the present disclosure, obtaining a target bird's-eye view feature based on a lightweight query matrix and a bird's-eye view feature may include: performing visual interaction processing on the lightweight query matrix and the bird's-eye view feature to obtain the target bird's-eye view feature.

[0130] For example, based on the orientation information represented by each image to be detected in the image sequence to be detected, a bird's-eye view sub-feature and a lightweight target query sub-matrix matching the bird's-eye view sub-feature are determined from the bird's-eye view feature. The bird's-eye view sub-feature and the lightweight target query sub-matrix are visually interactively processed to obtain a target bird's-eye view sub-feature. Based on multiple target bird's-eye view sub-features corresponding one-to-one to the image feature sequence, a target bird's-eye view feature is obtained.

[0131] According to an embodiment of the present disclosure, the bird's-eye view sub-features in the bird's-eye view feature can be used as key data and value data respectively, and the lightweight target query sub-matrix matching the bird's-eye view sub-features can be used as query data to perform cross-attention processing of decoding to obtain the target bird's-eye view sub-features. Based on multiple target bird's-eye view sub-feature sequences, the target bird's-eye view feature is obtained.

[0132] According to the embodiments of the present disclosure, a lightweight query matrix is ​​used to perform visual interaction processing with a bird's-eye view feature. The lightweight query matrix can be used to perform deep feature interaction on the bird's-eye view feature, thereby filtering out background information through the lightweight query matrix while utilizing the deep semantic features in the bird's-eye view feature, thereby reducing the amount of data processing while improving the accuracy of the target detection results.

[0133] According to an exemplary embodiment of the present disclosure, obtaining a target bird's-eye view feature based on a lightweight query matrix and a bird's-eye view feature may include: performing self-attention processing on the lightweight query matrix to obtain a target query matrix. Performing visual interaction processing on the target query matrix and an image feature sequence to obtain a target bird's-eye view feature.

[0134] According to an embodiment of the present disclosure, performing self-attention processing on a lightweight query matrix to obtain a target query matrix may include: using the lightweight query matrix as query data, key data, and value data, respectively, performing decoding self-attention processing to obtain a target query matrix.

[0135] According to an embodiment of the present disclosure, a lightweight query matrix is ​​subjected to self-attention processing to obtain a target query matrix, which can reduce the amount of data while retaining the position information characteristics of the learnable query elements, thereby improving subsequent processing efficiency.

[0136] According to another exemplary embodiment of the present disclosure, obtaining a target bird's-eye view feature based on a lightweight query matrix and a bird's-eye view feature may include: performing self-attention processing on the lightweight query matrix to obtain a target query matrix. Performing visual interaction processing on the target query matrix and the bird's-eye view feature to obtain the target bird's-eye view feature.

[0137] According to another exemplary embodiment of the present disclosure, a target bird's-eye view feature is obtained based on a lightweight query matrix and a bird's-eye view feature, including: repeatedly performing the following operations until the number of repeated rounds reaches a predetermined round threshold: when it is determined that the L+1th round is less than the predetermined round threshold, self-attention processing is performed on the Lth round target query matrix to obtain the L+1th round target query matrix. L is an integer greater than or equal to 1. Visual interaction processing is performed on the L+1th round target query matrix and the Lth round bird's-eye view feature to obtain the L+1th round bird's-eye view feature. When it is determined that the L+1th round is equal to the predetermined round threshold, the target bird's-eye view feature is determined based on the L+1th round bird's-eye view feature.

[0138] According to an embodiment of the present disclosure, when it is determined that the L+1th round is less than a predetermined round threshold, L=L+1 may also be set.

[0139] According to an embodiment of the present disclosure, the predetermined round threshold may include 2, 3 or more.

[0140] According to the embodiments of the present disclosure, through repeated operations, the target query matrix and the bird's-eye view features can be subjected to multiple visual interaction processes to extract deep semantic features and improve the accuracy of the target bird's-eye view features.

[0141] According to an embodiment of the present disclosure, performing target detection on a target bird's-eye view feature to obtain a target detection result may include: performing target category detection on the target bird's-eye view feature to obtain a category confidence. Performing target position detection on the target bird's-eye view feature to obtain a position detection result. Obtaining a target detection result based on the category confidence and the position detection result.

[0142] According to the embodiments of the present disclosure, target detection with two branches, target category detection and target position detection, is performed on the target bird's-eye view features respectively, which can achieve accurate detection of multiple target objects, thereby making the obtained target detection results highly accurate while improving the richness of the target detection results.

[0143] Figure 5 A flowchart of a target detection method according to another embodiment of the present disclosure is schematically shown.

[0144] like Figure 5 As shown, you can use Figure 5 The target detection model shown executes the target detection method. The target detection model includes a feature extraction module M510, a multi-layer decoding module, a target detection module M530 and a screening module M540.

[0145] like Figure 5As shown, the image sequence 510 to be detected can be obtained, and the image sequence 510 to be detected is input into the feature extraction module M510 to obtain the image feature sequence 520. The initial query matrix 530 is input into the self-attention submodule M521 of the decoding module to obtain the query matrix. The image feature sequence 520 and the query matrix are input into the visual interaction submodule M522 of the decoding module to obtain the bird's-eye view feature. The bird's-eye view feature is input into the target detection module M530 to obtain the initial target detection result. The initial target detection result and the query matrix are input into the screening module M540 to obtain the lightweight query matrix 540. The lightweight query matrix 540 is input into the self-attention submodule M523 of the decoding module to obtain the target query matrix. The target query matrix and the image feature sequence 520 are input into the visual interaction submodule M524 of the decoding module to obtain the target bird's-eye view feature. When the predetermined round threshold N is 2, the target bird's-eye view feature can be input into the target detection module M530 to obtain the target detection result 550.

[0146] like Figure 5 As shown, when the predetermined round threshold N is greater than 2, for example, 3, the lightweight query matrix 540 is input into the self-attention submodule M523 of the decoding module to obtain the second round target query matrix. The second round target query matrix and the image feature sequence 520 are input into the visual interaction submodule M524 of the decoding module to obtain the second round bird's-eye view features. The second round target query matrix is ​​input into the self-attention submodule of the decoding module to obtain the third round target query matrix. The third round target query matrix and the second round bird's-eye view features are input into the visual interaction submodule of the decoding module to obtain the third round bird's-eye view features. Repeat until the predetermined round threshold is reached. The bird's-eye view features of the predetermined round threshold are used as the target bird's-eye view features. The target bird's-eye view features are input into the target detection module M530 to obtain the target detection result 550.

[0147] According to the embodiments of the present disclosure, Figure 5 The target detection model shown can be used as Figure 6 The deep learning model shown is trained using the training method.

[0148] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0149] like Figure 6 As shown, the method includes operations S610 to S660.

[0150] In operation S610, feature extraction is performed on sample images in a sample image sequence to obtain a sample image feature sequence. The sample image sequence includes a plurality of sample images for representing different orientation information of a predetermined space, and the sample image feature sequence includes a plurality of sample image features arranged in a predetermined orientation order.

[0151] In operation S620, visual interaction processing is performed on the sample image feature sequence and the sample query matrix to obtain the sample bird's-eye view feature. The sample query matrix includes a plurality of sample query elements, and the sample query elements are used to characterize the grid space feature of the predetermined space.

[0152] In operation S630, target detection is performed on the bird's-eye view features of the sample to obtain an initial target detection result of the sample.

[0153] In operation S640 , based on the sample initial target detection result, the query elements in the sample query matrix are deleted to obtain a lightweight sample query matrix.

[0154] In operation S650, a sample object detection result is obtained based on the lightweight sample query matrix and the sample image feature sequence.

[0155] In operation S660, a deep learning model is trained based on the sample target detection results and the sample labels to obtain a trained deep learning model.

[0156] According to the embodiments of the present disclosure, Figure 6 The sample image sequence, sample image feature sequence, sample bird's-eye view feature, sample initial target detection result, sample target detection result and the like are shown in FIG. Figure 2 The image sequence to be detected, image feature sequence, bird's-eye view features, initial target detection results, interpretation of target detection results and processing methods shown are the same or similar, only to reflect the differences in the embodiments, and will not be repeated here.

[0157] According to an embodiment of the present disclosure, based on the sample target detection results and the sample labels, training the deep learning model to obtain the trained deep learning model may include: the sample detection results and the sample labels may be input into a cross loss function to obtain a cross loss value. Based on the cross loss value, the parameters of the deep learning model are adjusted to obtain the trained deep learning model.

[0158] In addition, based on the sample target detection results and sample labels, training the deep learning model to obtain the trained deep learning model may also include: inputting the sample target detection results and sample labels into a first loss function to obtain a first loss value. Inputting the intermediate sample data and the intermediate sample labels into a second loss function to obtain a second loss value. Based on the first loss value and the second loss value, adjusting the parameters of the deep learning model to obtain the trained deep learning model.

[0159] According to an embodiment of the present disclosure, the intermediate sample data may include at least one of the following: a sample image feature sequence, a sample bird's-eye view feature, and a sample initial target detection result. The intermediate sample label may include at least one of the following: a sample image feature sequence label, a sample bird's-eye view feature label, and a sample initial target detection result label.

[0160] According to an embodiment of the present disclosure, a sample image feature sequence and a sample query matrix are subjected to visual interaction processing to obtain a bird's-eye view feature of the sample, which is then processed using only the decoded visual interaction processing operation. While simplifying the processing operation, the query elements in the sample query matrix are deleted based on the sample initial target detection result, and the sample query matrix is ​​subjected to lightweight processing. In the subsequent process of performing deep feature processing based on the lightweight sample query matrix and the sample image feature sequence, the amount of calculation is reduced while the richness of the features in the retained image feature sequence is increased, thereby reducing the hardware performance requirements while improving the training efficiency.

[0161] According to an embodiment of the present disclosure, when executing Figure 7 After or simultaneously with the operation S760 shown, the training method of the deep learning model may further include: adjusting the sample query elements in the sample query matrix based on the sample target detection results and the sample labels to obtain a trained sample query matrix.

[0162] According to an embodiment of the present disclosure, the sample detection result and the sample label may be input into a cross loss function to obtain a cross loss value, and the sample query element in the sample query matrix is ​​adjusted based on the cross loss value to obtain a trained sample query matrix.

[0163] According to an embodiment of the present disclosure, a training method for a deep learning model is used to adjust the parameters of a sample query element, so that the sample query element becomes a learnable element, and then the trained sample query matrix has the corresponding perception and learning ability for a predetermined space, and can indicate the corresponding grid space features, and then the trained sample query matrix is ​​used to process the image sequence to be detected, thereby improving the accuracy of the target detection results.

[0164] According to an embodiment of the present disclosure, visual interaction processing is performed on a sample image feature sequence and a sample query matrix to obtain a sample bird's-eye view feature, which may include: determining a sample target image feature and a sample target query sub-matrix matching the sample target image feature from the sample image feature sequence based on the orientation information represented by the sample image in the sample image sequence. Visual interaction processing is performed on the sample target image feature and the sample target query sub-matrix to obtain a sample bird's-eye view sub-feature. A sample bird's-eye view feature is obtained based on multiple sample bird's-eye view sub-features.

[0165] According to an embodiment of the present disclosure, visual interaction processing is performed on sample target image features and sample target query submatrix to obtain sample bird's-eye view sub-features, which may include: using sample target image features as key data and value data respectively, using the sample target query submatrix as query data, performing cross-attention processing on decoding, and obtaining sample bird's-eye view sub-features.

[0166] According to an embodiment of the present disclosure, the sample initial target detection result includes multiple sample initial category confidences and multiple sample initial position detection results corresponding one-to-one to the multiple sample initial category confidences, and the multiple sample initial position detection results correspond one-to-one to multiple sample query elements in the query matrix.

[0167] According to an embodiment of the present disclosure, based on the sample initial target detection result, the sample query elements in the sample query matrix are deleted to obtain a lightweight sample query matrix, which may include: for each sample initial position detection result, determining the sample query element to be deleted from the sample query matrix that matches the sample initial position detection result. Based on the sample initial category confidence and the sample predetermined confidence threshold that match the sample initial position detection result, determine the sample screening result of the sample query element to be deleted. Based on multiple sample screening results, the sample query elements in the sample query matrix are deleted to obtain a lightweight sample query matrix.

[0168] According to an embodiment of the present disclosure, based on the sample initial target detection results, the sample query elements in the sample query matrix are deleted to obtain a lightweight sample query matrix, which may include: determining a sample query submatrix to be deleted that matches a predetermined orientation space in a predetermined space from the sample query matrix. Based on multiple sample initial position detection results, a predetermined number of sample target initial category confidences that match the predetermined orientation space are determined from multiple sample initial category confidences. Based on the sample initial position detection results of each of the predetermined number of sample target initial category confidences, the sample query elements in the sample query submatrix to be deleted are deleted to obtain a lightweight sample query submatrix. Based on multiple lightweight sample query submatrices, a lightweight sample query matrix is ​​obtained.

[0169] According to an embodiment of the present disclosure, a sample target detection result is obtained based on a lightweight sample query matrix and an image feature sequence, including: obtaining a bird's-eye view feature of a sample target based on a lightweight sample query matrix and a sample image feature sequence. Performing target detection on the bird's-eye view feature of the sample target to obtain a sample target detection result.

[0170] According to an embodiment of the present disclosure, based on a lightweight sample query matrix and a sample image feature sequence, a sample target bird's-eye view feature is obtained, including: performing self-attention processing on the lightweight sample query matrix to obtain a sample target query matrix. Performing visual interaction processing on the sample target query matrix and the sample image feature sequence to obtain a sample target bird's-eye view feature.

[0171] According to an embodiment of the present disclosure, based on a lightweight sample query matrix and a sample image feature sequence, a sample target bird's-eye view feature is obtained, including: repeatedly performing the following operations until the number of repeated rounds reaches a predetermined round threshold: when it is determined that the L+1th round is less than the sample predetermined round threshold, self-attention processing is performed on the Lth round sample target query matrix to obtain the L+1th round sample target query matrix. L is an integer greater than or equal to 1. Visual interaction processing is performed on the L+1th round sample target query matrix and the Lth round sample bird's-eye view feature to obtain the L+1th round sample bird's-eye view feature; and when it is determined that the L+1th round is equal to the sample predetermined round threshold, the sample target bird's-eye view feature is determined based on the L+1th round sample bird's-eye view feature.

[0172] According to an embodiment of the present disclosure, target detection is performed on the bird's-eye view features of the sample target to obtain the sample target detection result, including: performing target category detection on the bird's-eye view features of the sample target to obtain the sample category confidence. Performing target position detection on the bird's-eye view features of the sample target to obtain the sample position detection result. Based on the sample category confidence and the sample position detection result, the sample target detection result is obtained.

[0173] According to an embodiment of the present disclosure, target detection is performed on the bird's-eye view feature of the sample to obtain the sample initial target detection result, including: performing target category detection on the bird's-eye view feature of the sample to obtain the sample initial category confidence. Performing target position detection on the bird's-eye view feature of the sample to obtain the sample initial position detection result. Based on the sample initial category confidence and the sample initial position detection result, the sample initial target detection result is obtained.

[0174] Figure 7 A block diagram of a target detection device according to an embodiment of the present disclosure is schematically shown.

[0175] like Figure 7 As shown, the target detection device 700 of this embodiment includes a feature extraction module 710, a visual interaction module 720, an initial detection module 730, a deletion module 740 and a target detection module 750.

[0176] The feature extraction module 710 is used to extract features from the images to be detected in the image sequence to obtain an image feature sequence, wherein the image sequence to be detected includes multiple images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes multiple image features arranged in a predetermined orientation order.

[0177] The visual interaction module 720 is used to perform visual interaction processing on the image feature sequence and the query matrix to obtain the bird's-eye view feature. The query matrix includes a plurality of query elements, and the query elements are used to characterize the grid space feature of the predetermined space.

[0178] The initial detection module 730 is used to perform target detection on the bird's-eye view feature to obtain an initial target detection result.

[0179] The deletion module 740 is used to delete the query elements in the query matrix based on the initial target detection result to obtain a lightweight query matrix.

[0180] The target detection module 750 is used to obtain target detection results based on a lightweight query matrix and an image feature sequence.

[0181] According to an embodiment of the present disclosure, the visual interaction module 720 includes a sub-matrix determination sub-module, a visual interaction sub-module and a feature determination sub-module.

[0182] The submatrix determination submodule is used to determine the target image features and the target query submatrix matching the target image features from the image feature sequence based on the orientation information represented by the image to be detected in the image sequence to be detected.

[0183] The visual interaction submodule is used to perform visual interaction processing on the target image features and the target query submatrix to obtain the bird's-eye view subfeatures.

[0184] The feature determination submodule is used to obtain a bird's-eye view feature based on a plurality of bird's-eye view sub-features.

[0185] According to an embodiment of the present disclosure, the visual interaction submodule includes a cross-attention unit.

[0186] The cross attention unit is used to take the target image features as key data and value data respectively, and the target query sub-matrix as query data, perform cross attention processing on decoding, and obtain the bird's-eye view sub-features.

[0187] According to an embodiment of the present disclosure, the initial target detection result includes multiple initial category confidences and multiple initial position detection results corresponding one-to-one to the multiple initial category confidences, and the multiple initial position detection results correspond one-to-one to multiple query elements in the query matrix.

[0188] The deletion module 740 includes an element determination submodule, a screening determination submodule, and a deletion submodule.

[0189] The element determination submodule is used to determine, for each initial position detection result, from the query matrix, a query element to be deleted that matches the initial position detection result.

[0190] The screening determination submodule is used to determine the screening results of the query elements to be deleted based on the initial category confidence matching the initial position detection result and a predetermined confidence threshold.

[0191] The deletion submodule is used to delete the query elements in the query matrix based on multiple screening results to obtain a lightweight query matrix.

[0192] According to an embodiment of the present disclosure, the initial target detection result includes multiple initial category confidences and multiple initial position detection results corresponding one-to-one to the multiple initial position detection results, and the multiple initial position detection results correspond one-to-one to the multiple query elements in the query matrix.

[0193] The deletion module 740 includes a query sub-matrix determination sub-module, a confidence determination sub-module, an element deletion sub-module and a lightweight sub-module.

[0194] The query sub-matrix determination sub-module is used to determine, from the query matrix, a query sub-matrix to be deleted that matches a predetermined orientation space in a predetermined space.

[0195] The confidence determination submodule is used to determine a predetermined number of target initial category confidences matching a predetermined orientation space from a plurality of initial category confidences based on a plurality of initial position detection results.

[0196] The element deletion submodule is used to delete the query elements in the query submatrix to be deleted based on the initial position detection results of the initial category confidences of a predetermined number of targets, so as to obtain a lightweight query submatrix.

[0197] The lightweight sub-module is used to obtain a lightweight query matrix based on multiple lightweight query sub-matrices.

[0198] According to an embodiment of the present disclosure, the target detection module 750 includes a target feature determination submodule and a target detection submodule.

[0199] The target feature determination submodule is used to obtain the target bird's-eye view features based on a lightweight query matrix and image feature sequence.

[0200] The target detection submodule is used to perform target detection on the target bird's-eye view features to obtain the target detection result.

[0201] According to an embodiment of the present disclosure, the target feature determination submodule includes a self-attention unit and a first visual interaction unit.

[0202] The self-attention unit is used to perform self-attention processing on the lightweight query matrix to obtain the target query matrix.

[0203] The first visual interaction unit is used to perform visual interaction processing on the target query matrix and the image feature sequence to obtain the target bird's-eye view features.

[0204] According to an embodiment of the present disclosure, the target feature determination submodule includes an operation repetition execution unit.

[0205] The operation repetition execution unit is used to repeatedly execute the following operations until the repetition round reaches a predetermined round threshold: when it is determined that the L+1th round is less than the predetermined round threshold, self-attention processing is performed on the Lth round target query matrix to obtain the L+1th round target query matrix, where L is an integer greater than or equal to 1. Visual interaction processing is performed on the L+1th round target query matrix and the Lth round bird's-eye view feature to obtain the L+1th round bird's-eye view feature. When it is determined that the L+1th round is equal to the predetermined round threshold, the target bird's-eye view feature is determined based on the L+1th round bird's-eye view feature.

[0206] According to an embodiment of the present disclosure, the target feature determination submodule includes a category detection unit, a position detection unit and a result determination unit.

[0207] The category detection unit is used to perform target category detection on the target bird's-eye view features to obtain category confidence.

[0208] The position detection unit is used to perform target position detection on the target bird's-eye view feature to obtain a position detection result.

[0209] The result determination unit is used to obtain the target detection result based on the category confidence and the position detection result.

[0210] According to an embodiment of the present disclosure, the initial detection module 730 includes a category detection submodule, a position detection submodule, and a result determination submodule.

[0211] The category detection submodule is used to detect the target category based on the bird's-eye view features and obtain the initial category confidence.

[0212] The position detection submodule is used to perform target position detection on the bird's-eye view feature to obtain an initial position detection result.

[0213] The result determination submodule is used to obtain the initial target detection result based on the initial category confidence and the initial position detection result.

[0214] According to an embodiment of the present disclosure, the target detection device also includes a self-attention module.

[0215] The self-attention module is used to perform self-attention processing on the initial query matrix to obtain the query matrix, wherein the initial query matrix includes multiple learnable query elements, and the learnable query elements are obtained by training a predetermined sample image sequence and a sample label matching the predetermined sample image sequence.

[0216] Figure 8 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0217] like Figure 8 As shown, the training device 800 of the deep learning model of this embodiment includes a sample feature extraction module 810, a sample visual interaction module 820, a sample initial detection module 830, a sample deletion module 840, a sample target detection module 850 and a model training module 860.

[0218] The sample feature extraction module 810 is used to extract features from sample images in a sample image sequence to obtain a sample image feature sequence, wherein the sample image sequence includes a plurality of sample images for representing different orientation information of a predetermined space, and the sample image feature sequence includes a plurality of sample image features arranged in a predetermined orientation order.

[0219] The sample visual interaction module 820 is used to perform visual interaction processing on the sample image feature sequence and the sample query matrix to obtain the sample bird's-eye view feature, wherein the sample query matrix includes multiple sample query elements, and the sample query elements are used to characterize the grid space feature of the predetermined space.

[0220] The sample initial detection module 830 is used to perform target detection on the sample bird's-eye view features to obtain a sample initial target detection result.

[0221] The sample deletion module 840 is used to delete the query elements in the sample query matrix based on the sample initial target detection result to obtain a lightweight sample query matrix.

[0222] The sample target detection module 850 is used to obtain a sample target detection result based on the lightweight sample query matrix and the sample image feature sequence.

[0223] The model training module 860 is used to train a deep learning model based on the sample target detection results and sample labels to obtain a trained deep learning model.

[0224] According to an embodiment of the present disclosure, the training device 800 for the deep learning model also includes an element adjustment module.

[0225] An element adjustment module is used to adjust the sample query elements in the sample query matrix based on the sample target detection results and sample labels to obtain a trained sample query matrix.

[0226] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle.

[0227] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as in the embodiment of the present disclosure.

[0228] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method as described in an embodiment of the present disclosure.

[0229] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0230] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0231] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0232] A number of components in the device 900 are connected to an input / output (I / O) interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0233] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the target detection method. For example, in some embodiments, the target detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the target detection method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the target detection method in any other appropriate manner (e.g., by means of firmware).

[0234] Based on the aforementioned electronic device, the electronic device can respond to the instruction information for the autonomous driving vehicle, extract features from the images to be detected in the image sequence to obtain an image feature sequence. The image sequence to be detected includes a plurality of images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes a plurality of image features arranged in a predetermined orientation order. The image feature sequence and the query matrix are visually interactively processed to obtain a bird's-eye view feature. The query matrix includes a plurality of query elements, and the query elements are used to represent the grid space features of the predetermined space. Target detection is performed on the bird's-eye view feature to obtain an initial target detection result. Based on the initial target detection result, the query elements in the query matrix are deleted to obtain a lightweight query matrix. Based on the lightweight query matrix and the image feature sequence, a target detection result is obtained.

[0235] Based on the aforementioned electronic device, the present disclosure also provides an autonomous driving vehicle, which may include the electronic device, and may also include a communication component, a display screen for realizing a human-machine interface, and an information collection device for collecting surrounding environment information, etc. The communication component, the display screen, the information collection device and the electronic device may be communicatively connected.

[0236] Among them, the electronic device can be integrated with the communication component, display screen, information collection device and electronic device, or can be separately arranged with the communication component, display screen, information collection device and electronic device.

[0237] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0238] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0239] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0240] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0241] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0242] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0243] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0244] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection method, comprising: Extracting features from an image to be detected in a sequence of images to be detected to obtain an image feature sequence, wherein the sequence of images to be detected includes a plurality of images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes a plurality of image features arranged in a predetermined orientation sequence; Performing visual interaction processing on the image feature sequence and a query matrix to obtain a bird's-eye view feature, wherein the query matrix includes a plurality of query elements, and the query elements are used to characterize the grid space feature of the predetermined space; Performing target detection on the bird's-eye view feature to obtain an initial target detection result; Based on the initial target detection result, the query elements in the query matrix are deleted to obtain a lightweight query matrix; and Based on the lightweight query matrix and the image feature sequence, a target detection result is obtained.

2. The method according to claim 1, wherein: The visual interaction processing of the image feature sequence and the query matrix to obtain the bird's-eye view features includes: Based on the orientation information represented by the image to be detected in the image sequence to be detected, determining a target image feature and a target query submatrix matching the target image feature from the image feature sequence; Performing visual interaction processing on the target image features and the target query sub-matrix to obtain bird's-eye view sub-features; and The bird's-eye view feature is obtained based on a plurality of the bird's-eye view sub-features.

3. The method according to claim 2, wherein: The step of performing visual interaction processing on the target image feature and the target query sub-matrix to obtain a bird's-eye view sub-feature includes: The target image features are used as key data and value data respectively, and the target query sub-matrix is ​​used as query data, and a cross-attention process of decoding is performed to obtain the bird's-eye view sub-features.

4. The method according to any one of claims 1 to 3, wherein: The initial target detection result includes a plurality of initial category confidences and a plurality of initial position detection results corresponding one-to-one to the plurality of initial category confidences, and the plurality of initial position detection results correspond one-to-one to a plurality of query elements in the query matrix; The step of deleting the query elements in the query matrix based on the initial target detection result to obtain a lightweight query matrix includes: For each of the initial position detection results, determining from the query matrix a to-be-deleted query element that matches the initial position detection result; Determining the screening result of the query element to be deleted based on the initial category confidence matching the initial position detection result and a predetermined confidence threshold; and Based on the plurality of screening results, query elements in the query matrix are deleted to obtain the lightweight query matrix.

5. The method according to any one of claims 1 to 3, wherein: The initial target detection result includes a plurality of initial category confidences and a plurality of initial position detection results corresponding one-to-one to the plurality of initial category confidences, and the plurality of initial position detection results correspond one-to-one to a plurality of query elements in the query matrix; The step of deleting the query elements in the query matrix based on the initial target detection result to obtain a lightweight query matrix includes: Determining from the query matrix a query sub-matrix to be deleted that matches a predetermined orientation space in the predetermined space; Based on the multiple initial position detection results, determining a predetermined number of target initial category confidences matching the predetermined orientation space from the multiple initial category confidences; Based on the initial position detection results of the predetermined number of target initial category confidences, the query elements in the to-be-deleted query sub-matrix are deleted to obtain the lightweight query sub-matrix; and The lightweight query matrix is ​​obtained based on the multiple lightweight query sub-matrices.

6. The method according to claim 1, wherein: The obtaining of the target detection result based on the lightweight query matrix and the image feature sequence includes: Obtaining a bird's-eye view feature of a target based on the lightweight query matrix and the image feature sequence; and Performing target detection on the target bird's-eye view feature to obtain the target detection result.

7. The method according to claim 6, wherein: The step of obtaining the target bird's-eye view features based on the lightweight query matrix and the image feature sequence includes: Performing self-attention processing on the lightweight query matrix to obtain a target query matrix; and Visual interaction processing is performed on the target query matrix and the image feature sequence to obtain the target bird's-eye view features.

8. The method according to claim 6, wherein: The step of obtaining the target bird's-eye view features based on the lightweight query matrix and the image feature sequence includes: Repeat the following operations until the number of repeated rounds reaches a predetermined round threshold; When it is determined that the L+1th round is less than the predetermined round threshold, self-attention processing is performed on the Lth round target query matrix to obtain the L+1th round target query matrix, where L is an integer greater than or equal to 1; Performing visual interaction processing on the L+1th round target query matrix and the Lth round bird's-eye view features to obtain the L+1th round bird's-eye view features; and When it is determined that the L+1th round is equal to the predetermined round threshold, the target bird's-eye view feature is determined based on the L+1th round bird's-eye view feature.

9. The method according to any one of claims 6 to 8, wherein: The performing target detection on the target bird's-eye view feature to obtain the target detection result includes: Performing target category detection on the target bird's-eye view features to obtain category confidence; Performing target position detection on the target bird's-eye view feature to obtain a position detection result; and The target detection result is obtained based on the category confidence and the position detection result.

10. The method according to any one of claims 1 to 3, wherein: The performing target detection on the bird's-eye view feature to obtain an initial target detection result includes: Performing target category detection on the bird's-eye view features to obtain initial category confidence; Performing target position detection on the bird's-eye view feature to obtain an initial position detection result; and The initial target detection result is obtained based on the initial category confidence and the initial position detection result.

11. The method according to any one of claims 1 to 3, further comprising: The query matrix is ​​obtained by performing self-attention processing on an initial query matrix, wherein the initial query matrix includes a plurality of learnable query elements, and the learnable query elements are obtained by training a predetermined sample image sequence and a sample label matching the predetermined sample image sequence.

12. A method for training a deep learning model, comprising: Extracting features from sample images in a sample image sequence to obtain a sample image feature sequence, wherein the sample image sequence includes a plurality of sample images for representing different orientation information of a predetermined space, and the sample image feature sequence includes a plurality of sample image features arranged in a predetermined orientation order; Performing visual interaction processing on the sample image feature sequence and a sample query matrix to obtain a sample bird's-eye view feature, wherein the sample query matrix includes a plurality of sample query elements, and the sample query elements are used to characterize the grid space feature of the predetermined space; Performing target detection on the bird's-eye view features of the sample to obtain an initial target detection result of the sample; Based on the initial target detection result of the sample, the query elements in the sample query matrix are deleted to obtain a lightweight sample query matrix; Obtaining a sample target detection result based on the lightweight sample query matrix and the sample image feature sequence; and Based on the sample target detection results and sample labels, a deep learning model is trained to obtain a trained deep learning model.

13. The method according to claim 12, further comprising: Based on the sample target detection results and sample labels, the sample query elements in the sample query matrix are adjusted to obtain a trained sample query matrix.

14. A target detection device, comprising: A feature extraction module, used for extracting features from an image to be detected in a sequence of images to be detected, to obtain an image feature sequence, wherein the sequence of images to be detected includes a plurality of images to be detected for representing different orientations of a predetermined space, and the image feature sequence includes a plurality of image features arranged in a predetermined orientation sequence; A visual interaction module, used for performing visual interaction processing on the image feature sequence and a query matrix to obtain a bird's-eye view feature, wherein the query matrix includes a plurality of query elements, and the query elements are used to characterize the grid space feature of the predetermined space; An initial detection module, used to perform target detection on the bird's-eye view feature to obtain an initial target detection result; a deletion module, configured to delete query elements in the query matrix based on the initial target detection result to obtain a lightweight query matrix; and The target detection module is used to obtain a target detection result based on the lightweight query matrix and the image feature sequence.

15. The device according to claim 14, wherein: The visual interaction module comprises: A submatrix determination submodule, used to determine target image features and a target query submatrix matching the target image features from the image feature sequence based on the orientation information represented by the image to be detected in the image sequence to be detected; a visual interaction submodule, configured to perform visual interaction processing on the target image features and the target query submatrix to obtain bird's-eye view subfeatures; and The feature determination submodule is used to obtain the bird's-eye view feature based on a plurality of the bird's-eye view sub-features.

16. The device according to claim 15, wherein: The visual interaction submodule comprises: The cross attention unit is used to use the target image features as key data and value data respectively, and the target query sub-matrix as query data, to perform cross attention processing of decoding, so as to obtain the bird's-eye view sub-features.

17. The device according to any one of claims 14 to 16, wherein: The initial target detection result includes a plurality of initial category confidences and a plurality of initial position detection results corresponding one-to-one to the plurality of initial category confidences, and the plurality of initial position detection results correspond one-to-one to a plurality of query elements in the query matrix; The deletion module comprises: An element determination submodule, configured to determine, for each of the initial position detection results, from the query matrix a query element to be deleted that matches the initial position detection result; a screening determination submodule, configured to determine the screening result of the query element to be deleted based on the initial category confidence matching the initial position detection result and a predetermined confidence threshold; and The deletion submodule is used to delete the query elements in the query matrix based on the plurality of screening results to obtain the lightweight query matrix.

18. The device according to any one of claims 14 to 16, wherein: The initial target detection result includes a plurality of initial category confidences and a plurality of initial position detection results corresponding one-to-one to the plurality of initial category confidences, and the plurality of initial position detection results correspond one-to-one to a plurality of query elements in the query matrix; The deletion module comprises: A query submatrix determination submodule, used to determine, from the query matrix, a query submatrix to be deleted that matches a predetermined orientation space in the predetermined space; A confidence determination submodule, configured to determine, based on the multiple initial position detection results, a predetermined number of target initial category confidences that match the predetermined orientation space from the multiple initial category confidences; an element deletion submodule, configured to delete the query elements in the query submatrix to be deleted based on the initial position detection results of the respective initial category confidences of the predetermined number of targets, so as to obtain the lightweight query submatrix; and The lightweight submodule is used to obtain the lightweight query matrix based on a plurality of the lightweight query sub-matrices.

19. The device according to claim 14, wherein: The target detection module comprises: a target feature determination submodule, configured to obtain a bird's-eye view feature of a target based on the lightweight query matrix and the image feature sequence; and The target detection submodule is used to perform target detection on the target bird's-eye view features to obtain the target detection result.

20. The device according to claim 19, wherein The target feature determination submodule includes: A self-attention unit, configured to perform self-attention processing on the lightweight query matrix to obtain a target query matrix; and The first visual interaction unit is used to perform visual interaction processing on the target query matrix and the image feature sequence to obtain the target bird's-eye view feature.

21. The device according to claim 19, wherein The target feature determination submodule includes: The repetitive execution unit is used to repeatedly execute the following operations until the number of repetitive rounds reaches a predetermined round threshold: When it is determined that the L+1th round is less than the predetermined round threshold, self-attention processing is performed on the Lth round target query matrix to obtain the L+1th round target query matrix, where L is an integer greater than or equal to 1; Performing visual interaction processing on the L+1th round target query matrix and the Lth round bird's-eye view features to obtain the L+1th round bird's-eye view features; and When it is determined that the L+1th round is equal to the predetermined round threshold, the target bird's-eye view feature is determined based on the L+1th round bird's-eye view feature.

22. The device according to any one of claims 19 to 21, wherein: The target feature determination submodule includes: A category detection unit, used to perform target category detection on the target bird's-eye view feature to obtain category confidence; a position detection unit, configured to perform target position detection on the target bird's-eye view feature to obtain a position detection result; and A result determination unit is used to obtain the target detection result based on the category confidence and the position detection result.

23. The device according to any one of claims 14 to 16, wherein: The initial detection module comprises: A category detection submodule, used to perform target category detection on the bird's-eye view feature to obtain an initial category confidence; a position detection submodule, configured to perform target position detection on the bird's-eye view feature to obtain an initial position detection result; and The result determination submodule is used to obtain the initial target detection result based on the initial category confidence and the initial position detection result.

24. The apparatus according to any one of claims 14 to 16, further comprising: The self-attention module is used to perform self-attention processing on an initial query matrix to obtain the query matrix, wherein the initial query matrix includes a plurality of learnable query elements, and the learnable query elements are obtained by training a predetermined sample image sequence and a sample label matching the predetermined sample image sequence.

25. A training device for a deep learning model, comprising: A sample feature extraction module, used for extracting features from sample images in a sample image sequence to obtain a sample image feature sequence, wherein the sample image sequence includes a plurality of sample images for representing different orientation information of a predetermined space, and the sample image feature sequence includes a plurality of sample image features arranged in a predetermined orientation order; A sample visual interaction module, used for performing visual interaction processing on the sample image feature sequence and a sample query matrix to obtain a sample bird's-eye view feature, wherein the sample query matrix includes a plurality of sample query elements, and the sample query elements are used to characterize the grid space feature of the predetermined space; A sample initial detection module is used to perform target detection on the bird's-eye view features of the sample to obtain a sample initial target detection result; A sample deletion module, used to delete the query elements in the sample query matrix based on the initial target detection result of the sample, so as to obtain a lightweight sample query matrix; A sample target detection module, used to obtain a sample target detection result based on the lightweight sample query matrix and the sample image feature sequence; and The model training module is used to train the deep learning model based on the sample target detection results and sample labels to obtain a trained deep learning model.

26. The apparatus according to claim 25, further comprising: An element adjustment module is used to adjust the sample query elements in the sample query matrix based on the sample target detection results and sample labels to obtain a trained sample query matrix.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

30. An autonomous driving vehicle comprising: The electronic device as claimed in claim 27.

Images