Object detection method and apparatus, and electronic device and storage medium

Abstract

Provided in the present description are an object detection method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring an image to be subjected to detection, and point cloud data matching said image; extracting image feature information of said image, and on the basis of the point cloud data, determining three-dimensional coordinate information of points; on the basis of the three-dimensional coordinate information, generating positional encoding feature information corresponding to feature points in the image feature information; and on the basis of the image feature information and the positional encoding feature information, determining an object in said image.

Description

Target detection method and device, electronic device, and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a target detection method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Target detection is a core task in computer vision, aiming to detect and locate specific target objects from images or videos. Unlike image classification, target detection not only identifies the class of target objects, but also accurately labels the location of target objects (usually represented by a bounding box). In the field of autonomous driving, target detection is mainly used to identify pedestrians, vehicles, traffic signs, traffic lights, road markings, obstacles, and the like on the road, to ensure that autonomous vehicles can perceive the surrounding environment in real time and thus make correct driving decisions.

[0003] Existing target detection methods mainly rely on 2D images for target detection. Since the target position information in a 2D image is limited to two-dimensional coordinates on the image plane, there is a lack of understanding of the spatial position and pose of the object, and it may be difficult to accurately detect such target objects when dealing with target objects with complex textures or non-obvious surface information.

[0004] SUMMARY

[0005] In view of this, embodiments of the present disclosure provide a target detection method, device, electronic device, and storage medium to solve the technical problems in the related art.

[0006] According to a first aspect of an embodiment of the present disclosure, a target detection method is provided, the method comprising:

[0007] obtaining a to-be-detected image and point cloud data matched with the to-be-detected image;

[0008] extracting image feature information of the to-be-detected image, and determining three-dimensional coordinate information of a point position based on the point cloud data; and generating position encoding feature information corresponding to a feature point in the image feature information based on the three-dimensional coordinate information;

[0009] determining a target object in the to-be-detected image based on the image feature information and the position encoding feature information.

[0010] According to a second aspect of an embodiment of the present disclosure, a target detection method is provided, the method comprising:

[0011] for a target vehicle equipped with an image acquisition device and a radar, obtaining a to-be-detected image acquired by the image acquisition device and point cloud data matched with the to-be-detected image acquired by the radar;

[0012] extract image feature information of the image to be detected, and determine three-dimensional coordinate information of the point positions around the target vehicle based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information;

[0013] determine a target object in the image to be detected based on the image feature information and the position coding feature information.

[0014] According to a third aspect of the embodiments of the present specification, a target detection device is provided, and the device comprises:

[0015] an acquisition module configured to acquire an image to be detected and point cloud data matched with the image to be detected;

[0016] a feature processing module configured to extract image feature information of the image to be detected, and determine three-dimensional coordinate information of point positions based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information;

[0017] a target detection module configured to determine a target object in the image to be detected based on the image feature information and the position coding feature information.

[0018] According to a fourth aspect of the embodiments of the present specification, a target detection device is provided, and the device comprises:

[0019] an acquisition module configured to, for a target vehicle equipped with an image acquisition device and a radar, acquire an image to be detected collected by the image acquisition device and point cloud data matched with the image to be detected collected by the radar;

[0020] a feature processing module configured to extract image feature information of the image to be detected, and determine three-dimensional coordinate information of point positions around the target vehicle based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information;

[0021] a target detection module configured to determine a target object in the image to be detected based on the image feature information and the position coding feature information.

[0022] According to a fifth aspect of the embodiments of the present specification, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method according to the first aspect or the second aspect.

[0023] According to a sixth aspect of the embodiments of the present specification, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the method according to the first aspect or the second aspect.

[0024] In the embodiments of the present specification, the real three-dimensional coordinate information is extracted from the point cloud data matched with the image to be detected, and the position coding feature information corresponding to the feature points of the image feature information of the image to be detected is generated. When determining the target object in the image to be detected, since the position coding feature information is introduced into the image feature information, and the position coding feature information can provide the real data of the depth and position of the feature points of the image, the target object with complex texture or non-obvious surface information can be accurately detected. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] FIG. 1 is a flowchart of a target detection method according to an exemplary embodiment of the present specification.

[0027] FIG. 2 is a schematic diagram of a target detection process according to an exemplary embodiment of the present specification.

[0028] FIG. 3 is a schematic diagram of extracting three-dimensional coordinate information according to an exemplary embodiment of the present specification.

[0029] FIG. 4 is a schematic diagram of a target detection module according to an exemplary embodiment of the present specification.

[0030] FIG. 5 is a flowchart of another target detection method according to an exemplary embodiment of the present specification.

[0031] FIG. 6 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification.

[0032] FIG. 7 is a block diagram of a target detection apparatus according to an exemplary embodiment of the present specification.

[0033] FIG. 8 is a block diagram of another target detection apparatus according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0034] With reference to the drawings and the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all the embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present disclosure.

[0035] The terms used in the embodiments of the present disclosure are merely for the purpose of describing the specific embodiments, and are not intended to limit the embodiments of the present disclosure. The singular forms "a", "an" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, without departing from the scope of the embodiments of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0037] For the purpose of brevity and ease of understanding, the terms "greater than" or "less than", "higher than" or "lower than" are used herein when characterizing the size relationship. However, it can be understood by those skilled in the art that the term "greater than" also covers the meaning of "greater than or equal to", and the term "less than" also covers the meaning of "less than or equal to"; the term "higher than" covers the meaning of "higher than or equal to", and the term "lower than" also covers the meaning of "lower than or equal to".

[0038] Object detection is a core task in computer vision, aiming to detect and locate specific target objects from images or videos. Unlike image classification, object detection not only identifies the class of target objects, but also accurately labels the location of target objects (usually represented by a bounding box). In the field of autonomous driving, object detection is mainly used to identify pedestrians, vehicles, traffic signs, signal lights, road markings, obstacles, etc. on the road, to ensure that autonomous vehicles can perceive the surrounding environment in real time and make correct driving decisions.

[0039] Existing object detection methods mainly rely on 2D images for object detection. Since the target position information in 2D images is limited to two-dimensional coordinates on the image plane, there is a lack of understanding of the spatial position and pose of the object, and when dealing with target objects with complex texture or non-obvious surface information, the detection performance may be poor.

[0040] To solve the above problems, the present specification provides a target detection method, which provides 3D point cloud data of a target object in a 2D image in a target detection method for the 2D image, and combines 2D image information with 3D point cloud data to improve the robustness and accuracy of target detection.

[0041] The target detection method disclosed in the present specification can be applied to an autonomous vehicle to assist the autonomous vehicle in identifying and locating vehicles, pedestrians, traffic signs, traffic lights, etc. on the road to ensure that the autonomous vehicle can make reasonable decisions based on the target detection result. Of course, the target detection method can also be applied to the application fields of security monitoring systems, unmanned aerial vehicles, etc. The present specification does not limit the application field of the target detection method.

[0042] As shown in FIG. 1, FIG. 1 is a flowchart of a target detection method according to an exemplary embodiment of the present specification, which includes steps 101-103:

[0043] Step 101: obtaining a to-be-detected image and point cloud data matched with the to-be-detected image.

[0044] Step 102: extracting image feature information of the to-be-detected image, and determining three-dimensional coordinate information of a point based on the point cloud data; and generating position coding feature information corresponding to a feature point in the image feature information based on the three-dimensional coordinate information.

[0045] Step 103: determining a target object in the to-be-detected image based on the image feature information and the position coding feature information.

[0046] The target of the target detection task is to identify the category and bounding box of the target object appearing in the to-be-detected image, and the bounding box is used to frame the position of the target object in the to-be-detected image. For example, in the target detection task of an unmanned vehicle, the to-be-detected image can be an image of the surrounding environment collected by the camera configured on the unmanned vehicle. The to-be-detected image can be a single image collected by one camera of the vehicle, or an image set composed of images of different angles collected by multiple cameras equipped on the vehicle. For example, the to-be-detected image can be an image frame extracted from a video, such as an image frame extracted as a to-be-detected image, or multiple image frames extracted continuously or discontinuously (such as only key frames) as to-be-detected images, etc. The to-be-detected image can be an RGB image, a grayscale image, or even an infrared image, depending on the type of camera used to collect the to-be-detected image.

[0047] The point cloud data can be collected by radar or LiDAR. The point cloud data contains various information of the collected object. The content of the point cloud data used in the method can include, but is not limited to, three-dimensional coordinate information of point cloud instances, point intensity values, colors, timestamps, point densities, and the like.

[0048] In the method, the relationship between each pair of the to-be-detected image and the point cloud data matched with the to-be-detected image can be the same environmental scene at the same time point. For example, the matching relationship between the to-be-detected image and the point cloud data can be determined by using the timestamps of the to-be-detected image and the point cloud data.

[0049] After obtaining the to-be-detected image, the image feature information of the to-be-detected image can be extracted. Any mature neural network can be used to extract the image feature information, such as ResNet, DarkNet, and the like. Of course, the neural network architecture for extracting the image feature information of the to-be-detected image can also be designed according to the characteristics of the detection target by those skilled in the art. The present specification does not limit the way of extracting the image feature information of the to-be-detected image.

[0050] After obtaining the point cloud data, the three-dimensional coordinate information of the point position can be determined based on the point cloud data. The point position can include a detected point position detected by the radar in the current space and an undetected empty point position. The point cloud instance in the point cloud data can be description information of the detected point position, which includes the three-dimensional coordinate information of the detected point position. Therefore, the three-dimensional coordinate information of the detected point position can be determined based on the three-dimensional coordinate information recorded in the point cloud data. However, the radar cannot detect everywhere in the current space. For example, the position where the target is not detected (i.e., the aforementioned “empty point position”) can be determined by using the ground point coordinate information. In the method, only the three-dimensional coordinate information of the detected point position can be used. Of course, the three-dimensional coordinate information of the undetected empty point position can also be combined based on the three-dimensional coordinate information of the detected point position. By increasing the ground point coordinate information that cannot be detected, the limitation of the radar detection blind area can be compensated, and the three-dimensional coordinate information of the point position in the blind area can be supplemented, thereby providing more environmental context information for the to-be-detected target object, which is helpful to improve the comprehensiveness of target detection.

[0051] After determining the image feature information and the three-dimensional coordinate information of the point position, the position coding feature information corresponding to the feature points in the image feature information can be generated based on the three-dimensional coordinate information. For example, the configuration parameters of the image acquisition device when collecting the image to be detected can be obtained, the mapping relationship between the feature points in the image feature information and the three-dimensional coordinate information is determined based on the configuration parameters, and then the position coding feature information corresponding to the feature points in the image feature information is generated based on the mapping relationship. Specifically, for any three-dimensional coordinate information, the three-dimensional coordinate information can be mapped to the two-dimensional plane of the image according to the intrinsic matrix and extrinsic matrix of the image acquisition device, and the mapping relationship between the feature points in the two-dimensional plane and the any three-dimensional coordinate information is established, and then the position coding feature information corresponding to the feature points in the image feature information is generated based on the mapping relationship.

[0052] After determining the image feature information and the position coding feature information, the target object in the image to be detected can be determined based on the image feature information and the position coding feature information. For example, the image feature information and the position coding feature information can be jointly input into a target detection module in a target detection network for further processing to determine the target object in the image to be detected. The target detection network can be, for example, a target detection model based on a convolutional neural network and a target detection model based on a Transformer which are well known in the industry. The target detection module is used to perform target object classification tasks and bounding box regression tasks according to the image feature information and the position coding feature information. The present scheme associates the real three-dimensional coordinate information of the object in the image to be detected with the image feature information (such as texture features, edge point features, etc.), introduces real-world three-dimensional coordinate information for the feature points of the image feature information, and can overcome the uncertainty problem of estimating target depth information when the target detection module of the traditional target detection model performs target object classification tasks and bounding box regression tasks.

[0053] In order to more easily understand the implementation steps of the method, as shown in FIG. 2, the present specification provides a schematic diagram of a target detection process:

[0054] The point cloud data 21 can be input to the three-dimensional coordinate information extraction module 22, the three-dimensional coordinate information extraction module 22 extracts the three-dimensional coordinate information 23 from the point cloud data 21 and outputs to the position feature extraction module 25. The image to be detected 24 is input to the image feature extraction module 26, and the image feature extraction module 26 extracts the image feature information 27 from the image to be detected 24 and outputs it to the target detection module 29. The image feature extraction module 26 can be a neural network for extracting image feature information such as ResnNet, DarkNet, etc. The position feature extraction module 25 can generate position encoding feature information 28 corresponding to the feature points of the image feature information 27 based on the three-dimensional coordinate information 23. The generation method can refer to the foregoing embodiments, which will not be repeated here. Then, the image feature information 27 and the position encoding feature information 28 are jointly input to the target detection module 29, and the target detection module 29 can determine the target object in the image to be detected 24 based on the image feature information 27 and the position encoding feature information 28. Wherein, the target object can be the category of the target object and / or the bounding box of the target object. The target detection module 29 can use the target detection module in the target detection network for performing the target object classification task and the bounding box regression task. The present specification does not limit the neural network architecture of the target detection module 29.

[0055] Preferably, as shown in FIG. 3, the ground point coordinate information 31 and the point cloud data 21 can be jointly input to the three-dimensional coordinate information extraction module 22 to obtain the three-dimensional coordinate information 23. The ground point coordinate information 31 can be the ground coordinate information of the empty point position not detected by the radar.

[0056] Preferably, the present specification provides a neural network framework of the target detection module 29 to further optimize the role of the point cloud data 21 in the present target detection method, which will be described in the following embodiments:

[0057] As shown in FIG. 4, the target detection module 29 can include a fusion processing module 291, an encoder 293, a decoder 295, and a target object output module 298.

[0058] The image feature information 27 and the position encoding feature information 28 are input to the fusion processing module 291, and the second image feature information 292 is output after fusion processing by the fusion processing module 291. The fusion processing method of the fusion processing module 291 can be to add the image feature information 27 and the position encoding feature information 28 in the corresponding dimension; or, the image feature information 27 and the position encoding feature information 28 are spliced in the same dimension, and the present specification does not limit the fusion processing method. After fusion processing, the association between the three-dimensional coordinate information in the point cloud data and the image feature information is established, and the real three-dimensional coordinate information is introduced into the image feature information.

[0059] The second image feature information 292 is input to an encoder 293 and output third image feature information 294. The encoder 293 is mainly used to convert the second image feature information 292 into more abstract and expressive third image feature information 294. The network architecture of the encoder 293 can draw on classic encoder architectures such as VGG and ResNet. Of course, those skilled in the art can design the network architecture of the encoder 293 according to actual needs, and the present specification does not make any limitation on the network architecture of the encoder 293.

[0060] The query vector generation module 296 can generate a query vector 297 of the decoder 295 according to the point cloud data 21. Then, the third image feature information 294 and the query vector 297 are jointly input into the decoder 295 to obtain fourth image feature information 297. The decoder 295 is used to restore the third image feature information generated by the encoder 293 to the fourth image feature information 297 used to represent the pixel information of the image to be detected 24. The decoder 295 can be a Transformer decoder, or a neural network architecture customized by those skilled in the art according to actual needs, and the present specification does not make any limitation on this.

[0061] The query vector 297 represents the target object that may appear in the image to be detected 24. For example, if there are 100 query vectors and 10 target objects appear in the image to be detected, whether the target object is detected is determined according to the confidence of the last output query vector. Therefore, in order to prevent missing detection of target objects, the actual situation is that the number of target objects appearing in the image to be detected 24 is often less than the number of query vectors. The query vector designed by the prior art is randomly initialized and generated, and the number of query vectors is a fixed value that needs to be set by those skilled in the art according to experience. Since the prior information of the target object is unknown, it may increase the risk that the query vector does not retrieve the real target object, thereby causing a high false detection rate to occur. In addition, even if there are not enough target objects in the image to match all the query vectors, the full spatial information of the image still needs to be traversed, which also causes waste of computing resources. The present scheme generates the query vector 297 of the decoder 295 according to the point cloud data 21. Since the point cloud data 21 contains rich prior three-dimensional spatial information about the target objects in the image to be detected 24, the decoder 295 can more accurately capture the geometric features and spatial distribution characteristics of the target objects, thereby reducing the false detection rate. Optionally,

[0062] A random value with the same dimension as the query vector 297 can be generated, and then the query vector 297 of the decoder 295 can be generated based on the random value and the point cloud data 21. Alternatively, the query vector 297 of the decoder 295 can be generated based on the random value and each point instance in the point cloud data 21. Specifically, the query vector 297 of the decoder 295 can be generated based on the random value and the three-dimensional coordinate information of each point instance in the point cloud data 21. Alternatively, an aggregated value of the point instances of each target object in the point cloud data 21 can be determined, and the query vector 297 of the decoder 295 can be generated based on the random value and the aggregated value. The aggregated value can be specifically an aggregated value of the three-dimensional coordinate information of the point instances of each target object. The aggregated value can be an average value, a central value, or the like of the three-dimensional coordinate information of the point instances of each target object. By dividing the point instances of the point cloud data 21 into different categories according to target objects, and then taking the aggregated value of each category as the query vector, the number of query vectors can be significantly reduced, thereby reducing the computational complexity. Moreover, the point instances of each target object of the point cloud data often correspond to the same local area in the scene, and by obtaining the aggregated value of the point instances of each target object, the feature information of the local area can be effectively summarized to reduce the noise influence of useless point instances.

[0063] Alternatively, the decoder 295 can include an attention mechanism module, and the first key-value vector in the attention mechanism module can be fused with the position encoding feature information 28 to obtain a second key-value vector. The weight of each value vector in the attention mechanism module can be determined based on the second key-value vector and the query vector 297. The fusion manner can be that the initialized first key-value vector and the position encoding feature information 28 are added in the same dimension, or the position encoding feature vector 28 can be spliced behind the initialized first key-value vector. The present specification does not limit the fusion manner in any way. In the present scheme, by introducing the real three-dimensional coordinate information into the key-value vector of the attention mechanism module, and using the point cloud data as the query vector in the query vector, the detection information corresponding to the target object can be directly obtained by searching the image feature through the attention mechanism. Compared with the full-space query, the retrieval complexity can be effectively reduced, and the false detection rate can be reduced.

[0064] The fourth image feature information 297 input target object output module 298 can determine the target object in the image to be detected. The target object output module 298 can be an FFN network (feed forward network), which can be specifically an MLP or a fully connected layer. The target object can be the position information of the object category appearing in the image to be detected or the bounding box framing the object position.

[0065] As shown in FIG. 5, FIG. 5 is a flow chart of another target detection method according to an exemplary embodiment of the present specification, including steps 501-503:

[0066] Step 501: For a target vehicle equipped with an image acquisition device and a radar, obtain a to-be-detected image acquired by the image acquisition device and point cloud data acquired by the radar and matched with the to-be-detected image.

[0067] Step 502: Extract image feature information of the to-be-detected image, and determine three-dimensional coordinate information of point positions around the target vehicle based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information.

[0068] Step 503: Determine a target object in the to-be-detected image based on the image feature information and the position coding feature information.

[0069] The method can be applied to the field of unmanned driving technology, and is used for assisting an autonomous vehicle to identify and locate vehicles, pedestrians, traffic signs, traffic lights and other obstacles on the road, so as to ensure that the autonomous vehicle can make reasonable decisions according to the target detection result. The to-be-detected image can be an image of the surrounding environment acquired by a camera configured on the unmanned vehicle. The to-be-detected image can be a single image acquired by one camera of the vehicle, or an image set composed of images of different angles acquired by multiple cameras equipped on the vehicle. For example, the to-be-detected image can be an image frame extracted from a video, such as an image frame extracted as a to-be-detected image, or a plurality of image frames extracted continuously or discontinuously (such as only key frames) as to-be-detected images. The to-be-detected image can be an RGB image, a grayscale image, or even an infrared image, depending on the type of camera used to acquire the to-be-detected image. The point cloud data can be acquired by a radar (Radar) or a laser radar (LiDAR) equipped on the vehicle.

[0070] In the method, the three-dimensional coordinate information is extracted from the point cloud data acquired by the radar of the vehicle, and the position coding feature information corresponding to the feature points of the image feature information of the to-be-detected image is generated. When determining the target object in the to-be-detected image, the position coding feature information is introduced into the image feature information, and the position coding feature information can provide real data of the depth and position of the feature points of the image. Therefore, when processing a target object with complex texture or unclear surface information, the target object can be accurately detected.

[0071] Corresponding to the embodiments of the foregoing method, the present specification also provides embodiments of a device and a terminal to which the device is applied.

[0072] As shown in FIG. 6, FIG. 6 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and of course can also include other hardware required by the business. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 610 into the memory 608 by the processor 602 and then running. Of course, in addition to the software implementation manner, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or a logic device.

[0073] As shown in FIG. 7, FIG. 7 is a block diagram of a target detection apparatus according to an exemplary embodiment of the present specification. The apparatus can be applied to the electronic device 600 as shown in FIG. 6 to implement the technical solutions of the present specification. The apparatus includes:

[0074] An acquisition module 702 is configured to acquire a to-be-detected image and point cloud data matched with the to-be-detected image;

[0075] A feature processing module 704 is configured to extract image feature information of the to-be-detected image, determine three-dimensional coordinate information of a point based on the point cloud data, and generate position coding feature information corresponding to a feature point in the image feature information based on the three-dimensional coordinate information.

[0076] A target detection module 706 is configured to determine a target object in the to-be-detected image based on the image feature information and the position coding feature information.

[0077] Optionally, the point includes a detected point in a current space and an undetected empty point, and the feature processing module 704 is specifically configured to determine three-dimensional coordinate information of the detected point based on three-dimensional coordinate information recorded in the point cloud data, and determine three-dimensional coordinate information of the empty point based on ground point coordinate information.

[0078] Optionally, the feature processing module 704 is specifically configured to acquire configuration parameters when an image acquisition device acquires the to-be-detected image, determine a mapping relationship between a feature point in the image feature information and the three-dimensional coordinate information based on the configuration parameters, and generate position coding feature information corresponding to the feature point in the image feature information based on the mapping relationship.

[0079] Optionally, the target detection module 706 is specifically configured to perform fusion processing on the image feature information and the position encoding feature information to obtain second image feature information; input the second image feature information into an encoder to obtain third image feature information; generate a query vector of a decoder according to the point cloud data, and input the third image feature information and the query vector into the decoder together to obtain fourth image feature information; and determine the target object in the to-be-detected image based on the fourth image feature information.

[0080] Optionally, the target detection module 706 is specifically configured to generate a random numerical value with the same dimension as the query vector; and generate the query vector of the decoder based on the random numerical value and the point cloud data.

[0081] Optionally, the target detection module 706 is specifically configured to generate the query vector of the decoder based on the random numerical value and each point instance in the point cloud data; or determine an aggregated value of point instances of each target object in the point cloud data; and generate the query vector of the decoder based on the random numerical value and the aggregated value.

[0082] Optionally, the decoder further includes an attention mechanism module, and the apparatus further includes a weight determination module 708, which is specifically configured to perform fusion processing on a first key-value vector in the attention mechanism module and the position encoding feature information to obtain a second key-value vector; and determine the weight of each value vector in the attention mechanism module based on the second key-value vector and the query vector.

[0083] Optionally, the to-be-detected image is a single image or a set of images.

[0084] As shown in FIG. 8, FIG. 8 is a block diagram of another target detection apparatus according to an exemplary embodiment of the present specification. The apparatus can be applied to the electronic device 600 as shown in FIG. 6 to implement the technical solutions of the present specification. The apparatus includes:

[0085] An acquisition module 802 is configured to, for a target vehicle equipped with an image acquisition apparatus and a radar, acquire a to-be-detected image collected by the image acquisition apparatus and point cloud data collected by the radar and matched with the to-be-detected image;

[0086] A feature processing module 804 is configured to extract image feature information of the to-be-detected image, determine three-dimensional coordinate information of point positions around the target vehicle based on the point cloud data, and generate position encoding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information.

[0087] The target detection module 806 is configured to determine a target object in the image to be detected based on the image feature information and the position coding feature information.

[0088] The implementation process of the functions and roles of each module in the device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0089] For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, that is, they can be located in one place or distributed on multiple network modules. Some or all of the modules can be selected to achieve the purpose of the scheme of the present specification according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0090] The present specification also provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the foregoing target detection method provided by the present application.

[0091] Specifically, the computer readable medium suitable for storing computer program instructions and data includes all forms of non-volatile memory, media and memory devices, including semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks.

[0092] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practice of the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including any variations of the present disclosure that are obvious to those skilled in the art in light of the present disclosure. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0093] It should be understood that the present disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

[0094] It is to be noted that, in the present text, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0095] The above describes in detail the method and device provided by the embodiments of the present disclosure. The principles and implementation manners of the present disclosure are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present disclosure and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present disclosure, the specific implementation manners and application ranges can be changed. In summary, the content of the present description should not be understood as a limitation of the present disclosure.

Claims

1. A target detection method characterized by, The method comprises: acquiring an image to be detected and point cloud data matched with the image to be detected; extracting image feature information of the image to be detected, and determining three-dimensional coordinate information of a point based on the point cloud data; and generating position coding feature information corresponding to a feature point in the image feature information based on the three-dimensional coordinate information; determining a target object in the image to be detected based on the image feature information and the position coding feature information.

2. The method of claim 1, wherein, The point comprises a detected point in a current space and an undetected empty point, and the determination of the three-dimensional coordinate information of the point based on the point cloud data comprises: determining the three-dimensional coordinate information of the detected point based on the three-dimensional coordinate information recorded in the point cloud data; determining the three-dimensional coordinate information of the empty point based on ground point coordinate information.

3. The method of claim 1, wherein, The generation of the position coding feature information corresponding to the feature point in the image feature information based on the three-dimensional coordinate information comprises: acquiring configuration parameters when an image acquisition device acquires the image to be detected; determining a mapping relationship between the feature point in the image feature information and the three-dimensional coordinate information based on the configuration parameters, and generating position coding feature information corresponding to the feature point in the image feature information based on the mapping relationship.

4. The method of claim 1, wherein, The determination of the target object in the image to be detected based on the image feature information and the position coding feature information comprises: performing fusion processing on the image feature information and the position coding feature information to obtain second image feature information; inputting the second image feature information into an encoder to obtain third image feature information; generating a query vector of a decoder according to the point cloud data, and inputting the third image feature information and the query vector into the decoder to obtain fourth image feature information; determining the target object in the image to be detected based on the fourth image feature information.

5. The method of claim 4, wherein, The generation of the query vector of the decoder according to the point cloud data comprises: generating a random value with the same dimension as the query vector, and generating the query vector of the decoder based on the random value and the point cloud data.

6. The method of claim 5, wherein, The generation of the query vector of the decoder based on the random value and the point cloud data comprises: generating the query vector of the decoder based on the random value and each point instance in the point cloud data; or determining an aggregate value of point instances of each target object in the point cloud data, and generating the query vector of the decoder based on the random value and the aggregate value.

7. The method of claim 5, wherein, The decoder further comprises an attention mechanism module, and the method further comprises: performing fusion processing on a first key-value vector in the attention mechanism module and the position coding feature information to obtain a second key-value vector; determining the weight of each value vector in the attention mechanism module based on the second key-value vector and the query vector.

8. The method of claim 1, wherein, The image to be detected is a single image or an image set.

9. A target detection method characterized by, The method comprises: for a target vehicle equipped with an image acquisition device and a radar, acquiring an image to be detected collected by the image acquisition device and point cloud data matched with the image to be detected and collected by the radar; extract image feature information of the to-be-detected image, and determine three-dimensional coordinate information of the point positions around the target vehicle based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information; determine a target object in the to-be-detected image based on the image feature information and the position coding feature information.

10. A target detection apparatus characterized by comprising: The apparatus comprises: an acquisition module configured to acquire a to-be-detected image and point cloud data matched with the to-be-detected image; a feature processing module configured to extract image feature information of the to-be-detected image, and determine three-dimensional coordinate information of point positions based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information; a target detection module configured to determine a target object in the to-be-detected image based on the image feature information and the position coding feature information.

11. A target detection apparatus characterized by comprising: The apparatus comprises: an acquisition module configured to acquire, for a target vehicle equipped with an image acquisition device and a radar, a to-be-detected image acquired by the image acquisition device and point cloud data matched with the to-be-detected image and acquired by the radar; a feature processing module configured to extract image feature information of the to-be-detected image, and determine three-dimensional coordinate information of point positions around the target vehicle based on the point cloud data; and generate position coding feature information corresponding to feature points in the image feature information based on the three-dimensional coordinate information; a target detection module configured to determine a target object in the to-be-detected image based on the image feature information and the position coding feature information. The processor implements the steps of the method of any one of claims 1 to 9 when executing the program.

12. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The program, when executed by the processor, implements the steps of the method of any one of claims 1 to 9.

13. A computer readable storage medium having stored thereon a computer program, characterized in that ​

Images