High-definition map target detection method and device, equipment and medium

Abstract

This disclosure provides a target detection method, apparatus, device, and medium, relating to the field of artificial intelligence technology, particularly to the fields of autonomous driving, high-precision maps, and navigation technology. The implementation scheme includes: acquiring point cloud data and at least one image data in a target scene; acquiring at least one point cloud feature; acquiring at least one feature map corresponding to each of the at least one image data; predicting a first reference coordinate and a first offset corresponding to each point cloud feature; correcting the first reference coordinate based on the first offset of each point cloud feature to obtain a second reference coordinate; acquiring a first image feature based on the second reference coordinate; acquiring at least one target image feature based on at least one first image feature of each feature map; acquiring at least one fusion feature based on at least one target image feature and the corresponding at least one point cloud feature; and predicting the detection result of a road target, such as a lane line, based on at least one fusion feature.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of autonomous driving, high-precision maps, and navigation technology, specifically to a target detection method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] In the generation of high-precision maps for autonomous driving, lane line semantic segmentation and road target detection are of great importance. Lane line semantic segmentation and target detection methods based on the fusion features of point cloud data and image data have improved the accuracy of lane line recognition and target detection to some extent.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a target detection method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0006] According to one aspect of this disclosure, a target detection method is provided, comprising: acquiring point cloud data and at least one image data in a target scene; performing feature extraction on the point cloud data to obtain at least one point cloud feature; performing feature extraction on each image data in the at least one image data to obtain at least one feature map corresponding to each of the at least one image data; predicting a first reference coordinate and a first offset corresponding to each of the at least one point cloud feature based on the at least one point cloud feature; correcting the corresponding first reference coordinate based on the first offset corresponding to each of the at least one point cloud feature to obtain a second reference coordinate; for each feature map in the at least one feature map, obtaining at least one first image feature corresponding to at least one first position in the feature map based on at least one second reference coordinate corresponding to the at least one point cloud feature, wherein the at least one first position corresponds to at least one second reference coordinate respectively; obtaining at least one target image feature based on at least one first image feature of each feature map in the at least one feature map, wherein the at least one target image feature corresponds to at least one point cloud feature respectively; obtaining at least one fusion feature based on at least one target image feature and the corresponding at least one point cloud feature respectively; and predicting a detection result of a road target based on the at least one fusion feature.

[0007] According to another aspect of this disclosure, a target detection apparatus is provided, comprising: a first acquisition unit configured to acquire point cloud data and at least one image data in a target scene; a second acquisition unit configured to perform feature extraction on the point cloud data to acquire at least one point cloud feature; a third acquisition unit configured to perform feature extraction on each of the at least one image data to acquire at least one feature map corresponding to each of the at least one image data; a first prediction unit configured to predict a first reference coordinate and a first offset corresponding to each of the at least one point cloud feature based on the at least one point cloud feature; and a correction unit configured to correct the corresponding first reference coordinate based on the first offset corresponding to each of the at least one point cloud feature to acquire a second reference coordinate. The fourth acquisition unit is configured to acquire at least one first image feature corresponding to at least one first position in each of the at least one feature maps, based on at least one second reference coordinate corresponding to at least one point cloud feature, wherein the at least one first position corresponds to at least one second reference coordinate; the fifth acquisition unit is configured to acquire at least one target image feature based on at least one first image feature of each of the at least one feature maps, wherein the at least one target image feature corresponds to at least one point cloud feature; the sixth acquisition unit is configured to acquire at least one fusion feature based on at least one target image feature and at least one corresponding point cloud feature; and the second prediction unit is configured to predict the detection result of the road target based on at least one fusion feature.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the target detection method described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described target detection method.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-described target detection method when executed by a processor.

[0011] According to one or more embodiments of this disclosure, the accuracy of reference coordinate offset correction can be improved, thereby improving the consistency between target image features and point cloud features, and improving the accuracy of target detection.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0015] Figure 2 A flowchart of a target detection method according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A schematic diagram of the structure of a feature fusion network according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A flowchart of a target detection method according to an embodiment of the present disclosure is shown;

[0018] Figure 5A A detection result diagram of lane line detection according to an embodiment of the present disclosure is shown;

[0019] Figure 5B The image shows the detection results of lane line detection based on relevant technologies;

[0020] Figure 6 A structural block diagram of a target detection apparatus according to an embodiment of the present disclosure is shown;

[0021] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0024] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0025] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0026] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0027] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of the target detection method described above.

[0028] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105, and / or 106 under a Software as a Service (SaaS) model.

[0029] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0030] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to acquire point cloud data and image data. The client devices can provide an interface that allows users to interact with them. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0031] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0032] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0033] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0034] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0035] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0036] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0037] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0038] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0039] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0040] According to embodiments of this disclosure, such as Figure 2 As shown, a target detection method is provided, including: step S201, acquiring point cloud data and at least one image data in a target scene; step S202, performing feature extraction on the point cloud data to obtain at least one point cloud feature; step S203, performing feature extraction on each image data in the at least one image data to obtain at least one feature map corresponding to each image data; step S204, predicting a first reference coordinate and a first offset corresponding to each point cloud feature based on the at least one point cloud feature; step S205, correcting the corresponding first reference coordinate based on the first offset corresponding to each point cloud feature to obtain a second reference coordinate; step S206... 206. For each feature map in at least one feature map, based on at least one second reference coordinate corresponding to at least one point cloud feature, obtain at least one first image feature corresponding to at least one first position in the feature map, wherein at least one first position corresponds to at least one second reference coordinate; Step S207. Based on at least one first image feature of each feature map in at least one feature map, obtain at least one target image feature, wherein at least one target image feature corresponds to at least one point cloud feature; Step S208. Based on at least one target image feature and at least one corresponding point cloud feature, obtain at least one fusion feature; and Step S209. Based on at least one fusion feature, predict the target detection result.

[0041] This improves the accuracy of reference coordinate offset correction, thereby enhancing the consistency between target image features and point cloud features, and ultimately improving the accuracy of target detection.

[0042] In some embodiments, the methods described above in this disclosure can be used, for example, to detect road targets such as lane lines, vehicles, pedestrians, and traffic signs.

[0043] In some embodiments, the fusion features of point cloud data and image data obtained by the above methods can be predicted based on a trained semantic segmentation model and object detection model, thereby obtaining the detection results of the road targets. The following will use lane line detection as an example to describe the method of this disclosure.

[0044] In some embodiments, image data refers to two-dimensional image data acquired for a target scene; point cloud data refers to three-dimensional point cloud data acquired for a target scene. Image data and point cloud data can be acquired from different angles for the same target scene. Image data can be obtained by capturing images of the scene using an image acquisition device such as a camera; point cloud data can be obtained by scanning the scene using a LiDAR, wherein the calibration parameters of the image acquisition device are predetermined.

[0045] In some embodiments, during feature extraction from point cloud data, the point cloud data can first be interpolated based on the intensity of the point cloud reflectance to obtain a three-dimensional point cloud image, in which the coordinate information of the point cloud is preserved. Subsequently, a two-dimensional point cloud image of the road plane can be obtained based on the three-dimensional point cloud image. Specifically, the point cloud image within a certain height of the three-dimensional point cloud image (e.g., within a certain height above the road surface area) can be obtained first, and the point cloud image can be divided into multiple point cloud regions based on a preset length and width, with each point cloud region including one or more points; based on the coordinates of one or more points in each point cloud region, the coordinates of the center point of the point cloud region can be determined, and the distance from each point in the region to the center point can be obtained respectively; subsequently, a weighted average can be performed based on the reflectance intensity of each point in the region according to the corresponding weights, and the resulting value can be used as the reflectance intensity of the center point, where the corresponding weight of each point can be the reciprocal of its distance from the center point coordinates. In some embodiments, when the number of points in a certain point cloud region is less than a preset threshold, the center point can be marked as zero so that it does not participate in the subsequent feature extraction and model prediction process.

[0046] Based on the above method, the center point of each point cloud region and its corresponding coordinate information and reflection intensity information are obtained respectively, and a two-dimensional point cloud image of the road surface in the target scene can be obtained based on the above center point.

[0047] In some embodiments, a backbone network can be used to extract features from the two-dimensional point cloud image to obtain at least one point cloud feature, wherein each point cloud feature corresponds to its corresponding center point coordinate information.

[0048] In some embodiments, image features can be extracted from each image data in at least one image dataset using a backbone network to obtain a feature map corresponding to each image data. Each feature map includes multiple image features, and each image feature corresponds to a specific image coordinate.

[0049] In some embodiments, a first reference coordinate and a first offset corresponding to each point cloud feature can first be predicted based on at least one point cloud feature. Specifically, at least one first point cloud feature can be input into different trained neural networks to obtain at least one first reference coordinate and at least one first offset output by the two neural networks respectively.

[0050] The two neural networks mentioned above can be obtained by training on a multilayer perceptron neural network, respectively.

[0051] In some embodiments, each point cloud feature in at least one point cloud feature includes corresponding point cloud coordinates. Based on at least one point cloud feature, predicting the first reference coordinates and first offset corresponding to each point cloud feature in at least one point cloud feature may include: inputting at least one point cloud coordinate corresponding to at least one point cloud feature into a first neural network to obtain at least one first reference coordinate output by the first neural network; and inputting at least one point cloud feature into a second neural network to obtain the first offset corresponding to each point cloud feature in at least one point cloud feature output by the second neural network, wherein the first neural network and the second neural network are trained based on sample point cloud data and sample image data containing target annotation information.

[0052] Therefore, reference coordinates are obtained by using the corresponding point cloud coordinates of each point cloud feature; at the same time, the offset corresponding to each reference coordinate is obtained based on each point cloud feature, thereby further enhancing the application of the information contained in the point cloud data (i.e., point cloud features and their corresponding point cloud coordinates), which can further improve the accuracy of reference point offset correction and the consistency between image features and point cloud features.

[0053] Figure 3 A schematic diagram of the structure of a feature fusion network according to an embodiment of the present disclosure is shown.

[0054] See Figure 3 At least one point cloud feature and its corresponding at least one point cloud coordinate can be input into a first neural network 301 to obtain at least one first reference coordinate output by the first neural network 301. Simultaneously, at least one point cloud feature can be input into a second neural network 302 to obtain at least one first offset output by the second neural network 302. Subsequently, based on the first offset, the corresponding first reference coordinate can be corrected to obtain at least one second reference coordinate.

[0055] Subsequently, based on the aforementioned at least one second reference coordinate and the calibration parameters of the image acquisition device, each second reference coordinate can be projected onto the corresponding first position in each feature map to obtain the first image feature corresponding to that first position. Compared to directly projecting at least one point cloud coordinate onto each feature map, obtaining the aforementioned first reference coordinate and performing coordinate correction based on the corresponding first offset can neutralize the projection error caused by the error in the calibration parameters, avoiding situations where the extracted image features and point cloud features cannot correspond, thereby improving the matching and consistency between image features and point cloud features, and thus improving the accuracy of subsequent lane line detection.

[0056] In some embodiments, the first neural network and the second neural network described above can be trained based on a multilayer perceptual neural network, respectively. Specifically, the training method may involve first training the feature fusion network (e.g., reference...) Figure 3 Then, an output network is added to perform semantic segmentation tasks. Subsequently, the above network can be trained based on point cloud sample data with lane lines labeled and at least one image sample data to obtain the first neural network, the second neural network and the output network.

[0057] In some embodiments, after obtaining at least one first image feature in each feature map, a weighted average or a stitching operation can be performed on the first image features at corresponding positions in at least one feature map to obtain at least one target image feature.

[0058] In some embodiments, the output of the second neural network further includes a first attention weight corresponding to each point cloud feature in at least one point cloud feature. Obtaining at least one target image feature based on at least one first image feature of each feature map in at least one feature map may include: obtaining at least one target image feature based on at least one first image feature of each feature map in at least one feature map and the corresponding at least one first attention weight.

[0059] Therefore, by simultaneously acquiring the attention weights corresponding to each image feature based on the second neural network, the importance differences of features at different locations can be further obtained when performing subsequent image feature extraction, thereby introducing richer feature information for subsequent tasks (such as object detection tasks and lane recognition tasks).

[0060] In some embodiments, the output of the second neural network may further include a first attention weight corresponding to each point cloud feature, which can represent the importance of features at corresponding locations in the point cloud image and feature map. After obtaining at least one first image feature in each feature map, a weighted average or concatenation operation can first be performed on the first image features at corresponding locations in at least one feature map to obtain at least one intermediate feature. Subsequently, each intermediate feature can be multiplied by its corresponding first attention weight to obtain at least one target image feature, thereby introducing the importance information of each image feature into the prediction process and further improving the accuracy of subsequent target tasks such as lane line detection.

[0061] In some embodiments, after obtaining at least one target image feature, at least one fusion feature can be obtained based on the at least one target image feature and at least one corresponding point cloud feature.

[0062] In some embodiments, the target image features and the corresponding point cloud features can be fused by weighted summation or direct stitching to obtain the corresponding fused features.

[0063] Subsequently, at least one fused feature can be input into the output network used to perform the object detection task to obtain the corresponding object detection result. For example, for the lane line detection task, the aforementioned fused feature can be input into the semantic segmentation output network trained based on the above method to obtain the lane line detection result.

[0064] In some embodiments, such as Figure 4 As shown, the above-mentioned target detection method may further include: performing the following target image feature update operation based on at least one target image feature: step S401, obtaining the corresponding third reference coordinate and second offset for each target image feature in the at least one target image feature based on at least one target image feature; step S402, correcting the corresponding third reference coordinate based on the corresponding second offset for each target image feature in the at least one target image feature to obtain the corresponding fourth reference coordinate; step S403, for each feature map in the at least one feature map, obtaining at least one second image feature corresponding to at least one second position in the feature map based on at least one fourth reference coordinate corresponding to at least one target image feature, wherein at least one second position corresponds to at least one fourth reference coordinate respectively; and step S404, obtaining the updated at least one target image feature based on at least one second image feature of each feature map in the at least one feature map; and step S405, obtaining the updated at least one fusion feature based on the updated at least one target image feature and the corresponding at least one point cloud feature respectively.

[0065] After performing one round of feature fusion according to the above method, at least one fused feature can be re-input into the feature fusion network and a similar feature fusion operation can be performed to obtain at least one updated fused feature. Therefore, based on the processed fused feature, the offset of the reference coordinates can be further obtained and the reference coordinates can be corrected again. By utilizing the information from each dimension predicted during the first round of feature fusion, the matching and consistency between image features and point cloud features are further enhanced, thereby improving the accuracy of subsequent lane detection.

[0066] In some embodiments, obtaining the third reference coordinate and the second offset corresponding to each of the at least one target image features based on at least one target image feature may include: inputting at least one target image feature into a first neural network to obtain at least one third reference coordinate output by the first neural network; and inputting at least one target image feature into a second neural network to obtain the second offset corresponding to each of the at least one target image features output by the second neural network.

[0067] In this process, at least one target image feature can be input into the first neural network to obtain at least one corresponding third reference coordinate, and at the same time, at least one target image feature can be input into the second neural network to obtain at least one corresponding second offset. This fully utilizes the information obtained from the previous prediction, thereby further improving the accuracy of the reference point offset correction and the consistency between image features and point cloud features.

[0068] In some embodiments, in response to the input of the second neural network being at least one target image feature, the output of the second neural network further includes a second attention weight corresponding to each target image feature in the at least one target image feature. Obtaining an updated at least one target image feature based on at least one second image feature of each feature map in at least one feature map may include: obtaining an updated at least one target image feature based on at least one second image feature of each feature map in at least one feature map and the corresponding at least one second attention weight.

[0069] Therefore, by simultaneously acquiring the attention weights corresponding to each image feature based on the second neural network, the importance differences of features at different locations can be further obtained when performing subsequent image feature extraction, thereby introducing richer feature information for subsequent tasks (such as object detection tasks and lane recognition tasks).

[0070] In some embodiments, the target detection method may further include: performing multiple update operations based on at least one updated target image feature to obtain at least one target image feature after multiple updates; and obtaining at least one updated fusion feature based on at least one target image feature after multiple updates and at least one corresponding point cloud feature.

[0071] In some embodiments, the above update operation can be repeated for example, up to six times, and at least one target image feature from the final output can be input into the corresponding target detection output network to obtain the corresponding target detection result. Thus, by performing offset correction multiple times, the consistency between the final obtained image features and point cloud features is stronger, and the resulting fused features are more accurate.

[0072] Figure 5A A detection result diagram of lane line detection according to an embodiment of the present disclosure is shown; Figure 5B The image shows the detection results of lane line detection based on relevant technologies.

[0073] based on Figure 5A and Figure 5B As can be seen, the lane line detection results obtained by the method provided according to the embodiments of this disclosure have clearer edges, and the method provided by the embodiments of this disclosure also has a more accurate detection effect in areas with weak reflectance intensity.

[0074] In some embodiments, such as Figure 6As shown, a target detection device 600 is provided, comprising: a first acquisition unit 610 configured to acquire point cloud data and at least one image data in a target scene; a second acquisition unit 620 configured to perform feature extraction on the point cloud data to acquire at least one point cloud feature; a third acquisition unit 630 configured to perform feature extraction on each of the at least one image data to acquire at least one feature map corresponding to each of the at least one image data; a first prediction unit 640 configured to predict a first reference coordinate and a first offset corresponding to each of the at least one point cloud feature based on the at least one point cloud feature; and a correction unit 650 configured to correct the corresponding first reference coordinate based on the first offset corresponding to each of the at least one point cloud feature to acquire a second reference coordinate. The fourth acquisition unit 660 is configured to acquire at least one first image feature corresponding to at least one first position in each feature map of at least one feature map, based on at least one second reference coordinate corresponding to at least one point cloud feature, wherein at least one first position corresponds to at least one second reference coordinate; the fifth acquisition unit 670 is configured to acquire at least one target image feature based on at least one first image feature of each feature map of at least one feature map, wherein at least one target image feature corresponds to at least one point cloud feature; the sixth acquisition unit 680 is configured to acquire at least one fusion feature based on at least one target image feature and at least one corresponding point cloud feature; and the second prediction unit 690 is configured to predict the target detection result based on at least one fusion feature.

[0075] The operations performed by units 610-690 in the target detection device 600 are similar to those of steps S201-S209 in the target detection method described above, and will not be described in detail here.

[0076] In some embodiments, each point cloud feature in at least one point cloud feature includes corresponding point cloud coordinates, and the first prediction unit may include: a first input subunit configured to input at least one point cloud coordinate corresponding to at least one point cloud feature into a first neural network to obtain at least one first reference coordinate output by the first neural network; and a second input subunit configured to input at least one point cloud feature into a second neural network to obtain a first offset corresponding to each point cloud feature in at least one point cloud feature output by the second neural network, wherein the first neural network and the second neural network are trained based on sample point cloud data and sample image data containing target annotation information.

[0077] In some embodiments, the output of the second neural network further includes a first attention weight corresponding to each point cloud feature in at least one point cloud feature, and the fifth acquisition unit can be further configured to: acquire at least one target image feature based on at least one first image feature of each feature map in at least one feature map and the corresponding at least one first attention weight.

[0078] In some embodiments, the target detection device may further include: a first update unit configured to perform an update operation on target image features based on at least one target image feature; the first update unit includes: a first acquisition subunit configured to acquire a third reference coordinate and a second offset corresponding to each of the at least one target image features based on the at least one target image feature; a correction subunit configured to correct the corresponding third reference coordinate based on the second offset corresponding to each of the at least one target image features to acquire a corresponding fourth reference coordinate; a second acquisition subunit configured to acquire at least one second image feature corresponding to at least one second position in the feature map for each feature map in the at least one feature map, based on at least one fourth reference coordinate corresponding to the at least one target image feature, wherein the at least one second position corresponds to at least one fourth reference coordinate; a third acquisition subunit configured to acquire an updated at least one target image feature based on at least one second image feature of each feature map in the at least one feature map; and a seventh acquisition unit configured to acquire an updated at least one fusion feature based on the updated at least one target image feature and at least one corresponding point cloud feature.

[0079] In some embodiments, the first acquisition subunit may be further configured to: input at least one target image feature into a first neural network to obtain at least one third reference coordinate of the output of the first neural network; and input at least one target image feature into a second neural network to obtain a second offset corresponding to each target image feature in at least one target image feature of the output of the second neural network.

[0080] In some embodiments, in response to the input of the second neural network being at least one target image feature, the output of the second neural network further includes a second attention weight corresponding to each target image feature in the at least one target image feature, and the third acquisition subunit can be further configured to: acquire an updated at least one target image feature based on at least one second image feature of each feature map in at least one feature map and the corresponding at least one second attention weight.

[0081] In some embodiments, the target detection apparatus may further include: a second updating unit configured to perform multiple update operations based on at least one updated target image feature to obtain at least one target image feature after multiple updates; and an eighth acquisition unit configured to acquire at least one updated fusion feature based on at least one target image feature after multiple updates and at least one corresponding point cloud feature.

[0082] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0083] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0084] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0085] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, hard disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0086] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the object detection method described above. For example, in some embodiments, the object detection method described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the object detection method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the object detection method described above by any other suitable means (e.g., by means of firmware).

[0087] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0089] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0090] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0091] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0092] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0093] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0094] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A target detection method, the method comprising: Acquire point cloud data from the target scene and at least one image data acquired by an image acquisition device; Feature extraction is performed on the point cloud data to obtain at least one point cloud feature; Feature extraction is performed on each of the at least one image data to obtain at least one feature map corresponding to each of the at least one image data; Based on the at least one point cloud feature, the first reference coordinate and the first offset of each point cloud feature are predicted; Based on the first offset of each point cloud feature in the at least one point cloud feature, the corresponding first reference coordinates are corrected to obtain the second reference coordinates; For each feature map in the at least one feature map, based on at least one second reference coordinate corresponding to the at least one point cloud feature, at least one first image feature corresponding to at least one first position in the feature map is obtained. The at least one first position is obtained by projecting the at least one second reference coordinate onto the feature map based on the calibration parameters of the image acquisition device. The first reference coordinate correction based on the first offset is used to neutralize the projection error caused by the error of the calibration parameters. Based on at least one first image feature of each feature map in the at least one feature map, at least one target image feature is obtained, wherein the at least one target image feature corresponds to the at least one point cloud feature respectively; Based on the at least one target image feature and the corresponding at least one point cloud feature, at least one fusion feature is obtained; as well as Based on the at least one fusion feature, the target detection result is predicted.

2. The method according to claim 1, wherein, Each point cloud feature in the at least one point cloud feature includes corresponding point cloud coordinates, and the prediction of the corresponding first reference coordinates and first offset for each point cloud feature based on the at least one point cloud feature includes: The at least one point cloud coordinate corresponding to the at least one point cloud feature is input into a first neural network to obtain at least one first reference coordinate output by the first neural network; and The at least one point cloud feature is input into the second neural network to obtain a first offset for each point cloud feature output by the second neural network, wherein the first neural network and the second neural network are trained based on sample point cloud data and sample image data containing target annotation information.

3. The method according to claim 2, wherein, The output of the second neural network also includes a first attention weight corresponding to each point cloud feature in the at least one point cloud feature, and the step of obtaining at least one target image feature based on at least one first image feature of each feature map in the at least one feature map includes: At least one target image feature is obtained based on at least one first image feature of each feature map and at least one corresponding first attention weight.

4. The method according to claim 2 or 3, further comprising: Based on the at least one target image feature, perform the following target image feature update operation: Based on the at least one target image feature, obtain the third reference coordinate and the second offset corresponding to each target image feature in the at least one target image feature; Based on the second offset of each target image feature in the at least one target image feature, the corresponding third reference coordinates are corrected to obtain the corresponding fourth reference coordinates; For each feature map in the at least one feature map, based on at least one fourth reference coordinate corresponding to the at least one target image feature, at least one second image feature corresponding to at least one second position in the feature map is obtained, wherein the at least one second position corresponds to the at least one fourth reference coordinate. as well as Based on at least one second image feature of each feature map in the at least one feature map, obtain at least one updated target image feature; as well as Based on the updated at least one target image feature and the corresponding at least one point cloud feature, an updated at least one fusion feature is obtained.

5. The method according to claim 4, wherein, The step of obtaining the third reference coordinate and second offset corresponding to each of the at least one target image features based on the at least one target image feature includes: The at least one target image feature is input into the first neural network to obtain at least one third reference coordinate output by the first neural network; and The at least one target image feature is input into the second neural network to obtain a second offset for each target image feature in the at least one target image feature output by the second neural network.

6. The method according to claim 5, wherein, In response to the input of the second neural network being the at least one target image feature, the output of the second neural network further includes a second attention weight corresponding to each of the at least one target image features. The step of obtaining the updated at least one target image feature based on at least one second image feature of each feature map in the at least one feature map includes: Based on at least one second image feature of each feature map and at least one corresponding second attention weight, at least one updated target image feature is obtained.

7. The method of claim 4, further comprising: Based on at least one updated target image feature, the update operation is performed multiple times to obtain at least one target image feature that has been updated multiple times. as well as Based on at least one target image feature that has been updated multiple times and the corresponding at least one point cloud feature, at least one updated fused feature is obtained.

8. A target detection device, the device comprising: The first acquisition unit is configured to acquire point cloud data in the target scene and at least one image data acquired by the image acquisition device; The second acquisition unit is configured to perform feature extraction on the point cloud data to acquire at least one point cloud feature. The third acquisition unit is configured to perform feature extraction on each of the at least one image data to obtain at least one feature map corresponding to each of the at least one image data. The first prediction unit is configured to predict, based on the at least one point cloud feature, the corresponding first reference coordinate and first offset of each point cloud feature in the at least one point cloud feature; The correction unit is configured to correct the corresponding first reference coordinates based on the first offset of each point cloud feature in the at least one point cloud feature, so as to obtain the second reference coordinates. The fourth acquisition unit is configured to acquire, for each feature map in the at least one feature map, at least one first image feature corresponding to at least one first position in the feature map based on at least one second reference coordinate corresponding to the at least one point cloud feature. The at least one first position is obtained by projecting the at least one second reference coordinate onto the feature map based on the calibration parameters of the image acquisition device. The first reference coordinate correction based on the first offset is used to neutralize the projection error caused by the error of the calibration parameters. The fifth acquisition unit is configured to acquire at least one target image feature based on at least one first image feature of each feature map in the at least one feature map, wherein the at least one target image feature corresponds to the at least one point cloud feature respectively; The sixth acquisition unit is configured to acquire at least one fusion feature based on the at least one target image feature and the corresponding at least one point cloud feature; as well as The second prediction unit is configured to predict the target detection result based on the at least one fusion feature.

9. The apparatus according to claim 8, wherein, Each of the at least one point cloud feature includes corresponding point cloud coordinates, and the first prediction unit includes: A first input subunit is configured to input at least one point cloud coordinate corresponding to the at least one point cloud feature into a first neural network to obtain at least one first reference coordinate output by the first neural network; and The second input subunit is configured to input the at least one point cloud feature into the second neural network to obtain a first offset corresponding to each point cloud feature in the at least one point cloud feature output by the second neural network, wherein the first neural network and the second neural network are trained based on sample point cloud data and sample image data containing target annotation information.

10. The apparatus according to claim 9, wherein, The output of the second neural network also includes a first attention weight corresponding to each point cloud feature in the at least one point cloud feature, and the fifth acquisition unit is further configured to: At least one target image feature is obtained based on at least one first image feature of each feature map and at least one corresponding first attention weight.

11. The apparatus according to claim 9 or 10, further comprising: A first update unit is configured to perform an update operation on target image features based on the at least one target image feature. The first update unit includes: The first acquisition subunit is configured to acquire, based on the at least one target image feature, the corresponding third reference coordinate and second offset of each target image feature in the at least one target image feature; The correction subunit is configured to correct the corresponding third reference coordinates based on the second offset of each target image feature in the at least one target image feature, so as to obtain the corresponding fourth reference coordinates; The second acquisition subunit is configured to, for each feature map in the at least one feature map, acquire at least one second image feature corresponding to at least one second position in the feature map based on at least one fourth reference coordinate corresponding to the at least one target image feature, wherein the at least one second position corresponds to the at least one fourth reference coordinate; and The third acquisition subunit is configured to acquire at least one updated target image feature based on at least one second image feature of each feature map in the at least one feature map; and The seventh acquisition unit is configured to acquire at least one updated fusion feature based on at least one updated target image feature and the corresponding at least one point cloud feature.

12. The apparatus according to claim 11, wherein, The first acquisition subunit is further configured as follows: The at least one target image feature is input into the first neural network to obtain at least one third reference coordinate output by the first neural network; and The at least one target image feature is input into the second neural network to obtain a second offset for each target image feature in the at least one target image feature output by the second neural network.

13. The apparatus according to claim 12, wherein, In response to the input of the second neural network being the at least one target image feature, the output of the second neural network further includes a second attention weight corresponding to each of the at least one target image features, and the third acquisition subunit is further configured as follows: Based on at least one second image feature of each feature map and at least one corresponding second attention weight, at least one updated target image feature is obtained.

14. The apparatus of claim 11, further comprising: The second update unit is configured to perform the update operation multiple times based on at least one updated target image feature to obtain at least one target image feature after multiple updates. as well as The eighth acquisition unit is configured to acquire at least one updated fusion feature based on at least one target image feature that has been updated multiple times and the corresponding at least one point cloud feature.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-7.

Images