Target detection method, device and equipment, computer readable medium and program product

Abstract

The embodiment of the invention discloses a target detection method, device and equipment, a computer readable medium and a program product. A specific embodiment of the method comprises the following steps: acquiring a shot image set corresponding to a target vehicle in a first frame time and a first three-dimensional feature information group corresponding to a second frame time; generating two-dimensional feature information corresponding to each shot image in the shot image set to obtain a two-dimensional feature information set; according to the first three-dimensional feature information group, generating a three-dimensional feature information group corresponding to the target detection object set under the first frame time, and taking the three-dimensional feature information group as a target three-dimensional feature information group; generating a second three-dimensional feature information group under the first frame time according to the target three-dimensional feature information group, the object query information group and the two-dimensional feature information set; and generating target detection information corresponding to the first frame time according to the second three-dimensional feature information group. The implementation mode is related to artificial intelligence, and the target detection information under the current frame time can be accurately generated under the condition of relatively low calculation complexity.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to target detection methods, devices, equipment, computer-readable media, and program products. Background Art

[0002] At present, in the field of autonomous driving, the visual 3D object detection technology in the autonomous driving system is crucial to the perception ability of the vehicle. For 3D object detection, the usual method is: first, obtain multiple continuous historical 3D frame image information for the current frame time. Then, the image feature information of multiple historical 3D frame image information is fused to generate fused feature information. Finally, the fused feature information is input into the 3D object detection model to generate the 3D object detection result.

[0003] However, the inventors have found that when the above method is used to generate 3D object detection results, the following technical problems often occur:

[0004] For the processing of multiple continuous historical 3D frame image information, the memory needs to retain and process all relevant data within a time window, which requires higher computing power. As the time window increases, the amount of data to be processed increases significantly, resulting in extremely high computational complexity.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose target detection methods, devices, equipment, computer-readable media, and program products to solve the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a target detection method, including: obtaining a captured image set corresponding to a target vehicle at a first frame time and a first three-dimensional feature information group corresponding to a second frame time, wherein the second frame time is a frame time that has a historical continuity relationship with the first frame time; generating two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set; based on the first three-dimensional feature information group, generating a three-dimensional feature information group corresponding to a target detection object set at the first frame time as a target three-dimensional feature information group; based on the target three-dimensional feature information group, a pre-constructed object query information group and the two-dimensional feature information set, generating a second three-dimensional feature information group at the first frame time, wherein the number of object query information in the object query information group is pre-set; based on the second three-dimensional feature information group, generating target detection information corresponding to the first frame time.

[0009] Optionally, the above-mentioned three-dimensional feature information group corresponding to the target detection object set in the first frame time is generated as the target three-dimensional feature information group based on the above-mentioned first three-dimensional feature information group, including: screening out the first three-dimensional feature information whose corresponding object confidence is greater than a preset confidence from the above-mentioned first three-dimensional feature information group as candidate three-dimensional feature information, and obtaining a candidate three-dimensional feature information group; obtaining spatial transformation information for the above-mentioned first frame time and the above-mentioned second frame time; and performing feature information transformation on each candidate three-dimensional feature information in the above-mentioned candidate three-dimensional feature information group according to the above-mentioned spatial transformation information to generate target three-dimensional feature information in a target spatial coordinate system, and obtaining the above-mentioned target three-dimensional feature information group, wherein the above-mentioned target spatial coordinate system is the spatial coordinate system corresponding to the above-mentioned first frame time.

[0010] Optionally, the second three-dimensional feature information group is generated at the first frame time based on the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set, including: combining the target three-dimensional feature information group and the object query information group to generate a combined information group; inputting the combined information group into a first attention mechanism model to generate attention mechanism information; transforming the two-dimensional feature information set into a three-dimensional space dimension to generate three-dimensional feature information as third three-dimensional feature information; and inputting the third three-dimensional feature information and the attention mechanism information into a second attention mechanism model to generate the second three-dimensional feature information group.

[0011] Optionally, generating target detection information corresponding to the first frame time based on the second three-dimensional feature information group includes: inputting the second three-dimensional feature information group into a pre-trained detection information generation model to generate three-dimensional target detection information as the target detection information.

[0012] Optionally, the above-mentioned generation of two-dimensional feature information corresponding to each captured image in the above-mentioned captured image set includes: performing image preprocessing on the above-mentioned captured images to generate preprocessed images; inputting the above-mentioned preprocessed images into a pre-trained two-dimensional image feature extraction model to generate the above-mentioned two-dimensional feature information.

[0013] Optionally, the method further includes: storing the second three-dimensional feature information group and the first frame time in correspondence; in response to receiving target detection information for a third frame time, generating target detection information for the third frame time using the stored second three-dimensional feature information group, wherein the third frame time is a frame time that has a future continuous relationship with the first frame time.

[0014] In a second aspect, some embodiments of the present disclosure provide a target detection device, including: an acquisition unit, configured to acquire a captured image set corresponding to a target vehicle at a first frame time and a first three-dimensional feature information group corresponding to a second frame time, wherein the second frame time is a frame time that has a historical continuity relationship with the first frame time; a first generation unit, configured to generate two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set; a second generation unit, configured to generate a three-dimensional feature information group corresponding to a target detection object set at the first frame time based on the first three-dimensional feature information group, as a target three-dimensional feature information group; a third generation unit, configured to generate a second three-dimensional feature information group at the first frame time based on the target three-dimensional feature information group, a pre-constructed object query information group and the two-dimensional feature information set, wherein the number of object query information in the object query information group is pre-set; and a fourth generation unit, configured to generate target detection information corresponding to the first frame time based on the second three-dimensional feature information group.

[0015] Optionally, the second generation unit is configured to: filter out first three-dimensional feature information whose corresponding object confidence is greater than a preset confidence from the above-mentioned first three-dimensional feature information group as candidate three-dimensional feature information, and obtain a candidate three-dimensional feature information group; obtain spatial transformation information for the above-mentioned first frame time and the above-mentioned second frame time; and perform feature information transformation on each candidate three-dimensional feature information in the above-mentioned candidate three-dimensional feature information group according to the above-mentioned spatial transformation information to generate target three-dimensional feature information in a target spatial coordinate system, and obtain the above-mentioned target three-dimensional feature information group, wherein the above-mentioned target spatial coordinate system is the spatial coordinate system corresponding to the above-mentioned first frame time.

[0016] Optionally, the third generation unit is configured to: combine the above-mentioned target three-dimensional feature information group and the above-mentioned object query information group to generate a combined information group; input the above-mentioned combined information group into the first attention mechanism model to generate attention mechanism information; transform the above-mentioned two-dimensional feature information set into a three-dimensional space dimension to generate three-dimensional feature information as third three-dimensional feature information; input the above-mentioned third three-dimensional feature information and the above-mentioned attention mechanism information into the second attention mechanism model to generate the above-mentioned second three-dimensional feature information group.

[0017] Optionally, the fourth generating unit is configured to: input the second three-dimensional feature information group into a pre-trained detection information generation model to generate three-dimensional target detection information as the target detection information.

[0018] Optionally, the second generating unit is configured to: perform image preprocessing on the captured image to generate a preprocessed image; and input the preprocessed image into a pre-trained two-dimensional image feature extraction model to generate the two-dimensional feature information.

[0019] Optionally, the device also includes: storing the above-mentioned second three-dimensional feature information group and the above-mentioned first frame time in correspondence; in response to receiving the target detection information for the third frame time, using the stored above-mentioned second three-dimensional feature information group to generate the target detection information under the above-mentioned third frame time, wherein the above-mentioned third frame time is a frame time that has a future continuous relationship with the above-mentioned first frame time.

[0020] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0021] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0022] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner in the above-mentioned first aspect when executed by a processor.

[0023] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the target detection method of some embodiments of the present disclosure, the target detection information at the current frame time can be accurately generated under the condition of low computational complexity. Specifically, the reason for the high computational complexity of the relevant target detection information is that for the processing of multiple continuous historical 3D frame image information, the memory needs to retain and process all relevant data in a time window, resulting in the need for higher computing power, and as the time window increases, the amount of data to be processed increases significantly, resulting in extremely high computational complexity. Based on this, the target detection method of some embodiments of the present disclosure, first, obtains the captured image set corresponding to the target vehicle at the first frame time and the first three-dimensional feature information group corresponding to the second frame time, wherein the second frame time is the frame time with a historical continuous relationship with the first frame time. Here, the 3D detection feature information group for the detection object corresponding to a frame of historical time (i.e., the second frame time) with a historical continuous relationship with the current frame and the captured image set at the current frame time (i.e., the first frame time) can be used to subsequently accurately generate the 3D object feature information at the current frame time (i.e., the first frame time). Then, the two-dimensional feature information corresponding to each captured image in the captured image set is generated to obtain a two-dimensional feature information set, so as to facilitate the use of the image feature information corresponding to the 2D captured image. Then, according to the first three-dimensional feature information group, the three-dimensional feature information group corresponding to the target detection object set under the first frame time can be accurately generated as the target three-dimensional feature information group. Here, the preliminary 3D object feature information under the current frame time can be accurately mapped and generated by the 3D object feature information under the second frame time. Here, it is not necessary to process multiple continuous historical 3D frame image information under a long time in the historical time period, which can effectively avoid the problem of large amount of calculation. Then, according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set, the second three-dimensional feature information group under the first frame time can be accurately generated, wherein the number of object query information in the object query information group is preset. Here, by setting the query information for each object under the first frame time, the three-dimensional feature information group (i.e., the second three-dimensional feature information group) under the current frame time can be accurately generated by fully considering the target three-dimensional feature information group preliminarily generated at the current time and the two-dimensional feature information set of the current frame time. Finally, according to the second three-dimensional feature information group, the target detection information corresponding to the first frame time can be accurately generated. In summary, the target three-dimensional feature information group after feature information conversion considering the three-dimensional feature information group at the previous frame time, the two-dimensional feature information set at the current frame time, and the object query information group corresponding to the pre-set multiple objects can accurately generate the 3D target detection result at the current frame time with a small amount of calculation without considering multiple historical 3D frame image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0025] Figure 1 is a schematic diagram of an application scenario of a target detection method according to some embodiments of the present disclosure;

[0026] Figure 2 is a flow chart of some embodiments of the target detection method according to the present disclosure;

[0027] Figure 3 are flow charts of other embodiments of the target detection method according to the present disclosure;

[0028] Figure 4 is a schematic diagram of the structure of some embodiments of the target detection device according to the present disclosure;

[0029] Figure 5 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] With regard to the collection, storage, and use of user personal information (such as target detection information) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0036] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0037] Figure 1 It is a schematic diagram of an application scenario of the target detection method according to some embodiments of the present disclosure.

[0038] exist Figure 1 In the application scenario, first, the electronic device 101 can obtain the captured image set 104 corresponding to the target vehicle 102 at the first frame time 103 and the first three-dimensional feature information group 107 corresponding to the second frame time 105. Among them, the above-mentioned second frame time 105 is a frame time that has a historical continuity relationship with the above-mentioned first frame time 103. In this application scenario, the first frame time 103 can be "the 11th second time". The target vehicle 102 can be "Vehicle A". The second frame time 105 can be "the 10.5th second time". Then, the electronic device 101 can generate the two-dimensional feature information corresponding to each captured image in the above-mentioned captured image set 104, and obtain the two-dimensional feature information set 106. Next, the electronic device 101 can generate the three-dimensional feature information group corresponding to the target detection object set at the above-mentioned first frame time 103 according to the above-mentioned first three-dimensional feature information group 107, as the target three-dimensional feature information group 108. Furthermore, the electronic device 101 can generate the second three-dimensional feature information group 110 under the first frame time 103 according to the target three-dimensional feature information group 108, the pre-constructed object query information group 109 and the two-dimensional feature information set 106. Among them, the number of object query information in the object query information group 109 is preset. Finally, the electronic device 101 can generate the target detection information 111 corresponding to the first frame time 103 according to the second three-dimensional feature information group 110. In this application scenario, the target detection information 111 can be "spatial position: *** position; length: 31cm; width: 21cm; height: 12cm".

[0039] It should be noted that the electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the electronic device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0040] It should be understood that Figure 1 The number of electronic devices in the embodiment is only for illustration. Any number of electronic devices may be provided according to implementation requirements.

[0041] Continue to refer Figure 2 , shows a process 200 of some embodiments of the target detection method according to the present disclosure. The target detection method comprises the following steps:

[0042] Step 201, obtaining a captured image set corresponding to a target vehicle in a first frame time and a first three-dimensional feature information group corresponding to a second frame time.

[0043] In some embodiments, the execution subject of the above target detection method (for example Figure 1The electronic device 101 shown can obtain the captured image set corresponding to the target vehicle at the first frame time and the first three-dimensional feature information group corresponding to the second frame time through a wired connection method or a wireless connection method. Among them, the above-mentioned second frame time is a frame time that has a historical continuous relationship with the above-mentioned first frame time. The above-mentioned first frame time may be a time point at which the target object detection is to be performed. For example, the first frame time may be the current frame time. The captured image may be an image captured by a camera in the target vehicle. That is, the captured image set is each image captured by a multi-channel camera of the target vehicle. The second frame time is the previous frame time under the first frame time. In practice, for the autonomous driving scene, the target vehicle may be an autonomous driving vehicle. The historical continuous relationship may be a relationship that represents the continuity between times under historical time. For example, if the first frame time is the current frame, then the second frame time is the previous frame time. The first three-dimensional feature information group may represent the detection content information of each detection object at the second frame time. In practice, there is a one-to-one correspondence between the detection object in each detection object and the first three-dimensional feature information in the first three-dimensional feature information group. The first three-dimensional feature information may be feature information of the corresponding detection object in the three-dimensional (3D) space corresponding to the second frame time. The first three-dimensional feature information may be information in the form of a vector. At the second frame time, the target vehicle has a corresponding driving position and posture, and a corresponding three-dimensional space coordinate system may be established for the current target vehicle. That is, the first three-dimensional feature information may be information that characterizes the corresponding detection features of the detection object in the corresponding three-dimensional space coordinate system at the second frame time. Each detection object at the second frame time has a corresponding detection confidence, that is, each first three-dimensional feature information has a corresponding confidence, which characterizes the accuracy of the first three-dimensional feature information in characterizing the corresponding detection features of the detection object in the corresponding three-dimensional coordinate system. The detection confidence may be a value between 0 and 1, and the higher the value, the higher the accuracy of the corresponding first three-dimensional feature information. In practice, the first three-dimensional feature information group may be a feature information group for each detection object that is preliminarily generated at the second frame time.

[0044] Step 202: Generate two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set.

[0045] In some embodiments, the execution subject may generate two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set. The two-dimensional feature information may represent feature information in a two-dimensional (2D) space corresponding to the captured image. The two-dimensional feature information may be information in vector form that represents the image semantics corresponding to the captured image.

[0046] As an example, the above-mentioned execution entity can input each captured image into a convolutional neural network model to generate two-dimensional feature information and obtain a two-dimensional feature information set.

[0047] In some optional implementations of some embodiments, the execution subject may generate two-dimensional feature information corresponding to each captured image in the captured image set, which may include the following steps:

[0048] In the first step, the captured image is preprocessed to generate a preprocessed image, wherein the image preprocessing may include but is not limited to at least one of the following: image format conversion, image size scaling, and image normalization.

[0049] In the second step, the preprocessed image is input into a pre-trained two-dimensional image feature extraction model to generate the two-dimensional feature information. The two-dimensional image feature extraction model may be a neural network model for extracting image feature information from a two-dimensional image. In practice, the two-dimensional image feature extraction model may be a convolutional layer with multiple layers connected in series. The two-dimensional feature information may represent the semantic content of the image features in the preprocessed image.

[0050] Step 203: Generate a three-dimensional feature information group corresponding to the target detection object set in the first frame time according to the first three-dimensional feature information group as a target three-dimensional feature information group.

[0051] In some embodiments, the execution entity may generate a three-dimensional feature information group corresponding to the target detection object set at the first frame time according to the first three-dimensional feature information group, as a target three-dimensional feature information group. The target detection object set is each detection object detected at the second frame time. The target three-dimensional feature information group may characterize the feature information of each target detection object in the target detection object set at the three-dimensional coordinate system corresponding to the first frame time. The target three-dimensional feature information may be feature information in the form of a vector. In practice, the number of target detection objects in the target detection object set is preset. The three-dimensional coordinate system may be a three-dimensional coordinate system established based on the target vehicle at the first frame time.

[0052] As an example, first, a first image set corresponding to a first frame time and a second image set corresponding to a second frame time are obtained. According to the first image set and the second image set, a position transformation information generation model is used to determine the change information of each object between the first image set and the second image set. In practice, the position transformation information generation model may be a neural network model that generates position transformation information. In practice, the position transformation information generation model may include: a position detection model and a position mapping model. Among them, the position detection model may be a neural network model that generates position information in an image. The position mapping model may be a mapping model that generates position transformation between position information between two identical objects. The position mapping model may be a multi-layer fully connected layer connected in series. Finally, according to the change information, the first three-dimensional feature information group is mapped to generate a three-dimensional feature information group corresponding to the target detection object set under the above-mentioned first frame time as the target three-dimensional feature information group.

[0053] In some optional implementations of some embodiments, generating, according to the first three-dimensional feature information group, a three-dimensional feature information group corresponding to the target detection object set at the first frame time as the target three-dimensional feature information group may include the following steps:

[0054] The first step is to select the first three-dimensional feature information whose corresponding object confidence is greater than the preset confidence from the above-mentioned first three-dimensional feature information group as candidate three-dimensional feature information, and obtain the candidate three-dimensional feature information group. Among them, the object confidence can characterize the detection accuracy of the object detection result corresponding to the first three-dimensional feature information. The higher the object confidence, the higher the detection accuracy of the object detection result corresponding to the first three-dimensional feature information. There is a corresponding object confidence for each first three-dimensional feature information. The first three-dimensional feature information can characterize the feature semantic content of the object in the three-dimensional dimension.

[0055] The second step is to obtain the spatial transformation information for the first frame time and the second frame time. The spatial transformation information may be the spatial transformation information between the space where the target vehicle is located at the second frame time and the space where the target vehicle is located at the first frame time. In practice, the spatial transformation information may be generated based on the coordinate system transformation between the spatial coordinate system corresponding to the first frame time and the spatial coordinate system corresponding to the second frame time. The spatial transformation information may be transformation information in the form of a vector group, which may characterize the spatial changes of various objects between the first frame time and the second frame time. Various objects may include: a target vehicle.

[0056] In the third step, according to the spatial transformation information, feature information transformation is performed on each candidate 3D feature information in the candidate 3D feature information group to generate target 3D feature information in the target spatial coordinate system, thereby obtaining the target 3D feature information group. The target spatial coordinate system is the spatial coordinate system corresponding to the first frame time.

[0057] As an example, the above-mentioned execution entity can perform feature information mapping transformation on each candidate three-dimensional feature information in the candidate three-dimensional feature information group according to the spatial transformation information to generate target three-dimensional feature information in the target space coordinate system to obtain the above-mentioned target three-dimensional feature information group.

[0058] Step 204: Generate a second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set.

[0059] In some embodiments, the execution entity may generate the second three-dimensional feature information group under the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set. Among them, the object query information in the object query information group may have a one-to-one correspondence with the estimated objects in each estimated object under the first frame time. Each estimated object may be each object estimated to appear under the first frame time. The number of objects of the estimated objects corresponding to each estimated object is preset. The object query information may be information in the form of a vector characterizing object feature information. The object feature information may characterize feature information of various relevant features of the object. For example, various relevant features may include: object position features, object posture features.

[0060] As an example, first, the execution subject may splice the target three-dimensional feature information group and the object query information group to generate a spliced ​​information group. The spliced ​​information group may represent a feature information group of an object set that is the sum of the object group corresponding to the target three-dimensional feature information group and the object group corresponding to the object query information group. Then, the spliced ​​information group and the two-dimensional feature information set are input into a pre-trained attention mechanism model to output an attention information group at a position corresponding to the object query information group in the spliced ​​information group. Finally, the attention information group is determined as a second three-dimensional feature information group.

[0061] In some optional implementations of some embodiments, the execution subject may generate the second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set, which may include the following steps:

[0062] The first step is to combine the target three-dimensional feature information group and the object query information group to generate a combined information group.

[0063] As an example, the execution entity may perform splicing processing on the target three-dimensional feature information group and the object query information group to generate a combined information group.

[0064] In the second step, the combined information group is input into the first attention mechanism model to generate attention mechanism information. The first attention mechanism model may be a pre-trained neural network model based on the attention mechanism. In practice, the first attention mechanism model may be a Transformer model. The attention mechanism information may be information in the form of a vector.

[0065] In the third step, the two-dimensional feature information set is transformed into a three-dimensional space dimension to generate three-dimensional feature information as third three-dimensional feature information.

[0066] As an example, the execution entity may use a pre-acquired three-dimensional feature conversion model to perform three-dimensional spatial dimension conversion on each two-dimensional feature information in the two-dimensional feature information set to generate third three-dimensional feature information.

[0067] In the fourth step, the third three-dimensional feature information and the attention mechanism information are input into the second attention mechanism model to generate the second three-dimensional feature information group. The second attention mechanism model can be a pre-trained neural network model based on the attention mechanism. In practice, the second attention mechanism model can be a Transformer model.

[0068] Step 205: Generate target detection information corresponding to the first frame time according to the second three-dimensional feature information group.

[0069] In some embodiments, the execution subject may generate target detection information corresponding to the first frame time according to the second three-dimensional feature information group. The target detection information may be

[0070] As an example, first, the execution subject may determine the similarity between the first three-dimensional feature information in the first three-dimensional feature information group and the second three-dimensional feature information group to obtain a similarity set. Then, the first three-dimensional feature information whose corresponding similarity is greater than the target similarity is selected from the first three-dimensional feature information group as the target three-dimensional feature information to obtain at least one target three-dimensional feature information. The at least one target three-dimensional feature information is spliced ​​with the corresponding second three-dimensional feature information with a similarity association relationship to generate a spliced ​​second three-dimensional feature information group. Finally, the spliced ​​second three-dimensional feature information group is input into the regression model and the classification model to output regression information and classification information. Among them, the regression information may be the position information of the corresponding object. The classification information may be the corresponding object confidence. Finally, the regression information and the classification information are integrated with the corresponding object information to generate target detection information.

[0071] In some optional implementations of some embodiments, the execution entity may input the second three-dimensional feature information group into a pre-trained detection information generation model to generate three-dimensional target detection information as the target detection information. The detection information generation model may be a neural network model that generates 3D target detection information. For example, the detection information generation model may be a YOLO model. The detection information generation model may include: a regression model and a classification model. The classification model may output a model of an object detection category. The regression model may be a model that outputs object confidence.

[0072] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the target detection method of some embodiments of the present disclosure, the target detection information at the current frame time can be accurately generated under the condition of low computational complexity. Specifically, the reason for the high computational complexity of the relevant target detection information is that for the processing of multiple continuous historical 3D frame image information, the memory needs to retain and process all relevant data in a time window, resulting in the need for higher computing power, and as the time window increases, the amount of data to be processed increases significantly, resulting in extremely high computational complexity. Based on this, the target detection method of some embodiments of the present disclosure, first, obtains the captured image set corresponding to the target vehicle at the first frame time and the first three-dimensional feature information group corresponding to the second frame time, wherein the second frame time is the frame time with a historical continuous relationship with the first frame time. Here, the 3D detection feature information group for the detection object corresponding to a frame of historical time (i.e., the second frame time) with a historical continuous relationship with the current frame and the captured image set at the current frame time (i.e., the first frame time) can be used to subsequently accurately generate the 3D object feature information at the current frame time (i.e., the first frame time). Then, the two-dimensional feature information corresponding to each captured image in the captured image set is generated to obtain a two-dimensional feature information set, so as to facilitate the use of the image feature information corresponding to the 2D captured image. Then, according to the first three-dimensional feature information group, the three-dimensional feature information group corresponding to the target detection object set under the first frame time can be accurately generated as the target three-dimensional feature information group. Here, the preliminary 3D object feature information under the current frame time can be accurately mapped and generated by the 3D object feature information under the second frame time. Here, it is not necessary to process multiple continuous historical 3D frame image information under a long time in the historical time period, which can effectively avoid the problem of large amount of calculation. Then, according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set, the second three-dimensional feature information group under the first frame time can be accurately generated, wherein the number of object query information in the object query information group is preset. Here, by setting the query information for each object under the first frame time, the three-dimensional feature information group (i.e., the second three-dimensional feature information group) under the current frame time can be accurately generated by fully considering the target three-dimensional feature information group preliminarily generated at the current time and the two-dimensional feature information set of the current frame time. Finally, according to the second three-dimensional feature information group, the target detection information corresponding to the first frame time can be accurately generated. In summary, the target three-dimensional feature information group after feature information conversion considering the three-dimensional feature information group at the previous frame time, the two-dimensional feature information set at the current frame time, and the object query information group corresponding to the pre-set multiple objects can accurately generate the 3D target detection result at the current frame time with a small amount of calculation without considering multiple historical 3D frame image information.

[0073] Further references Figure 3 , shows a process 300 of another embodiment of the target detection method according to the present disclosure. The target detection method comprises the following steps:

[0074] Step 301, obtaining a captured image set corresponding to a target vehicle in a first frame time and a first three-dimensional feature information group corresponding to a second frame time.

[0075] Step 302: Generate two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set.

[0076] Step 303: Generate a three-dimensional feature information group corresponding to the target detection object set in the first frame time according to the first three-dimensional feature information group as a target three-dimensional feature information group.

[0077] Step 304: Generate a second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set.

[0078] Step 305: Generate target detection information corresponding to the first frame time according to the second three-dimensional feature information group.

[0079] In some embodiments, the specific implementation of steps 301-305 and the technical effects thereof can be referred to in Figure 2 The steps 201-205 in the corresponding embodiment are not described in detail here.

[0080] Step 306: Store the second three-dimensional feature information group and the first frame time in correspondence.

[0081] In some embodiments, an execution entity (e.g. Figure 1 The electronic device 101 shown can store the second three-dimensional feature information group and the first frame time in correspondence.

[0082] Step 307 , in response to receiving the target detection information for the third frame time, using the stored second three-dimensional feature information group to generate the target detection information for the third frame time.

[0083] In some embodiments, in response to receiving the target detection information for the third frame time, the execution entity may generate the target detection information for the third frame time using the second three-dimensional feature information group stored. The third frame time is a frame time that has a future continuous relationship with the first frame time. The target detection information may be a detection request information for 3D object detection for the third frame time. The target detection information for the third frame time may refer to the generation of the target detection information for the first frame time. The future continuous relationship may be a future adjacent frame time.

[0084] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 In the corresponding process 300 of the target detection method in some embodiments, by storing the second three-dimensional feature information group and the corresponding first frame time, it is possible to accurately generate target detection information corresponding to the next frame time from the first frame time with less calculation amount.

[0085] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a target detection device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the target detection device can be specifically applied to various electronic devices.

[0086] like Figure 4 As shown, a target detection device 400 includes: an acquisition unit 401, a first generation unit 402, a second generation unit 403, a third generation unit 404 and a fourth generation unit 405. Among them, the acquisition unit 401 is configured to acquire a captured image set corresponding to the target vehicle at a first frame time and a first three-dimensional feature information group corresponding to a second frame time, wherein the above-mentioned second frame time is a frame time that has a historical continuity relationship with the above-mentioned first frame time; the first generation unit 402 is configured to generate two-dimensional feature information corresponding to each captured image in the above-mentioned captured image set to obtain a two-dimensional feature information set; the second generation unit 403 is configured to generate a three-dimensional feature information group corresponding to the target detection object set at the above-mentioned first frame time according to the above-mentioned first three-dimensional feature information group, as a target three-dimensional feature information group; the third generation unit 404 is configured to generate a second three-dimensional feature information group at the above-mentioned first frame time according to the above-mentioned target three-dimensional feature information group, a pre-constructed object query information group and the above-mentioned two-dimensional feature information set, wherein the number of object query information in the above-mentioned object query information group is pre-set; the fourth generation unit 405 is configured to generate the target detection information corresponding to the above-mentioned first frame time according to the above-mentioned second three-dimensional feature information group.

[0087] In some optional implementation methods of some embodiments, the second generation unit 403 can be further configured to: filter out first three-dimensional feature information whose corresponding object confidence is greater than a preset confidence from the above-mentioned first three-dimensional feature information group as candidate three-dimensional feature information, and obtain a candidate three-dimensional feature information group; obtain spatial transformation information for the above-mentioned first frame time and the above-mentioned second frame time; perform feature information transformation on each candidate three-dimensional feature information in the above-mentioned candidate three-dimensional feature information group according to the above-mentioned spatial transformation information to generate target three-dimensional feature information in the target space coordinate system, and obtain the above-mentioned target three-dimensional feature information group, wherein the above-mentioned target space coordinate system is the space coordinate system corresponding to the above-mentioned first frame time.

[0088] In some optional implementations of some embodiments, the third generation unit 404 can be further configured to: combine the above-mentioned target three-dimensional feature information group and the above-mentioned object query information group to generate a combined information group; input the above-mentioned combined information group into the first attention mechanism model to generate attention mechanism information; transform the above-mentioned two-dimensional feature information set into a three-dimensional space dimension to generate three-dimensional feature information as third three-dimensional feature information; input the above-mentioned third three-dimensional feature information and the above-mentioned attention mechanism information into the second attention mechanism model to generate the above-mentioned second three-dimensional feature information group.

[0089] In some optional implementations of some embodiments, the fourth generation unit 405 may be further configured to: input the second three-dimensional feature information group into a pre-trained detection information generation model to generate three-dimensional target detection information as the target detection information.

[0090] In some optional implementations of some embodiments, the first generation unit 402 may be further configured to: perform image preprocessing on the captured image to generate a preprocessed image; and input the preprocessed image into a pre-trained two-dimensional image feature extraction model to generate the two-dimensional feature information.

[0091] In some optional implementations of some embodiments, the device 400 further includes: a storage unit and a fifth generation unit (not shown). The storage unit may be configured to store the second three-dimensional feature information group and the first frame time in correspondence. The fifth generation unit may be configured to generate target detection information for the third frame time using the stored second three-dimensional feature information group in response to receiving target detection information for the third frame time, wherein the third frame time is a frame time that has a future continuous relationship with the first frame time.

[0092] It is understandable that the units described in the target detection device 400 are similar to those described in the reference Figure 2Therefore, the operations, features and beneficial effects described above for the method are also applicable to the target detection device 400 and the units included therein, and will not be described in detail here.

[0093] Reference below Figure 5 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the electronic device 101)500. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0094] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage device 508 to a random access memory 503. Various programs and data required for the operation of the electronic device 500 are also stored in the random access memory 503. The processing device 501, the read-only memory 502, and the random access memory 503 are connected to each other via a bus 504. An input / output interface 505 is also connected to the bus 504.

[0095] Typically, the following devices may be connected to the input / output interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0096] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the read-only memory 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0097] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0098] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0099] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a captured image set corresponding to the target vehicle at a first frame time and a first three-dimensional feature information group corresponding to a second frame time, wherein the second frame time is a frame time that has a historically continuous relationship with the first frame time; generates two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set; generates a three-dimensional feature information group corresponding to the target detection object set at the first frame time according to the first three-dimensional feature information group as a target three-dimensional feature information group; generates a second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, a pre-constructed object query information group and the two-dimensional feature information set, wherein the number of object query information in the object query information group is preset; generates target detection information corresponding to the first frame time according to the second three-dimensional feature information group.

[0100] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a first generation unit, a second generation unit, a third generation unit, and a fourth generation unit. The names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the first generation unit may also be described as "a unit that generates two-dimensional feature information corresponding to each captured image in the above-mentioned captured image set, and obtains a two-dimensional feature information set".

[0103] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0104] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned target detection methods when executed by a processor.

[0105] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A target detection method, comprising: Acquire a captured image set corresponding to the target vehicle at a first frame time and a first three-dimensional feature information group corresponding to a second frame time, wherein the second frame time is a frame time that has a historical continuous relationship with the first frame time; Generate two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set; generating, according to the first three-dimensional feature information group, a three-dimensional feature information group corresponding to the target detection object set at the first frame time as a target three-dimensional feature information group; generating a second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set, wherein the number of object query information in the object query information group is preset; According to the second three-dimensional feature information group, target detection information corresponding to the first frame time is generated.

2. The method according to claim 1, wherein: The step of generating, according to the first three-dimensional feature information group, a three-dimensional feature information group corresponding to the target detection object set at the first frame time as a target three-dimensional feature information group includes: Selecting first three-dimensional feature information whose corresponding object confidence is greater than a preset confidence from the first three-dimensional feature information group as candidate three-dimensional feature information to obtain a candidate three-dimensional feature information group; Acquire spatial transformation information for the first frame time and the second frame time; According to the spatial transformation information, feature information transformation is performed on each candidate three-dimensional feature information in the candidate three-dimensional feature information group to generate target three-dimensional feature information in a target spatial coordinate system to obtain the target three-dimensional feature information group, wherein the target spatial coordinate system is the spatial coordinate system corresponding to the first frame time.

3. The method according to claim 1, wherein: The step of generating a second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set includes: Combining the target three-dimensional feature information group and the object query information group to generate a combined information group; Inputting the combined information group into a first attention mechanism model to generate attention mechanism information; Converting the two-dimensional feature information set into a three-dimensional space to generate three-dimensional feature information as third three-dimensional feature information; The third three-dimensional feature information and the attention mechanism information are input into a second attention mechanism model to generate the second three-dimensional feature information group.

4. The method according to claim 1, wherein: The step of generating target detection information corresponding to the first frame time according to the second three-dimensional feature information group includes: The second three-dimensional feature information group is input into a pre-trained detection information generation model to generate three-dimensional target detection information as the target detection information.

5. The method according to claim 1, wherein: The generating of the two-dimensional feature information corresponding to each captured image in the captured image set includes: Performing image preprocessing on the captured image to generate a preprocessed image; The preprocessed image is input into a pre-trained two-dimensional image feature extraction model to generate the two-dimensional feature information.

6. The method according to claim 1, wherein: The method further comprises: storing the second three-dimensional feature information group and the first frame time in correspondence; In response to receiving target detection information for a third frame time, the target detection information for the third frame time is generated using the stored second three-dimensional feature information group, wherein the third frame time is a frame time that has a future continuous relationship with the first frame time.

7. A target detection device, comprising: An acquisition unit is configured to acquire a captured image set corresponding to the target vehicle at a first frame time and a first three-dimensional feature information group corresponding to a second frame time, wherein the second frame time is a frame time that has a historical continuous relationship with the first frame time; A first generating unit is configured to generate two-dimensional feature information corresponding to each captured image in the captured image set to obtain a two-dimensional feature information set; A second generating unit is configured to generate, according to the first three-dimensional feature information group, a three-dimensional feature information group corresponding to the target detection object set at the first frame time as a target three-dimensional feature information group; a third generating unit, configured to generate a second three-dimensional feature information group at the first frame time according to the target three-dimensional feature information group, the pre-constructed object query information group and the two-dimensional feature information set, wherein the number of object query information in the object query information group is preset; The fourth generating unit is configured to generate target detection information corresponding to the first frame time according to the second three-dimensional feature information group.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images