Object Detection Method, Training Method, Device, Electronic Device, and Storage Medium

Abstract

The present disclosure provides an object detection method, a training method, an apparatus, an electronic device, and a storage medium, which relate to the technical field of image processing, and particularly to the technical fields of autonomous driving and deep learning. The specific implementation solution is as follows: extracting image features from the image to be processed to obtain a first image feature and at least one second image feature, where the feature scale of the first image feature is greater than the feature scale of the second image feature; determining a depth information feature according to the at least one second image feature; and performing object detection on an object of interest related to the image to be processed according to the depth information feature and the first image feature to obtain an object detection result.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular to the fields of autonomous driving technologies and deep learning technologies. Background Art

[0002] With the rapid development of artificial intelligence technologies, more and more terminals can identify target objects such as obstacles and signs in the surrounding space based on artificial intelligence algorithms to assist the terminals in performing corresponding operations. For example, a driverless vehicle can detect the exact positions of obstacles in the surrounding space based on the acquired images, and then implement autonomous driving functions such as automatic parking and obstacle avoidance according to the detection results. Summary of the Invention

[0003] The present disclosure provides a target detection method, a training method, a device, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, there is provided a target detection method, including: extracting image features from a to-be-processed image to obtain a first image feature and at least one second image feature, where the feature scale of the first image feature is greater than the feature scale of the second image feature; determining a depth information feature according to at least one of the second image features; and performing target detection on a target object related to the to-be-processed image according to the depth information feature and the first image feature to obtain a target object detection result.

[0005] According to another aspect of the present disclosure, there is provided a training method for a deep learning model, including: obtaining training samples, where the training samples include sample images and sample labels related to sample target objects; inputting the sample images into an image feature extraction network of the deep learning model to output a sample first image feature and at least one sample second image feature, where the feature scale of the sample first image feature is greater than the feature scale of the sample second image feature; inputting at least one second image feature into a depth detection network of the deep learning model to output a sample depth information feature; inputting the sample depth information feature and the sample first image feature into a target detection network of the deep learning model to output a sample target object detection result; and training the deep learning model according to the sample target object detection result and the sample labels to obtain a trained deep learning model.

[0006] According to another aspect of the present disclosure, there is provided an object detection device, including: an image feature extraction module configured to perform image feature extraction on an image to be processed, obtaining a first image feature and at least one second image feature, wherein the feature scale of the first image feature is larger than that of the second image feature; a depth information feature determination module configured to determine a depth information feature based on the at least one second image feature; and an object detection result obtaining module configured to perform object detection on an object related to the image to be processed based on the depth information feature and the first image feature, obtaining an object detection result.

[0007] According to another aspect of the present disclosure, there is provided a training device for a deep learning model, including: a training sample obtaining module configured to obtain training samples, the training samples including sample images and sample labels related to sample target objects; a sample image feature extraction module configured to input the sample images into an image feature extraction network of the deep learning model, outputting a sample first image feature and at least one sample second image feature, wherein the feature scale of the sample first image feature is larger than that of the sample second image feature; a sample depth information feature obtaining module configured to input the at least one second image feature into a depth detection network of the deep learning model, outputting a sample depth information feature; a sample object detection result obtaining module configured to input the sample depth information feature and the sample first image feature into an object detection network of the deep learning model, outputting a sample object detection result; and a training module configured to train the deep learning model based on the sample object detection result and the sample labels, obtaining a trained deep learning model.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method provided according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method provided according to an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, there is provided a computer program product including a computer program, where the computer program, when executed by a processor, implements the method provided according to an embodiment of the present disclosure.

[0011] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understandable through the following description. Brief Description of the Drawings

[0012] The drawings are used to better understand the present solution and do not constitute a limitation to the present disclosure. Among them.

[0013] Figure 1 Schematically shows an exemplary system architecture to which the object detection method and apparatus according to an embodiment of the present disclosure can be applied.

[0014] Figure 2 Schematically shows a flowchart of the object detection method according to an embodiment of the present disclosure.

[0015] Figure 3 Schematically shows a schematic diagram of the object detection method according to an embodiment of the present disclosure.

[0016] Figure 4 Schematically shows a schematic diagram of the depth detection network according to an embodiment of the present disclosure.

[0017] Figure 5 Schematically shows a flowchart of the training method of the deep learning model according to an embodiment of the present disclosure.

[0018] Figure 6 Schematically shows a schematic diagram of the training method of the deep learning model according to an embodiment of the present disclosure.

[0019] Figure 7 Schematically shows a block diagram of the object detection apparatus according to an embodiment of the present disclosure.

[0020] Figure 8 Schematically shows a block diagram of the training apparatus of the deep learning model according to an embodiment of the present disclosure.

[0021] Figure 9 Shows a schematic block diagram of an exemplary electronic device 900 that can be used to implement the object detection method and the training method of the deep learning model according to an embodiment of the present disclosure. Detailed Description of the Embodiments

[0022] The following describes exemplary embodiments of the present disclosure with reference to the drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and public order and good customs are not violated.

[0024] With the rapid development of artificial intelligence technology, terminals such as driverless vehicles and intelligent robots can identify target objects such as obstacles and signs in the surrounding space based on object detection algorithms to perform corresponding operations. However, the inventor has found that the conventional object detection algorithms have the problem of low detection accuracy. In order to improve the detection accuracy of the target, it is usually necessary to increase the computational overhead required for object detection, which also causes problems such as long computational time for object detection and object detection latency, resulting in difficulties for related terminals to operate normally.

[0025] Embodiments of the present disclosure provide an object detection method, a training method, an apparatus, an electronic device, a storage medium, and a computer program product. The object detection method includes: extracting image features from a to-be-processed image to obtain a first image feature and at least one second image feature, where the feature scale of the first image feature is larger than the feature scale of the second image feature; determining a depth information feature according to the at least one second image feature; and performing object detection on a target object related to the to-be-processed image according to the depth information feature and the first image feature to obtain an object detection result of the target object.

[0026] According to the embodiments of the present disclosure, by extracting the first image feature and the second image feature with different feature scales from the to-be-processed image, and generating the depth information feature according to the second image feature with a smaller feature scale, the computational amount and computational time for obtaining the depth information feature can be reduced. At the same time, the first image feature can fully retain the image information in the to-be-processed image. Furthermore, object detection is performed according to the depth information feature and the first image feature, which can reduce the overall computational overhead and computational time required for object detection while ensuring the object detection accuracy, thereby improving the object detection efficiency and reducing the dependence of object detection on the hardware computing performance, and at least partially avoiding the time delay of the object detection result. Therefore, according to the object detection method provided by the embodiments of the present disclosure, the object detection efficiency for target objects such as obstacles and moving vehicles in scenarios such as autonomous assisted driving, intelligent driverless vehicles, and unmanned aerial vehicle automatic control can be improved, and the timeliness of obtaining the object detection result can be enhanced.

[0027] Figure 1 An exemplary system architecture to which the object detection method and apparatus according to the embodiments of the present disclosure can be applied is schematically shown.

[0028] It should be noted that Figure 1 The shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, the exemplary system architecture to which the object detection method and apparatus can be applied may include a vehicle, but the vehicle can implement the object detection method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0029] As Figure 1 shown, the system architecture 100 according to this embodiment may include a vehicle 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the vehicle 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] A user may use the vehicle 101 to interact with the server 103 via the network 102 to receive or send messages, etc. An image acquisition device, such as a monocular camera, etc., may be installed on the vehicle 101. A data processing device, such as a chip suitable for processing the acquired images, etc., may also be installed in the vehicle 101.

[0031] The vehicle 101 may be a vehicle with intelligent assisted driving functions, including but not limited to passenger cars, trucks, special operation vehicles, etc.

[0032] The server 103 may be a server that provides various services, such as a background management server, a cloud server, etc. (only for example) that supports the intelligent assisted driving functions of the vehicle 101. The background management server may analyze and process data such as received requests, etc., and feedback the processing results (such as vehicle driving control signals generated according to requests, etc.) to the vehicle. The server 103 may also be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (″Virtual Private Server″, or simply ″VPS″). The server 103 may also be a server of a distributed system, or a server combined with a blockchain.

[0033] It should be noted that the object detection method provided by the embodiments of the present disclosure may generally be executed by the vehicle 101. Correspondingly, the object detection device provided by the embodiments of the present disclosure may also be disposed in the vehicle 101.

[0034] Alternatively, the object detection method provided by the embodiments of the present disclosure may generally also be executed by the server 103. Correspondingly, the object detection device provided by the embodiments of the present disclosure may generally be disposed in the server 103. The object detection method provided by the embodiments of the present disclosure may also be executed by a server or a server cluster different from the server 103 and capable of communicating with the vehicle 101 and / or the server 103. Correspondingly, the object detection device provided by the embodiments of the present disclosure may also be disposed in a server or a server cluster different from the server 103 and capable of communicating with the vehicle 101 and / or the server 103.

[0035] It should be understood, Figure 1The numbers of vehicles, networks, and servers in [it] are merely illustrative. According to the implementation requirements, there can be any number of vehicles, networks, and servers.

[0036] Figure 2 A flowchart of a target detection method according to an embodiment of the present disclosure is schematically shown.

[0037] As Figure 2 shown, the target detection method includes operations S210 to S230.

[0038] In operation S210, image feature extraction is performed on the image to be processed to obtain a first image feature and at least one second image feature, and the feature scale of the first image feature is greater than the feature scale of the second image feature.

[0039] In operation S220, depth information features are determined according to at least one second image feature.

[0040] In operation S230, target detection is performed on a target object related to the image to be processed according to the depth information features and the first image feature to obtain a target object detection result.

[0041] According to an embodiment of the present disclosure, the image to be processed can be an image obtained after image acquisition of a space containing a target object by an image acquisition device. The image acquisition device can be, for example, a monocular image acquisition device such as a monocular camera. Alternatively, the image acquisition device can also be a surround-view image acquisition device. The specific device type of the image acquisition device in the embodiments of the present disclosure is not limited.

[0042] According to an embodiment of the present disclosure, image feature extraction can be performed on the image to be processed based on a neural network algorithm. For example, image feature extraction can be performed on the image to be processed based on a convolutional neural network algorithm. However, it is not limited thereto, and image feature extraction can also be performed on the image to be processed based on other types of neural network algorithms. The specific algorithm type for extracting image features in the embodiments of the present disclosure is not limited, and those skilled in the art can select according to actual needs.

[0043] According to an embodiment of the present disclosure, when performing image feature extraction on the image to be processed to obtain at least one second image feature, it can be by scaling the image to be processed and performing image feature extraction on the scaled image to be processed, so as to obtain a second image feature with a smaller feature scale. However, it is not limited thereto, and a second image feature with a smaller feature scale can also be obtained by downsampling the first image feature. The specific method for obtaining the second image feature in the embodiments of the present disclosure is not limited, and those skilled in the art can select according to actual needs.

[0044] It should be noted that the embodiments of the present disclosure do not limit the respective numbers of the first image feature and the second image feature.

[0045] According to the embodiments of the present disclosure, at least one second image feature can be processed based on a neural network algorithm to extract image information related to the depth attribute of the image to be processed from the second image feature, and then a depth information feature can be obtained. For example, at least one second image feature can be processed based on a convolutional neural network algorithm, or at least one second image feature can also be processed based on a long short-term memory network algorithm. The embodiments of the present disclosure do not limit the specific scoring type for obtaining the depth information feature, and those skilled in the art can select according to actual needs.

[0046] According to the embodiments of the present disclosure, by extracting the first image feature and the second image feature with different feature scales from the image to be processed, and generating the depth information feature based on the second image feature with a smaller feature scale, the calculation amount and calculation duration for obtaining the depth information feature can be reduced, and at the same time, the image information in the image to be processed can be fully retained based on the first image feature with a larger feature scale.

[0047] According to the embodiments of the present disclosure, the depth information feature and the first image feature can be processed based on an object detection algorithm. For example, the depth information feature and the first image feature can be processed based on the LSS (Lift Splat Shoot) algorithm. However, it is not limited thereto, and the depth information feature and the first image feature can also be processed based on other types of object detection algorithms. The embodiments of the present disclosure do not limit the specific algorithm type for processing the depth information feature and the first image feature.

[0048] According to the embodiments of the present disclosure, the first image feature with a larger feature scale can fully retain the image information in the image to be processed, and the depth information feature generated based on the second image feature with a smaller feature scale can accelerate the calculation speed of obtaining the depth information feature and reduce the calculation overhead. Therefore, based on the depth information feature and the first image feature, object detection of the target object related to the image to be processed can generally reduce the calculation overhead and calculation duration required for object detection, thereby improving the object detection efficiency, reducing the dependence of object detection on the hardware computing performance, at least partially avoiding the time delay of the target object detection result, and improving the real-time performance of obtaining the target object detection result.

[0049] It should be noted that the target object detection result can be a two-dimensional detection result representing the target object, or it can also be a three-dimensional detection result representing the target object. The embodiments of the present disclosure do not limit the specific type of the target object detection result, as long as it can represent any one or more object attributes such as the size, type, and motion state of the target object.

[0050] According to an embodiment of the present disclosure, performing image feature extraction on an image to be processed to obtain a first image feature and at least one second image feature may include: performing at least one convolution on the image to be processed to obtain an initial image feature; performing at least one downsampling on the initial image feature to obtain the first image feature; and performing at least one downsampling on the first image feature to obtain at least one second image feature.

[0051] Figure 3 Schematically shows the principle diagram of the object detection method according to an embodiment of the present disclosure.

[0052] As Figure 3 shown, the image 301 to be processed may be an image collected by a surround-view image acquisition device installed on a vehicle. For example, it may be an image of the space around the vehicle collected by image acquisition devices installed at multiple positions on the vehicle.

[0053] The image 301 to be processed may be input into a deep learning model including an image feature extraction network 310, a depth detection network 320, and an object detection network 330 to implement the object detection method provided by the embodiment of the present disclosure.

[0054] Specifically, performing at least one convolution on the image to be processed, for example, may be inputting the image 301 to be processed into the initial image feature extraction layer 311 of the image feature extraction network 310 to output the initial image feature 302. The initial image feature extraction layer 311 may be a lightweight backbone network layer constructed based on the Rep VGG (Visual Geometry Group) network block, so as to improve the calculation speed of obtaining the initial image feature 302 and reduce the calculation overhead.

[0055] As Figure 3 shown, performing at least one downsampling on the initial image feature may be inputting the initial image feature 302 into the first sampling layer 312 to output the first image feature 303. The first sampling layer 312 may be constructed based on a convolution network and a pooling network. The first image feature 303 may be, for example, an image feature obtained by downsampling 8 times relative to the original size of the image 301 to be processed.

[0056] As Figure 3 shown, performing at least one downsampling on the first image feature to obtain at least one second image feature, for example, may be inputting the first image feature 303 into the second downsampling layer 313 to output the first second image feature 304.

[0057] According to an embodiment of the present disclosure, by performing at least one downsampling on the first image feature, at least one second image feature is obtained, avoiding problems such as large computational overhead, long computational duration, and computational overfitting caused by directly performing downsampling on the image to be processed. Furthermore, the timeliness of obtaining the second image feature with a smaller feature size can be further reduced, thereby improving the target detection efficiency as a whole.

[0058] According to an embodiment of the present disclosure, by extracting the first image feature with a larger feature size, the first image feature can contain more image information in the image to be processed, which is beneficial to accurately obtaining object attribute detection results such as detection frames and positions corresponding to the target object based on the first image feature. By extracting the second image feature with a smaller feature size, the second image feature can more accurately correspond to the semantic features in the image to be processed, facilitating obtaining an accurate classification result of the target object subsequently, or it can also be beneficial to accurately perform semantic segmentation on the image to be processed. Therefore, by fusing the first image feature and the second image feature to perform target detection on the target object related to the image to be processed, the detection accuracy of the object attribute detection result and the requirements for target object classification can be satisfied simultaneously.

[0059] According to an embodiment of the present disclosure, there are multiple second image features, and different feature scales exist among the multiple second image features.

[0060] Among them, determining the depth information feature according to at least one second image feature may include: fusing multiple second image features to obtain a fused image feature; and performing depth information detection on the image to be processed according to the fused image feature to obtain the depth information feature.

[0061] As Figure 3 shown, the first second image feature 304 can also be input into the third downsampling layer 314, and the second second image feature 305 is output. The first second image feature 304 can be an image feature obtained by downsampling the original size of the image to be processed 301 by 16 times, and the second second image feature 305 can be an image feature obtained by downsampling the original size of the image to be processed 301 by 32 times.

[0062] As Figure 3 shown, fusing multiple second image features, for example, can be inputting the first second image feature 304 and the second second image feature 305 into the fusion layer 321 of the depth detection network 320, and outputting the fused image feature. The fusion layer 321 can be constructed based on the multi-layer perceptron algorithm, and the fusion layer 321 can also be constructed based on the splicing algorithm. The specific algorithm type for constructing the fusion layer 321 is not limited in the embodiment of the present disclosure.

[0063] As Figure 3As shown, based on the fused image features, depth information detection is performed on the image to be processed to obtain depth information features. For example, the fused image features output by the fusion layer 321 are input into the depth feature detection layer 322 to output depth information features 306. The depth feature detection layer 322 can be constructed based on a convolutional neural network, so as to perform feature extraction on the fused image features, realize the feature depth fusion between the first second image feature 304 and the second second image feature 305, increase the receptive field of the feature representation, and extract the semantic features in the image to be processed 301, so that the depth information features can learn more comprehensive semantic information in the image to be processed 301 and improve the depth representation accuracy of the depth information features.

[0064] It should be understood that Figure 3 the numbers of the second image features 304 and 305 shown in are only exemplary and are not used to limit the number of the second image features. Those skilled in the art can select the actual number of the second image features according to actual needs.

[0065] According to an embodiment of the present disclosure, target detection is performed on a target object related to the image to be processed according to the depth information features and the first image features, and the target object detection result may include: determining a bird's-eye view feature corresponding to the image to be processed according to the depth information features and the first image features; and inputting the bird's-eye view feature into a first target object detection layer constructed based on an attention mechanism to output a target object detection result.

[0066] As Figure 3 shown, the depth information features 306 and the first image features 303 can be input into the bird's-eye view feature detection layer 331 of the target detection network 330 to output a bird's-eye view feature. The bird's-eye view feature detection layer 331 can be, for example, a bird's-eye view detector constructed based on the LSS (Lift Splat Shoot) algorithm. The bird's-eye view feature can also be input into the first target object detection layer 332 to output a target object detection result 307. The first target object detection layer 332 can be an end-to-end detection head constructed based on the Transformer algorithm.

[0067] According to an embodiment of the present disclosure, the target object detection result can be, for example, a detection box corresponding to the target object, a classification result of the target object, a position of the target object, etc. The specific type of the target object detection result in the embodiments of the present disclosure is not limited.

[0068] According to an embodiment of the present disclosure, an end-to-end detection head constructed based on the Transformer algorithm is used to generate target object detection results, which can at least avoid the problem of too long computation time caused by constructing the target object detection head based on the NMS (Non-Maximum Suppression) algorithm, thereby improving the overall efficiency of target detection.

[0069] According to an embodiment of the present disclosure, fusing multiple second image features may include: determining at least one third image feature and at least one fourth image feature from the multiple second image features, where the feature scale of the third image feature is larger than that of the fourth image feature; performing at least one convolution on the fourth image feature to obtain a target fourth image feature; and obtaining a fused image feature according to the third image feature and the target fourth image feature.

[0070] Figure 4 The principle diagram of the depth detection network according to an embodiment of the present disclosure is schematically shown.

[0071] As Figure 4 shown, from two second image features, the second image feature with a larger feature scale can be determined as the third image feature 401, and the second image feature with a smaller feature scale can be determined as the fourth image feature 402. Performing at least one convolution on the fourth image feature may be inputting the fourth image feature 402 into the convolutional layer 411 of the depth detection network 410 to output the target fourth image feature 403. Obtaining a fused image feature according to the third image feature and the target fourth image feature, for example, may be splicing the third image feature 401 input to the depth detection network 410 and the target fourth image feature 403 to obtain the fused image feature. The obtained fused image feature is input to the depth detection layer 412 constructed based on the convolutional neural network algorithm to output the depth information feature 404.

[0072] It should be understood that Figure 4 the depth detection network 410 shown can be used in the target detection method provided in the embodiment of the present disclosure to facilitate extracting depth image features that accurately represent depth information.

[0073] Figure 5 The flowchart of the training method of the deep learning model according to an embodiment of the present disclosure is schematically shown.

[0074] As Figure 5 shown, the training method includes operations S510 to S550.

[0075] In operation S510, training samples are obtained, and the training samples include sample images and sample labels related to the sample target object.

[0076] In operation S520, the sample image is input into the image feature extraction network of the deep learning model, and the sample first image feature and at least one sample second image feature are output. The feature scale of the sample first image feature is larger than the feature scale of the sample second image feature.

[0077] In operation S530, at least one second image feature is input into the depth detection network of the deep learning model, and the sample depth information feature is output.

[0078] In operation S540, the sample depth information feature and the sample first image feature are input into the target detection network of the deep learning model, and the sample target object detection result is output.

[0079] In operation S550, the deep learning model is trained according to the sample target object detection result and the sample label to obtain the trained deep learning model.

[0080] According to an embodiment of the present disclosure, the trained deep learning model can be applied to the target detection method described above. Correspondingly, the deep learning model trained according to the training method of the deep learning model provided by the embodiment of the present disclosure can be applied to the target detection method provided by the embodiment of the present disclosure.

[0081] According to an embodiment of the present disclosure, the sample image may be an image obtained by image acquisition of a space containing the sample target object by an image acquisition device. The image acquisition device may be a monocular image acquisition device such as a monocular camera. Alternatively, the image acquisition device may also be a surround view image acquisition device. The specific device type of the image acquisition device is not limited in the embodiment of the present disclosure.

[0082] According to an embodiment of the present disclosure, the sample first image feature with a larger feature scale fully retains the image information in the sample image, and the sample depth information feature generated according to the sample second image feature with a smaller feature scale can accelerate the calculation speed of obtaining the sample depth information feature and reduce the calculation overhead. Therefore, the trained deep learning model can generally reduce the calculation overhead and calculation duration required for target detection, thereby improving the target detection efficiency, reducing the dependence of target detection on the hardware computing performance, at least partially avoiding the time delay of the target object detection result, and improving the real-time performance of obtaining the target object detection result.

[0083] It should be noted that the technical terms involved in the training method of the deep learning model provided in the embodiments of the present disclosure, including but not limited to sample images, sample first image features, sample second image features, etc., and the corresponding technical terms involved in the object detection method provided in the above embodiments, including but not limited to images to be processed, first image features, second image features, etc., have the same or corresponding technical attributes, and the embodiments of the present disclosure will not elaborate on this again.

[0084] Figure 6 Schematically shows the principle diagram of the training method of the deep learning model according to the embodiments of the present disclosure.

[0085] As Figure 6 shown, the deep learning model may include an image feature extraction network 610, a depth detection network 620, an object detection network 630, and a depth information detection layer 640.

[0086] The sample image 601 may be an image collected by a surround-view image acquisition device installed on a vehicle. For example, it may be an image of the space around the vehicle collected by image acquisition devices installed at multiple positions on the vehicle. The number of sample images may be multiple or may also be one, and the embodiments of the present disclosure do not limit the specific number of sample images.

[0087] As Figure 6 shown, the sample image 601 may be input into the initial image feature extraction layer 611 of the image feature extraction network 610 to output the sample initial image feature 602. The initial image feature extraction layer 611 may be, for example, a lightweight backbone network layer constructed based on the Rep VGG (Visual Geometry Group) network block, so as to improve the calculation speed of obtaining the sample initial image feature 602 and reduce the calculation overhead.

[0088] According to the embodiments of the present disclosure, the image feature extraction network may also be constructed by combining the Rep VGG (Visual Geometry Group) network block and the lightweight SimSPPF (Simplified Spatial Pyramid Pooling - Fast) network block, thereby further improving the object detection efficiency.

[0089] As Figure 6As shown, the sample initial image feature 602 can be input into the first sampling layer 612, and the sample first image feature 603 is output. The first sampling layer 612 can be constructed based on a convolutional network and a pooling network. The sample first image feature 603 can be, for example, an image feature obtained by downsampling the original size of the sample image 601 by 8 times. The sample first image feature 603 can be input into the second downsampling layer 313, and the first sample second image feature 604 is output.

[0090] As Figure 6 shown, the first sample second image feature 604 can also be input into the third downsampling layer 614, and the second sample second image feature 605 is output. The first sample second image feature 604 can be an image feature obtained by downsampling the original size of the sample image 601 by 16 times, and the second sample second image feature 605 can be an image feature obtained by downsampling the original size of the sample image 601 by 32 times.

[0091] As Figure 6 shown, the first sample second image feature 604 and the second sample second image feature 605 can be input into the depth detection network 620, and the sample depth information feature 606 is output. The depth detection network 620 can be constructed based on any type of neural network algorithm such as a multi-layer perceptron algorithm, a convolutional neural network algorithm, etc. Through the depth detection network 620, the feature depth fusion between the first sample second image feature 604 and the second sample second image feature 605 is realized, the receptive field of the feature representation is increased, and the semantic features in the sample image 601 are extracted, so that the sample depth information feature 606 can learn more comprehensive semantic information in the sample image 601 and improve the depth representation accuracy of the depth information feature.

[0092] It should be understood that Figure 6 the numbers of the sample second image features 604 and 605 shown in [[ ]] are only exemplary and are not used to limit the number of the second image features. Those skilled in the art can select the actual number of the second image features according to actual needs.

[0093] As Figure 6 shown, the sample depth information feature and the sample first image feature are input into the object detection network of the deep learning model, and the sample object detection result is output. For example, the sample depth information feature 606 and the sample first image feature 603 can be input into the bird's-eye view feature detection layer 631 of the object detection network 630, and the sample bird's-eye view feature is output. The sample bird's-eye view feature is input into the first object detection layer 632, and the first sample object detection result 6071 is output. The sample bird's-eye view feature can also be input into the second object detection layer 633, and the second sample object detection result 6072 is output.

[0094] According to an embodiment of the present disclosure, the sample target object detection result may include a sample first target object detection result 6071 corresponding to the first target object detection layer 632, and a sample second target object detection result 6072 corresponding to the second target object detection layer 633. The first target object detection layer 632 may be constructed based on an attention mechanism. For example, the first target object detection layer 632 may be constructed based on an attention neural network algorithm. The second target object detection layer 633 may be constructed based on a non-maximum suppression algorithm.

[0095] According to an embodiment of the present disclosure, training the deep learning model according to the sample target object detection result and the sample label may include: training the deep learning model according to the sample first target object detection result, the sample second target object detection result, and the sample label.

[0096] As Figure 6 shown, the sample label may include a sample detection result label 6081 corresponding to the sample first target object detection result 6071 and the sample second target object detection result 6072. The first loss value between the sample detection result label 6081 and the sample first target object detection result 6071 may be calculated respectively based on a loss function, and the second loss value between the sample detection result label 6081 and the sample second target object detection result 6082 may be calculated. The model parameters of the deep learning model are iteratively adjusted according to the first loss value and the second loss value to obtain the trained deep learning model.

[0097] According to an embodiment of the present disclosure, training the deep learning model through the first loss value and the second loss value can enable the obtained trained deep learning model to retain the target object detection advantages of the first target object detection layer constructed based on the attention mechanism and the second target object detection layer constructed based on the non-maximum suppression algorithm respectively, thereby improving the robustness of the deep learning model, while accelerating the convergence speed of the loss function and improving the training rate.

[0098] As Figure 6 shown, the sample label may further include a sample depth information label 6082 corresponding to the sample depth information feature 606. By inputting the sample depth information feature 606 into the depth information detection layer 640 of the deep learning model, the sample depth information 6073 is output. The third loss value between the sample depth information 6073 and the sample depth information label 6082 is calculated based on a loss function, and the model parameters of the deep learning model are iteratively adjusted according to the first loss value, the second loss value, and the third loss value to obtain the trained deep learning model.

[0099] According to an embodiment of the present disclosure, in the case where a trained deep learning model is applied to an object detection method, an object detection result can be generated only through a first object detection layer. By generating an object detection result through an end-to-end first object detection layer constructed based on an attention network algorithm, the problem of excessive computation time caused by constructing an object detection head based on the NMS (Non Maximum Suppression) algorithm can be avoided at least, thereby improving the overall efficiency of object detection.

[0100] Figure 7 A block diagram of an object detection device according to an embodiment of the present disclosure is schematically shown.

[0101] As Figure 7 shown, the object detection device 700 includes: an image feature extraction module 710, a depth information feature determination module 720, and an object detection result obtaining module 730.

[0102] The image feature extraction module 710 is configured to perform image feature extraction on an image to be processed to obtain a first image feature and at least one second image feature, and the feature scale of the first image feature is greater than that of the second image feature.

[0103] The depth information feature determination module 720 is configured to determine a depth information feature according to at least one second image feature.

[0104] The object detection result obtaining module 730 is configured to perform object detection on an object related to the image to be processed according to the depth information feature and the first image feature to obtain an object detection result.

[0105] According to an embodiment of the present disclosure, there are multiple second image features, and different feature scales exist among the multiple second image features.

[0106] Among them, the depth information feature determination module includes: a fused image feature obtaining unit and a depth information feature obtaining unit.

[0107] The fused image feature obtaining unit is configured to fuse multiple second image features to obtain a fused image feature.

[0108] The depth information feature obtaining unit is configured to perform depth information detection on the image to be processed according to the fused image feature to obtain a depth information feature.

[0109] According to an embodiment of the present disclosure, the fused image feature obtaining unit includes: an image feature determination subunit, a target fourth image feature obtaining subunit, and a fused image feature obtaining subunit.

[0110] An image feature determination subunit, configured to determine at least one third image feature and at least one fourth image feature from multiple second image features, where the feature scale of the third image feature is greater than that of the fourth image feature.

[0111] A target fourth image feature obtaining subunit, configured to perform at least one convolution on the fourth image feature to obtain a target fourth image feature.

[0112] A fused image feature obtaining subunit, configured to obtain a fused image feature according to the third image feature and the target fourth image feature.

[0113] According to an embodiment of the present disclosure, the target object detection result obtaining module includes: an aerial view feature obtaining unit and a target object detection result obtaining unit.

[0114] The aerial view feature obtaining unit is configured to determine an aerial view feature corresponding to the image to be processed according to the depth information feature and the first image feature.

[0115] The target object detection result obtaining unit is configured to input the aerial view feature into a first target object detection layer constructed based on an attention mechanism and output a target object detection result.

[0116] According to an embodiment of the present disclosure, the image feature extraction module includes: an initial image feature obtaining unit, a first image feature obtaining unit, and a second image feature obtaining unit.

[0117] The initial image feature obtaining unit is configured to perform at least one convolution on the image to be processed to obtain an initial image feature.

[0118] The first image feature obtaining unit is configured to perform at least one downsampling on the initial image feature to obtain a first image feature.

[0119] The second image feature obtaining unit is configured to perform at least one downsampling on the first image feature to obtain at least one second image feature.

[0120] Figure 8 Schematically shows a block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.

[0121] As Figure 8 shown, the training device 800 for a deep learning model includes: a training sample obtaining module 810, a sample image feature extraction module 820, a sample depth information feature obtaining module 830, a sample target object detection result obtaining module 840, and a training module 850.

[0122] The training sample obtaining module 810 is configured to obtain training samples, where the training samples include sample images and sample labels related to the sample target object.

[0123] The sample image feature extraction module 820 is configured to input a sample image into the image feature extraction network of the deep learning model, and output a sample first image feature and at least one sample second image feature, where the feature scale of the sample first image feature is larger than that of the sample second image feature.

[0124] The sample depth information feature obtaining module 830 is configured to input at least one second image feature into the depth detection network of the deep learning model, and output a sample depth information feature.

[0125] The sample target object detection result obtaining module 840 is configured to input the sample depth information feature and the sample first image feature into the target detection network of the deep learning model, and output a sample target object detection result.

[0126] The training module 850 is configured to train the deep learning model according to the sample target object detection result and the sample label, and obtain a trained deep learning model.

[0127] According to an embodiment of the present disclosure, the target detection network includes a first target object detection layer constructed based on an attention mechanism, and a second target object detection layer constructed based on a non-maximum suppression algorithm. The sample target object detection result includes a sample first target object detection result corresponding to the first target object detection layer, and a sample second target object detection result corresponding to the second target object detection layer.

[0128] Wherein, the training module includes a training unit.

[0129] The training unit is configured to train the deep learning model according to the sample first target object detection result, the sample second target object detection result and the sample label.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0131] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method as described above.

[0132] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0133] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method as described above is implemented.

[0134] Figure 9 FIG. shows a schematic block diagram of an exemplary electronic device 900 that can be used to implement the object detection method and the training method of the deep learning model according to the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0135] As Figure 9 shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 902 or the computer program loaded from the storage unit 908 into the random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The input / output (I / O) interface 905 is also connected to the bus 904.

[0136] A plurality of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0137] The computing unit 901 may be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes the various methods and processes described above, such as the object detection method and the training method of the deep learning model. For example, in some embodiments, the object detection method and the training method of the deep learning model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the object detection method and the training method of the deep learning model described above may be executed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the object detection method and the training method of the deep learning model by any other suitable means (e.g., by means of firmware).

[0138] The various embodiments of the systems and techniques described above in this document may be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, and the programmable processor may be a special or general programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes may be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0142] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0143] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs that run on the respective computers and have a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0144] It should be understood that the various forms of processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0145] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A target detection method, comprising: Inputting an image to be processed into an initial image feature extraction layer to output initial image features; Performing at least one downsampling on the initial image features to obtain first image features; performing downsampling on the first image features to obtain third image features, and performing downsampling on the third image features to obtain fourth image features; wherein, the initial image feature extraction layer is a lightweight backbone network layer, the first image features are image features obtained by downsampling the original size of the image to be processed by 8 times, the third image features are image features obtained by downsampling the original size of the image to be processed by 16 times, and the fourth image features are image features obtained by downsampling the original size of the image to be processed by 32 times; Inputting the fourth image features into the convolutional layer of the depth detection network to output target fourth image features; splicing the third image features input to the depth detection network with the target fourth image features to obtain fused image features; inputting the fused image features into the depth detection layer of the depth detection network to output depth information features; and Inputting the depth information features and the first image features into the bird's-eye view feature detection layer of the target detection network to output bird's-eye view features; inputting the bird's-eye view features into the first target object detection layer to output target object detection results.

2. The method according to claim 1, wherein, The target detection of the target object related to the image to be processed according to the depth information features and the first image features, and obtaining the target object detection result includes: Inputting the bird's-eye view features into the first target object detection layer constructed based on the attention mechanism to output the target object detection result.

3. A training method for a deep learning model, comprising: Obtaining training samples, the training samples including sample images and sample labels related to sample target objects; Inputting the sample images into an initial image feature extraction layer to output sample initial image features; performing at least one downsampling on the sample initial image features to obtain sample first image features; Performing downsampling on the sample first image features to obtain the first sample second image features, and performing downsampling on the first sample second image features to obtain the second sample second image features; wherein, the initial image feature extraction layer is a lightweight backbone network layer, the sample first image features are image features obtained by downsampling the original size of the sample image by 8 times, the first sample second image features are image features obtained by downsampling the original size of the sample image by 16 times, and the second sample second image features are image features obtained by downsampling the original size of the sample image by 32 times; Inputting the second sample second image features into the convolutional layer of the depth detection network to output target features; splicing the first sample image second features input to the depth detection network with the target features to obtain fused features; inputting the fused features into the depth detection layer of the depth detection network to output sample depth information features; Input the sample depth information feature and the sample first image feature into the bird's-eye view feature detection layer of the target detection network to output the sample bird's-eye view feature; input the sample bird's-eye view feature into the first target object detection layer to output the sample first target object detection result; and Train the deep learning model according to the sample target object detection result and the sample label to obtain the trained deep learning model.

4. The method according to claim 3, wherein The target detection network includes the first target object detection layer constructed based on the attention mechanism and the second target object detection layer constructed based on the non-maximum suppression algorithm. The sample target object detection result includes the sample first target object detection result corresponding to the first target object detection layer and the sample second target object detection result corresponding to the second target object detection layer; Among them, the training of the deep learning model according to the sample target object detection result and the sample label includes: Train the deep learning model according to the sample first target object detection result, the sample second target object detection result and the sample label.

5. A target detection device, comprising: An image feature extraction module, configured to input the image to be processed into the initial image feature extraction layer and output the initial image feature; Perform at least one downsampling on the initial image feature to obtain the first image feature; perform downsampling on the first image feature to obtain the third image feature, and perform downsampling on the third image feature to obtain the fourth image feature; wherein, the initial image feature extraction layer is a lightweight backbone network layer, the first image feature is the image feature obtained by downsampling 8 times relative to the original size of the image to be processed, the third image feature is the image feature obtained by downsampling 16 times relative to the original size of the image to be processed, and the fourth image feature is the image feature obtained by downsampling 32 times relative to the original size of the image to be processed; A depth information feature determination module, configured to input the fourth image feature into the convolutional layer of the depth detection network to output the target fourth image feature; splice the third image feature input to the depth detection network with the target fourth image feature to obtain a fused image feature; input the fused image feature into the depth detection layer of the depth detection network to output the depth information feature; and A target object detection result obtaining module, configured to input the depth information feature and the first image feature into the bird's-eye view feature detection layer of the target detection network to output the bird's-eye view feature; input the bird's-eye view feature into the first target object detection layer to output the target object detection result.

6. The apparatus according to claim 5, wherein, The target object detection result obtaining module includes: A target object detection result obtaining unit, configured to input the bird's-eye view feature into the first target object detection layer constructed based on the attention mechanism to output the target object detection result.

7. A training device for a deep learning model, comprising: A training sample obtaining module, configured to obtain training samples, where the training samples include sample images and sample labels related to the sample target object; A sample image feature extraction module, configured to input the sample image into an initial image feature extraction layer and output sample initial image features; perform at least one downsampling on the sample initial image features to obtain sample first image features; Perform downsampling on the sample first image features to obtain a first sample second image feature, and perform downsampling on the first sample second image feature to obtain a second sample second image feature; A sample depth information feature acquisition module, configured to input the second sample second image feature into a convolutional layer of a depth detection network and output target features; splice the first sample image second feature input to the depth detection network with the target features to obtain fused features; input the fused features into a depth detection layer of the depth detection network and output sample depth information features; A sample target object detection result acquisition module, configured to input the sample depth information features and the sample first image features into an aerial view feature detection layer of a target detection network and output sample aerial view features; Input the sample aerial view features into a first target object detection layer and output a first sample target object detection result; And A training module, configured to train the deep learning model according to the sample target object detection result and the sample label to obtain a trained deep learning model; Wherein, the initial image feature extraction layer is a lightweight backbone network layer, the sample first image features are image features obtained by downsampling 8 times relative to the original size of the sample image, the first sample second image feature is an image feature obtained by downsampling 16 times relative to the original size of the sample image, and the second sample second image feature is an image feature obtained by downsampling 32 times relative to the original size of the sample image.

8. The apparatus according to claim 7, wherein The target detection network includes the first target object detection layer constructed based on an attention mechanism and a second target object detection layer constructed based on a non-maximum suppression algorithm, and the sample target object detection result includes the first sample target object detection result corresponding to the first target object detection layer and the sample second target object detection result corresponding to the second target object detection layer; Wherein, the training module includes: A training unit, configured to train the deep learning model according to the first sample target object detection result, the second sample target object detection result and the sample label.

9. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 4.

11. A computer program product, comprising a computer program, where the computer program, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images