Target detection model training method and target detection method combined with camera parameters

Abstract

The present disclosure provides a target detection model training method and a target detection method combined with camera parameters, which can be applied to the technical field of computer vision. The method comprises: acquiring a target detection image, wherein the target detection image is an image collected at an inclination angle greater than a preset angle threshold, and the target detection image has target annotation information; inputting the target detection image into an initial target detection network to output an initial target detection result of the target detection image; determining a pixel depth detection image with pixel depth information according to the target detection image and camera parameter information; inputting the pixel depth detection image into a pixel depth detection network to output a pixel depth detection result of the pixel depth detection image; and training the initial target detection network and the pixel depth detection network based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain a trained target detection model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and more particularly, to a target detection model training method and a target detection method combined with camera parameters. BACKGROUND

[0002] In the related art, emerging devices such as optical satellites, aerostats, and unmanned aerial vehicles all use optical cameras to collect images on the ground. Due to the existence of a certain tilt angle when collecting images, there is a large difference between the target objects in the images and the actual situation, thereby causing difficulties in image classification, detection, and other tasks.

[0003] In the process of implementing the present disclosure, the inventors have found that at least the following problems exist in the related art: the information of the target detection image is not fully utilized in the related art, resulting in low target detection accuracy. SUMMARY

[0004] Therefore, the present disclosure provides a target detection model training method and device combined with camera parameters, and a target detection method.

[0005] One aspect of the present disclosure provides a target detection model training method combined with camera parameters, comprising:

[0006] obtaining a target detection image, wherein the target detection image is an image collected with a tilt angle greater than a preset angle threshold, and the target detection image has target annotation information;

[0007] inputting the target detection image into an initial target detection network to output an initial target detection result of the target detection image;

[0008] determining a pixel depth detection image with pixel depth information according to the target detection image and camera parameter information;

[0009] inputting the pixel depth detection image into a pixel depth detection network to output a pixel depth detection result of the pixel depth detection image;

[0010] training the initial target detection network and the pixel depth detection network based on the target annotation information, the pixel depth information, the initial target detection result, and the pixel depth detection result to obtain a trained target detection model.

[0011] According to an embodiment of the present disclosure, inputting the target detection image into the initial target detection network to output the initial target detection result of the target detection image comprises:

[0012] inputting the target detection image into a first down-sampling layer of the initial target detection network to output a plurality of scale first feature maps;

[0013] input the first feature maps of multiple scales to a first fusion feature layer of the initial detection network, and output second feature maps of multiple scales;

[0014] input the second feature maps of multiple scales to a region of interest extraction layer of the initial detection network, and output a region of interest of each of the second feature maps of multiple scales;

[0015] input the multiple regions of interest to a prediction layer of the initial detection network, and output the initial target detection result of the target detection image.

[0016] According to an embodiment of the present disclosure, wherein the pixel depth detection image is input into the pixel depth detection network, and a pixel depth detection result of the pixel depth detection image is output, comprising:

[0017] input the pixel depth detection image into a second down-sampling layer of the pixel depth detection network, and output third feature maps of multiple scales;

[0018] input the third feature maps of multiple scales into a second fusion feature layer of the pixel depth detection network, and output fourth feature maps of multiple scales;

[0019] output the fourth feature maps of multiple scales to three-dimensional depth perception layers of the pixel depth detection network respectively, and output the pixel depth detection result of the pixel depth detection image.

[0020] According to an embodiment of the present disclosure, wherein the initial target detection network and the pixel depth detection network are trained based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result, to obtain a trained target detection model, comprising:

[0021] determine a target detection loss value based on the initial target detection result and the target annotation information;

[0022] determine a pixel depth detection loss value based on the pixel depth detection result and the pixel depth information;

[0023] train the initial target detection network and the pixel depth detection network based on the target detection loss value and the pixel depth detection loss value, to obtain the trained target detection model.

[0024] According to an embodiment of the present disclosure, wherein the target annotation information comprises target coordinate information and target category information, and the target detection loss value is determined based on the initial target detection result and the target annotation information, comprising:

[0025] determine a target coordinate detection loss value based on the initial target detection result and the target coordinate information;

[0026] determine a target category detection loss value based on the initial target detection result and the target category information;

[0027] The target detection loss value is determined based on the target coordinate detection loss value and the target category detection loss value.

[0028] According to an embodiment of the present disclosure, the pixel depth detection image with pixel depth information is determined according to the target detection image and the camera parameter information, comprising:

[0029] The target detection image pixel depth information is determined according to the target annotation information and the camera parameter information, and a first sub-pixel depth detection image is obtained.

[0030] According to a preset pixel threshold, a plurality of clustering regions of the first sub-pixel depth detection image are determined by using a K-means clustering algorithm.

[0031] According to the pixel depth values in the plurality of clustering regions, a second sub-pixel depth detection image is determined from the plurality of clustering regions.

[0032] The pixel depth detection image is determined based on the first sub-pixel depth detection image and the second sub-pixel depth detection image.

[0033] According to an embodiment of the present disclosure, the target detection loss value, the target coordinate detection loss value and the target category detection loss value are respectively represented by the following formula:

[0034] L detection =L loc +L cls (1)

[0035]

[0036]

[0037] wherein L detection represents the target detection loss value, L loc represents the target coordinate detection loss value, L cls represents the target category detection loss value, x represents an x-axis coordinate value of the upper left corner of the target position frame in the target coordinate information, y represents a y-axis coordinate value of the upper left corner of the target position frame in the target coordinate information, w represents a width value of the target position frame in the target coordinate information, h represents a height value of the target position frame in the target coordinate information, t i represents the target coordinate information, μ i represents the target coordinate detection result in the initial target detection result, T represents the number of target categories in the target detection image, y i represents the target category information, S j represents the target category detection result in the initial target detection result.

[0038] Another aspect of the present disclosure provides a target detection method combined with camera parameter information, comprising:

[0039] inputting a target image to be detected into a target detection model trained by the target detection model training method combined with camera parameter information, and outputting an initial target detection result and a pixel depth detection result;

[0040] determining a target detection result by using a non-maximum suppression method based on the initial target detection result and the pixel depth detection result.

[0041] Another aspect of the present disclosure provides a target detection model training device combined with camera parameter information, comprising:

[0042] a first acquisition module configured to acquire a target detection image, wherein the target detection image is provided with target annotation information;

[0043] a first output module configured to input the target detection image into an initial target detection network, and output an initial target detection result of the target detection image;

[0044] a first determination module configured to determine a pixel depth detection image with pixel depth information according to the target detection image and camera parameter information;

[0045] a second output module configured to input the pixel depth detection image into a pixel depth detection network, and output a pixel depth detection result of the pixel depth detection image;

[0046] a obtaining module configured to train the initial target detection network and the pixel depth detection network based on the initial target detection result and the pixel depth detection result, and obtain a trained target detection model.

[0047] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the target detection model training method combined with camera parameter information.

[0048] According to the technical solution provided in the embodiment of the present disclosure, the target detection image with target annotation information is input into the target detection network to output an initial target detection result, the pixel depth detection image with pixel depth information determined according to the target detection image and the camera parameter is input into the pixel depth detection network to output a pixel depth detection result, the initial target detection network and the pixel depth detection network are trained based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain the target detection model, and the target information in the target detection image is combined with the pixel depth to more accurately determine the target in the target detection image, so that the technical problem of low target detection accuracy caused by the fact that the information of the target detection image is not fully utilized in the related art is at least partially overcome. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0050] Figure 1 An exemplary system architecture to which the target detection model training method incorporating camera parameter information according to an embodiment of the present disclosure can be applied is schematically shown;

[0051] Figure 2 A flowchart of the target detection model training method incorporating camera parameter information according to an embodiment of the present disclosure is schematically shown;

[0052] Figure 3 A structure diagram of the target detection model according to an embodiment of the present disclosure is schematically shown;

[0053] Figure 4 A structure diagram of a three-dimensional depth perception layer according to an embodiment of the present disclosure is schematically shown;

[0054] Figure 5 A flowchart of the target detection method incorporating camera parameter information according to an embodiment of the present disclosure is schematically shown;

[0055] Figure 6 A block diagram of the target detection model training apparatus incorporating camera parameter information according to an embodiment of the present disclosure is schematically shown;

[0056] Figure 7 A block diagram of the target detection apparatus incorporating camera parameter information according to an embodiment of the present disclosure is schematically shown; and

[0057] Figure 8 A block diagram of an electronic device suitable for implementing the target detection model training method incorporating camera parameter information described above according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0058] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that one or more embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have been omitted or simply referenced in order not to obscure an understanding of this disclosure.

[0059] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so on, mean the term "comprises," unless otherwise noted.

[0060] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings that are consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0061] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include any of them alone, any combination of two or more of them, and the like. In the case of using expressions similar to "at least one of A, B, or C, etc.", it should generally be interpreted to include any of them alone, any combination of two or more of them, and the like.

[0062] In the technical solutions of the present disclosure, the acquisition, storage, and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0063] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before acquiring or collecting user personal information.

[0064] In the related art, due to the diversity and flexibility of optical satellites, aerostats, unmanned aerial vehicles and other devices, the camera collects images on the ground in a complex and variable manner. In the ground shooting scene with variable carrier height and variable camera angle, there will be problems such as large difference in target feature of large-angle image and image domain difference. This will bring great challenges to downstream problems such as image classification, segmentation and detection of computer vision.

[0065] With the development of deep learning technology, target detection technology based on convolutional neural network has achieved good detection effect. Existing target detection methods can be divided into two categories according to whether candidate region extraction is used: Yolo (You Only Look Once, a target detector) and RetinaNet (a target detection network) and other one-stage series methods. This kind of method does not need candidate region extraction, directly regresses the classification prediction of the target and the position information of the target, and the detection speed is fast; Faster-Rcnn (a target detection network) and Cascade-Rcnn (a target detection network) and other two-stage series methods. This kind of method is based on candidate region extraction, classification prediction and position coordinate fine tuning, so the detection accuracy is higher.

[0066] However, most of the target detection methods in the related art are based on the extraction and post-processing method of two-dimensional features of the image. Due to the lack of perception and extraction of three-dimensional depth information contained in the two-dimensional image, the performance of the target detector is obviously reduced for images with large target depth variation range.

[0067] Therefore, the embodiments of the present disclosure provide a target detection model training method combined with camera parameters. The method comprises: acquiring a target detection image, wherein the target detection image is an image with a large inclination angle greater than a preset angle threshold, and the target detection image has target annotation information; inputting the target detection image into an initial target detection network to output an initial target detection result of the target detection image; determining a pixel depth detection image with pixel depth information according to the target detection image and camera parameter information; inputting the pixel depth detection image into a pixel depth detection network to output a pixel depth detection result of the pixel depth detection image; training the initial target detection network and the pixel depth detection network based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain a trained target detection model.

[0068] Figure 1 An exemplary system architecture 100 to which the target detection model training method combined with camera parameters according to embodiments of the present disclosure can be applied is schematically shown. It should be noted that, Figure 1The shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0069] As shown, the system architecture 100 according to the embodiment can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc. Figure 1

[0070] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0071] The terminal devices 101, 102, 103 can be various electronic devices with display screens and support for web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0072] The server 105 can be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101, 102, 103 (only as an example). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information or data obtained or generated according to user requests, etc.) to the terminal devices.

[0073] ​It should be noted that the target detection model training method provided by the embodiments of the present disclosure can be generally executed by the server 105. Accordingly, the target detection model training method device provided by the embodiments of the present disclosure can be generally arranged in the server 105. The target detection model training method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, 103 and / or the server 105. Accordingly, the target detection model training device provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, 103 and / or the server 105. Alternatively, the target detection model training method provided by the embodiments of the present disclosure can also be executed by the terminal device 101, 102, or 103, or by other terminal devices different from the terminal device 101, 102, or 103. Accordingly, the target detection model training method device provided by the embodiments of the present disclosure can also be arranged in the terminal device 101, 102, or 103, or in other terminal devices different from the terminal device 101, 102, or 103.

[0074] For example, the target detection image can be originally stored in any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto), or stored on an external storage device and can be imported into the terminal device 101. Then, the terminal device 101 can execute the target detection model training method provided by the embodiments of the present disclosure locally, or send the target detection image to other terminal devices, servers, or server clusters, and execute the target detection model training method provided by the embodiments of the present disclosure by other terminal devices, servers, or server clusters receiving the target detection image.

[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system shown in

[0076] Figure 2 A flowchart of the target detection model training method according to the embodiments of the present disclosure is schematically shown.

[0077] As Figure 2 shown, the method includes operations S201-S205.

[0078] In operation S201, a target detection image is acquired, where the target detection image is an image with a collection tilt angle greater than a preset angle threshold, and the target detection image has target annotation information.

[0079] According to an embodiment of the present disclosure, the target detection image can be an image taken from a satellite, an airship, a drone, or the like, and the collection tilt angle of the target detection image is greater than a preset angle threshold.

[0080] According to an embodiment of the present disclosure, the target annotation information of the target detection image can be information for labeling a target in the target detection image.

[0081] According to an embodiment of the present disclosure, there can be multiple target detection images so as to provide sufficient training data for the target detection model.

[0082] In operation S202, the target detection image is input to an initial target detection network to output an initial target detection result of the target detection image.

[0083] According to an embodiment of the present disclosure, the initial target detection network can be a network for roughly detecting the target detection image to obtain the initial target detection result.

[0084] In operation S203, a pixel depth detection image with pixel depth information is determined according to the target detection image and camera parameter information.

[0085] According to an embodiment of the present disclosure, the camera parameter information can be determined according to a shooting device of the target detection image, and can include a camera height, an aperture size, an exposure, and the like.

[0086] According to an embodiment of the present disclosure, the pixel depth information can be determined by combining a pixel point of the target detection image and the camera parameter information to determine the pixel depth detection image, and the pixel depth information can be a distance from a pixel point of an image to an actual object.

[0087] In operation S204, the pixel depth detection image is input to a pixel depth detection network to output a pixel depth detection result of the pixel depth detection image.

[0088] According to an embodiment of the present disclosure, the pixel depth detection network can be a network for detecting pixel depth in an image, and the pixel depth detection image can be input to the pixel depth detection network to output the pixel depth detection result of the pixel depth detection image.

[0089] In operation S205, the initial target detection network and the pixel depth detection network are trained based on the target annotation information, the pixel depth information, the initial target detection result, and the pixel depth detection result to obtain a trained target detection model.

[0090] According to an embodiment of the present disclosure, the deviation value of the initial target detection result and the target annotation information can be determined based on the target annotation information and the initial target detection result, the deviation value of the pixel depth detection result and the pixel depth information can be determined based on the pixel depth information and the pixel depth detection result, and the initial target detection network and the pixel depth detection network can be trained according to the two deviation values to obtain the target detection model.

[0091] According to an embodiment of the present disclosure, because the target detection image with target annotation information is input into the target detection network to output the initial target detection result, the pixel depth detection image with pixel depth information determined according to the target detection image and the camera parameter is input into the pixel depth detection network to output the pixel depth detection result, and the initial target detection network and the pixel depth detection network are trained based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain the target detection model, the target information in the target detection image is combined with the pixel depth, and the target in the target detection image can be determined more accurately, so that at least part of the technical problems that the information of the target detection image is not fully utilized in the related art, resulting in low target detection accuracy are overcome.

[0092] According to an embodiment of the present disclosure, wherein the initial target detection network is input into the target detection image, and the initial target detection result of the target detection image is output, comprising:

[0093] The target detection image is input into the first down-sampling layer of the initial target detection network, and a plurality of scales of first feature maps are output;

[0094] The plurality of scales of first feature maps are input into the first fusion feature layer of the initial detection network, and a plurality of scales of second feature maps are output;

[0095] The plurality of scales of second feature maps are input into the region of interest extraction layer of the initial detection network, and a region of interest of each of the plurality of scales of second feature maps is output;

[0096] The plurality of regions of interest are input into the prediction layer of the initial detection network, and the initial target detection result of the target detection image is output.

[0097] According to an embodiment of the present disclosure, the target detection image can be taken as a tensor with a size of bs×3×512×512, where bs is the number of target detection images input into the initial target detection network in each batch, for example, bs=8 can be taken, ‘3’ represents the RGB three channels of the target detection image, and ‘512×512’ represents the resolution size of the target detection image.

[0098] According to an embodiment of the present disclosure, an initial target detection network can be constructed by using a perception (MMDetection) platform.

[0099] According to an embodiment of the present disclosure, the first down-sampling layer can be constructed based on a backbone extraction network, and specifically, a ResNet50 network can be used. The target detection image can be input into the first down-sampling layer, and a plurality of scale first feature maps can be output. Each scale can correspond to a first down-sampling layer. Therefore, there can be a plurality of first down-sampling layers, for example, four first down-sampling layers, which can be a 4 times down-sampling layer, an 8 times down-sampling layer, a 16 times down-sampling layer, and a 32 times down-sampling layer. Four scale first feature maps can be output, and other multiple down-sampling can also be used.

[0100] According to an embodiment of the present disclosure, the first fusion feature layer can fuse information in the plurality of scale first feature maps, optimize the first feature maps, and obtain a plurality of scale second feature maps. The first fusion feature layer can be constructed based on a multi-level feature fusion network (Feature Pyramid Network, FPN).

[0101] According to an embodiment of the present disclosure, the region of interest extraction layer can extract a region where a target can exist in each scale second feature map, and output a region of interest in each second feature map. The region of interest extraction layer can be constructed based on a region proposal network (Region Proposal Network, RPN). The number of regions of interest can be specified, for example, the number of regions of interest can be specified as 1000.

[0102] According to an embodiment of the present disclosure, the prediction layer can use a cascaded Head to perform multiple times of serial classification of categories and fine-tuning of position coordinates on the basis of the region of interest output by the region of interest extraction layer, so as to output an initial target detection result of the target detection image. Specifically, three Heads can be cascaded, and the iou threshold of each cascaded Head can be 0.5, 0.6, and 0.7 in turn.

[0103] According to an embodiment of the present disclosure, the target in the target detection image can be detected by using the initial target detection network, and the position of the target can be preliminarily determined.

[0104] According to an embodiment of the present disclosure, the pixel depth detection image is input into the pixel depth detection network, and a pixel depth detection result of the pixel depth detection image is output, including:

[0105] The pixel depth detection image is input into a second down-sampling layer of the pixel depth detection network, and a plurality of scale third feature maps are output.

[0106] The third feature maps of multiple scales are input to a second fusion feature layer of the pixel depth detection network, and fourth feature maps of multiple scales are output.

[0107] The fourth feature maps of multiple scales are respectively output to three-dimensional depth perception layers of the pixel depth detection network, and a pixel depth detection result of the pixel depth detection image is output.

[0108] According to an embodiment of the present disclosure, the second down-sampling layer can be the same network layer as the first down-sampling layer, and the second fusion feature layer can be the same network layer as the first fusion feature layer, which will not be described herein again.

[0109] According to an embodiment of the present disclosure, the fourth feature maps of each scale can correspond to one three-dimensional depth perception layer, and the feature information extracted in each fourth feature map is fused, so as to output the pixel depth detection result of the pixel depth detection image.

[0110] According to an embodiment of the present disclosure, a depth residual convolution layer is added in series after the three-dimensional depth perception layer, the depth residual convolution layer includes a convolution layer, a BN layer, a ReLU layer and a residual connection layer, taking the 4 times down-sampling feature map, the 8 times down-sampling feature map and the 16 times down-sampling feature map as examples, the 16 times down-sampling feature map is spliced with the 8 times down-sampling feature map after up-sampling, and after dimension reduction by convolution, the 16 times down-sampling feature map is taken as the input of the depth residual convolution layer of the 8 times down-sampling feature map; similarly, the 8 times down-sampling feature map is spliced with the 4 times down-sampling feature map after up-sampling, and taken as the input of the depth residual convolution layer; the 4 times down-sampling feature map is output by the depth residual convolution layer and is output by 2 times up-sampling to output the pixel depth detection result.

[0111] Figure 3 A structure diagram of a target detection model according to an embodiment of the present disclosure is schematically shown.

[0112] As shown in Figure 3 , the target detection model 300 includes an initial target detection network 310 and a pixel depth detection network 320, the initial target detection network 310 includes a first down-sampling layer, a first fusion feature layer, a region of interest extraction layer and a prediction layer, and the pixel depth detection network 320 includes a second down-sampling layer, a second fusion feature layer and a three-dimensional depth perception layer.

[0113] Figure 4 A structure diagram of a three-dimensional depth perception layer according to an embodiment of the present disclosure is schematically shown.

[0114] As shown in Figure 4As shown, taking the third feature maps of multiple scales as examples of 4 times down-sampling feature maps, 8 times down-sampling feature maps and 16 times down-sampling feature maps, a three-dimensional depth perception layer is connected after each scale of the third feature maps, wherein, a 3*3 convolution layer, a normalization layer, a ReLU activation layer, a 3*3 convolution layer, a normalization layer, a ReLU activation layer and a depth residual convolution layer are connected after the 16 times down-sampling feature maps, and the three-dimensional depth perception layers connected with the 4 times down-sampling feature maps and the 8 times down-sampling feature maps are similar to the three-dimensional depth perception layer connected with the 16 times down-sampling feature maps, and details are not repeated here.

[0115] According to an embodiment of the present disclosure, wherein the initial target detection network and the pixel depth detection network are trained based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain the trained target detection model, comprising:

[0116] Based on the initial target detection result and the target annotation information, a target detection loss value is determined;

[0117] Based on the pixel depth detection result and the pixel depth information, a pixel depth detection loss value is determined;

[0118] The initial target detection network and the pixel depth detection network are trained based on the target detection loss value and the pixel depth detection loss value to obtain the trained target detection model.

[0119] According to an embodiment of the present disclosure, based on the initial target detection result and the target annotation information, a target detection loss value in the initial target detection network can be determined, and based on the pixel depth detection result and the pixel depth information, a pixel depth detection loss value in the pixel depth detection network can be determined.

[0120] According to an embodiment of the present disclosure, all parameters in the initial target detection network and the pixel depth detection network can be updated by using the gradient descent method until the network converges to obtain the trained target detection model.

[0121] According to an embodiment of the present disclosure, wherein the target annotation information includes target coordinate information and target category information, based on the initial target detection result and the target annotation information, a target detection loss value is determined, comprising:

[0122] Based on the initial target detection result and the target coordinate information, a target coordinate detection loss value is determined;

[0123] Based on the initial target detection result and the target category information, a target category detection loss value is determined;

[0124] Based on the target coordinate detection loss value and the target category detection loss value, a target detection loss value is determined.

[0125] According to an embodiment of the present disclosure, the target coordinate information can be coordinates of upper left corner, lower left corner, upper right corner and lower right corner of the target position box and width and height of the target position box, and the target category information can be category information of the target, for example, can be a person, a vehicle, an animal, etc.

[0126] According to an embodiment of the present disclosure, the initial target detection result can include a target coordinate detection result and a target category detection result, a target coordinate detection loss value can be determined based on the target coordinate detection result and the target coordinate information, and a target category detection loss value can be determined based on the target category detection result and the target category information.

[0127] According to an embodiment of the present disclosure, the target coordinate detection loss value and the target category detection loss value can be added to determine a target detection loss value.

[0128] According to an embodiment of the present disclosure, wherein, based on the target detection image and the camera parameter information, a pixel depth detection image with pixel depth information is determined, comprising:

[0129] According to the target annotation information and the camera parameter information, target detection image pixel depth information is determined to obtain a first sub-pixel depth detection image;

[0130] According to a preset pixel threshold, a plurality of clustering regions of the first sub-pixel depth detection image are determined by using a K-means clustering algorithm;

[0131] According to the pixel depth values in the plurality of clustering regions, a second sub-pixel depth detection image is determined from the plurality of clustering regions;

[0132] Based on the first sub-pixel depth detection image and the second sub-pixel depth detection image, a pixel depth detection image is determined.

[0133] According to an embodiment of the present disclosure, based on the assumption that all pixels in the target detection image are on the same plane, the rough depth of each pixel can be calculated using the target annotation information and the camera parameter information, the relative depth information in the same target detection image can be generated according to the actual size of each target and the longest side and the shortest side of the minimum circumscribed rectangle of the target coordinate information in the target annotation information, and the relatively accurate primary pixel depth map can be calculated by combining the rough depth and the relative depth information. Then, the primary pixel depth map is optimized into the first sub-pixel depth detection image by using the conditional radiation field.

[0134] According to an embodiment of the present disclosure, the first sub-pixel depth detection image can be converted into a mask image in 0-1 format by using a preset pixel threshold. Specifically, the preset pixel threshold can be set to 0.65, the mask image can be clustered by using a K-means clustering algorithm to obtain a plurality of clustering regions, and each clustering region can be an image.

[0135] According to an embodiment of the present disclosure, the pixel depth values in each cluster region are different, and according to the difference of the pixel depth values in each cluster region, the image corresponding to the two cluster regions with larger pixel depth values is reserved as the second sub-pixel depth detection image.

[0136] According to an embodiment of the present disclosure, the first sub-pixel depth detection image and the second sub-pixel depth detection image are used as the pixel depth detection images for training the pixel depth detection network.

[0137] According to an embodiment of the present disclosure, since the target in the second sub-pixel depth detection image is more accurate, using the second sub-pixel depth detection image as the training sample of the pixel depth detection network can make the pixel depth detection network determine the pixel depth of the image more quickly.

[0138] According to an embodiment of the present disclosure, the target detection loss value, the target coordinate detection loss value and the target category detection loss value are respectively represented by the following formula:

[0139] L detection =L loc +L cls (1)

[0140]

[0141]

[0142] wherein, L detection represents the target detection loss value, L loc represents the target coordinate detection loss value, L cls represents the target category detection loss value, x represents the x-axis coordinate value of the upper left corner of the target position frame in the target coordinate information, y represents the y-axis coordinate value of the upper left corner of the target position frame in the target coordinate information, w represents the width value of the target position frame in the target coordinate information, h represents the height value of the target position frame in the target coordinate information, t i represents the target coordinate information, μ i represents the target coordinate detection result in the initial target detection result, T represents the number of target categories in the target detection image, y i represents the target category information, S j represents the target category detection result in the initial target detection result.

[0143] According to an embodiment of the present disclosure, the pixel depth detection loss value can be represented by the following formula (4):

[0144]

[0145] wherein, L depth represents the pixel depth detection loss value, y prepresenting a pixel depth detection result, representing a pixel depth information, p represents a pth pixel depth detection image, and n represents n pixel depth detection images.

[0146] Figure 5 A flowchart of a target detection method combined with camera parameter information according to an embodiment of the present disclosure is schematically shown.

[0147] As Figure 5 shown, the method includes operation S501 to operation S502.

[0148] In operation S501, the target image to be detected is input into the target detection model trained by the target detection model training method combined with camera parameter information as described above, and an initial target detection result and a pixel depth detection result are output;

[0149] In operation S502, the target detection result is determined by using a non-maximum suppression method based on the initial target detection result and the pixel depth detection result.

[0150] According to an embodiment of the present disclosure, the initial target detection result and the first sub-pixel depth detection image of the target image to be detected are calculated by the target detection model trained in the embodiment of the present disclosure, the first sub-pixel depth detection image is clustered to obtain two second sub-pixel depth detection images, the second sub-pixel depth detection images and the pixel depth detection result are fused by using the non-maximum suppression method, and the target detection result is determined.

[0151] Figure 6 A block diagram of a target detection model training device combined with camera parameter information according to an embodiment of the present disclosure is schematically shown.

[0152] As Figure 6 shown, the target detection model training device combined with camera parameter information 600 includes a first acquisition module 610, a first output module 620, a first determination module 630, a second output module 640, and an obtaining module 650.

[0153] The first acquisition module 610 is configured to acquire a target detection image, wherein the target detection image has target annotation information.

[0154] The first output module 620 is configured to input the target detection image into an initial target detection network, and output an initial target detection result of the target detection image.

[0155] The first determination module 630 is configured to determine a pixel depth detection image with pixel depth information according to the target detection image and the camera parameter information.

[0156] The second output module 640 is configured to input the pixel depth detection image into the pixel depth detection network, and output a pixel depth detection result of the pixel depth detection image.

[0157] The obtaining module 650 is configured to train the initial target detection network and the pixel depth detection network based on the initial target detection result and the pixel depth detection result, and obtain a trained target detection model.

[0158] According to an embodiment of the present disclosure, the first output module 620 for inputting the target detection image into the initial target detection network and outputting an initial target detection result of the target detection image comprises:

[0159] The first output unit is configured to input the target detection image into a first down-sampling layer of the initial target detection network, and output a plurality of scales of first feature maps.

[0160] The second output unit is configured to input the plurality of scales of first feature maps into a first fusion feature layer of the initial detection network, and output a plurality of scales of second feature maps.

[0161] The third output unit is configured to input the plurality of scales of second feature maps into a region of interest extraction layer of the initial detection network, and output a region of interest of each of the plurality of scales of second feature maps.

[0162] The fourth output unit is configured to input the plurality of regions of interest into a prediction layer of the initial detection network, and output the initial target detection result of the target detection image.

[0163] According to an embodiment of the present disclosure, the second output module 640 for inputting the pixel depth detection image into the pixel depth detection network and outputting a pixel depth detection result of the pixel depth detection image comprises:

[0164] The fifth output unit is configured to input the pixel depth detection image into a second down-sampling layer of the pixel depth detection network, and output a plurality of scales of third feature maps.

[0165] The sixth output unit is configured to input the plurality of scales of third feature maps into a second fusion feature layer of the pixel depth detection network, and output a plurality of scales of fourth feature maps.

[0166] The seventh output unit is configured to input the plurality of scales of fourth feature maps into a three-dimensional depth perception layer of the pixel depth detection network respectively, and output the pixel depth detection result of the pixel depth detection image.

[0167] According to an embodiment of the present disclosure, the obtaining module 650 for training the initial target detection network and the pixel depth detection network based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain the trained target detection model comprises:

[0168] The first obtaining unit comprises determining a target detection loss value based on the initial target detection result and the target annotation information.

[0169] The second obtaining unit comprises determining a pixel depth detection loss value based on the pixel depth detection result and the pixel depth information.

[0170] The third obtaining unit comprises training the initial target detection network and the pixel depth detection network based on the target detection loss value and the pixel depth detection loss value to obtain the trained target detection model.

[0171] According to an embodiment of the present disclosure, the target annotation information comprises target coordinate information and target category information, and the first obtaining unit for determining the target detection loss value based on the initial target detection result and the target annotation information comprises:

[0172] The first obtaining subunit is configured to determine a target coordinate detection loss value based on the initial target detection result and the target coordinate information.

[0173] The second obtaining subunit is configured to determine a target category detection loss value based on the initial target detection result and the target category information.

[0174] The third obtaining subunit is configured to determine the target detection loss value based on the target coordinate detection loss value and the target category detection loss value.

[0175] According to an embodiment of the present disclosure, the first determining module 630 for determining the pixel depth detection image with the pixel depth information according to the target detection image and the camera parameter information comprises:

[0176] The first determining unit is configured to determine target detection image pixel depth information according to the target annotation information and the camera parameter information to obtain a first sub-pixel depth detection image.

[0177] The second determining unit is configured to determine a plurality of clustering regions of the first sub-pixel depth detection image by using a K-means clustering algorithm according to a preset pixel threshold.

[0178] The third determining unit is configured to determine a second sub-pixel depth detection image from the plurality of clustering regions according to the pixel depth values in the plurality of clustering regions.

[0179] The fourth determining unit is configured to determine the pixel depth detection image based on the first sub-pixel depth detection image and the second sub-pixel depth detection image.

[0180] According to embodiments of this disclosure, the target detection loss value, the target coordinate detection loss value, and the target category detection loss value are respectively expressed by the following formulas:

[0181] L detection =L loc +L cls (1)

[0182]

[0183]

[0184] Among them, L detection L represents the target detection loss value. loc L represents the target coordinate detection loss value. cls represents the target category detection loss value, where x represents the x-axis coordinate of the top-left corner of the target location bounding box in the target coordinate information, y represents the y-axis coordinate of the top-left corner of the target location bounding box in the target coordinate information, w represents the width of the target location bounding box in the target coordinate information, h represents the height of the target location bounding box in the target coordinate information, and t represents the target category detection loss value. i Represents target coordinate information, μ i This represents the target coordinate detection result in the initial target detection result, where T represents the number of target categories in the target detection image, and y... i S represents target category information. j This represents the target category detection result in the initial target detection results.

[0185] Figure 7 A block diagram of a target detection apparatus incorporating camera parameter information according to an embodiment of the present disclosure is illustrated schematically.

[0186] like Figure 7 As shown, the target detection device 700, which combines camera parameter information, includes a third output module 710 and a second determination module 720.

[0187] The third output module 710 is used to input the target image to be detected into the target detection model trained by the target detection model training method that combines camera parameter information as described above, and output the initial target detection result and pixel depth detection result.

[0188] The second determining module 720 is used to determine the target detection result based on the initial target detection result and the pixel depth detection result using a non-maximum suppression method.

[0189] Any of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure, or at least part of functions of any of them, can be implemented in one module. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be implemented at least in part as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging circuits, or in any one of software, hardware, and firmware, or in a proper combination of any of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present disclosure can be implemented at least in part as computer program modules, which can perform corresponding functions when running.

[0190] For example, any of the first obtaining module 610, the first output module 620, the first determining module 630, the second output module 640, and the obtaining module 650 can be combined in one module / unit / sub-unit for implementation, or any of them can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of the modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units, and implemented in one module / unit / sub-unit. According to embodiments of the present disclosure, at least one of the first obtaining module 610, the first output module 620, the first determining module 630, the second output module 640, and the obtaining module 650 can be implemented at least in part as a hardware circuit, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application-specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware by integrating or packaging circuits, or in any one of software, hardware, and firmware, or in a proper combination of any of them. Alternatively, at least one of the first obtaining module 610, the first output module 620, the first determining module 630, the second output module 640, and the obtaining module 650 can be implemented at least in part as computer program modules, which can perform corresponding functions when running.

[0191] It should be noted that the camera parameter combined target detection model training device part in the embodiments of the present disclosure corresponds to the camera parameter combined target detection model training method part in the embodiments of the present disclosure, and the description of the camera parameter combined target detection model training device part is specifically referred to the camera parameter combined target detection model training method part, which will not be repeated here.

[0192] Figure 8 A block diagram of an electronic device suitable for implementing the camera parameter combined target detection model training method described above according to an embodiment of the present disclosure is schematically shown. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0193] As shown in Figure 8 The electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded into a random access memory (RAM) 803 from a storage portion 808. The processor 801 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (such as an application-specific integrated circuit (ASIC)), and so on. The processor 801 can also include an on-board memory for cache use. The processor 801 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to the embodiments of the present disclosure.

[0194] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. The processor 801 performs various operations of the method processes according to the embodiments of the present disclosure by executing the programs in the ROM 802 and / or the RAM 803. It should be noted that the programs can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method processes according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0195] According to an embodiment of the present disclosure, the electronic device 800 can further include an input / output (I / O) interface 805 also connected to the bus 804. The system 800 can further include one or more of the following components connected to the I / O interface 805: an input part 806 including a keyboard, a mouse, etc.; an output part 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 808 including a hard disk, etc.; and a communication part 809 including a network interface card such as a LAN card, a modem, etc. The communication part 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 810 as necessary, so that a computer program read therefrom is installed in the storage part 808 as necessary.

[0196] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0197] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A target detection model training method combining camera parameters, comprising: obtaining a target detection image, wherein the target detection image is an image collected at an inclination angle greater than a preset angle threshold, and the target detection image has target annotation information; inputting the target detection image into an initial target detection network to output an initial target detection result of the target detection image, including: inputting the target detection image into a first down-sampling layer of the initial target detection network to output a plurality of scales of first feature maps; inputting the plurality of scales of first feature maps into a first fusion feature layer of the initial target detection network to output a plurality of scales of second feature maps; inputting the plurality of scales of second feature maps into a region of interest extraction layer of the initial target detection network to output a region of interest of each of the plurality of scales of second feature maps; and inputting the plurality of regions of interest into a prediction layer of the initial target detection network to output the initial target detection result of the target detection image; determining a pixel depth detection image with pixel depth information according to the target detection image and camera parameter information; inputting the pixel depth detection image into a pixel depth detection network to output a pixel depth detection result of the pixel depth detection image, including: inputting the pixel depth detection image into a second down-sampling layer of the pixel depth detection network to output a plurality of scales of third feature maps; inputting the plurality of scales of third feature maps into a second fusion feature layer of the pixel depth detection network to output a plurality of scales of fourth feature maps; and outputting the plurality of scales of fourth feature maps into a three-dimensional depth perception layer of the pixel depth detection network respectively to output the pixel depth detection result of the pixel depth detection image; training the initial target detection network and the pixel depth detection network based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain a trained target detection model.

2. The method of claim 1, wherein, The training of the initial target detection network and the pixel depth detection network based on the target annotation information, the pixel depth information, the initial target detection result and the pixel depth detection result to obtain a trained target detection model, comprises: determining a target detection loss value based on the initial target detection result and the target annotation information; determining a pixel depth detection loss value based on the pixel depth detection result and the pixel depth information; training the initial target detection network and the pixel depth detection network based on the target detection loss value and the pixel depth detection loss value to obtain a trained target detection model.

3. The method of claim 2, wherein, The target annotation information includes target coordinate information and target category information, and the determination of the target detection loss value based on the initial target detection result and the target annotation information comprises: determining a target coordinate detection loss value based on the initial target detection result and the target coordinate information; determining a target category detection loss value based on the initial target detection result and the target category information; The target detection loss value is determined based on the target coordinate detection loss value and the target category detection loss value.

4. The method of claim 1, wherein, The determination of the pixel depth detection image with pixel depth information based on the target detection image and the camera parameter information comprises: The target detection image pixel depth information is determined based on the target annotation information and the camera parameter information, to obtain a first sub-pixel depth detection image. A plurality of clustering regions of the first sub-pixel depth detection image are determined based on a preset pixel threshold and a K-means clustering algorithm. A second sub-pixel depth detection image is determined from the plurality of clustering regions based on pixel depth values in the plurality of clustering regions. The pixel depth detection image is determined based on the first sub-pixel depth detection image and the second sub-pixel depth detection image.

5. The method of claim 3, wherein the target detection loss value, the target coordinate detection loss value, and the target category detection loss value are respectively represented by the following formulas: （1） （2） （3） wherein, represents the target coordinate detection loss value, represents the target coordinate detection loss value, represents the target category detection loss value, x represents the x-axis coordinate value of the upper left corner of the target position frame in the target coordinate information, y represents the y-axis coordinate value of the upper left corner of the target position frame in the target coordinate information, w represents the width value of the target position frame in the target coordinate information, and h represents the height value of the target position frame in the target coordinate information, represents the target coordinate information, represents the target coordinate detection result in the initial target detection result, and T represents the number of target categories in the target detection image, represents the target category information, represents the target category detection result in the initial target detection result.

6. A target detection method combined with camera parameter information, comprising: inputting a target image to be detected into a target detection model trained by the method of any one of claims 1 to 5, and outputting an initial target detection result and a pixel depth detection result; determining a target detection result based on the initial target detection result and the pixel depth detection result by using a non-maximum suppression method.

7. A target detection model training device combined with camera parameter information, applied to the method of any one of claims 1 to 5, comprising: a first acquisition module configured to acquire a target detection image, wherein the target detection image is provided with target annotation information; a first output module configured to input the target detection image into an initial target detection network, and output an initial target detection result of the target detection image; a first determination module configured to determine a pixel depth detection image with pixel depth information based on the target detection image and camera parameter information; a second output module configured to input the pixel depth detection image into a pixel depth detection network, and output a pixel depth detection result of the pixel depth detection image; a obtaining module configured to train the initial target detection network and the pixel depth detection network based on the initial target detection result and the pixel depth detection result, and obtain a trained target detection model.

8. An electronic device, comprising: one or more processors; a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method of any one of claims 1 to 6.

Images