Training method and device of target detection model, target detection method, device and equipment

Abstract

The application provides a target detection model training method, a target detection method, a device and equipment. The target detection model training method comprises: obtaining a sample image, the sample image comprising a sample color image and a sample depth image; inputting the sample color image and the sample depth image into a target detection model to obtain a sample color image mask and a sample depth image mask; calculating a first loss value according to the sample color image mask, the sample depth image mask and a fusion mask; the fusion mask is obtained by fusing a depth image mask obtained according to a preset detection model and a color image mask corresponding to the sample color image; if the first loss value does not satisfy a first preset condition, the target detection model is continuously trained using other sample images until the first loss value satisfies the first preset condition. The application can improve the efficiency of target detection.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image detection, and in particular to a target detection model training method, a target detection method, a device and equipment. BACKGROUND

[0002] Three-dimensional target detection is widely used in safety cruise, intelligent vehicles, unmanned aerial vehicles and other autonomous driving mobile platforms, and is a key technology for realizing intelligent perception of the environment. Three-dimensional target detection needs to handle the near-far changes, occlusion and truncation of targets in the environment, and also needs to meet the high real-time requirements of mobile platforms. The multi-modal fusion method has obvious advantages over the single modal method. Images have rich semantic information, and point clouds have depth information. How to use multi-modal fusion methods to improve detection accuracy is currently a more mainstream approach.

[0003] The data captured by the camera sensor is an RGB image, and the data captured by the laser radar sensor is a LiDAR point cloud. The former belongs to two-dimensional space, and the latter belongs to three-dimensional space. The ultimate goal of three-dimensional target detection is obviously the state and attributes of the target in the three-dimensional space it is in. If the image modal method is used, it will involve a transformation process from two-dimensional space to three-dimensional space. Therefore, the existing target detection efficiency is low. SUMMARY

[0004] The present application provides a target detection model training method, a target detection method, a device and equipment to improve the efficiency of target detection.

[0005] In a first aspect, a target detection model training method is provided. The method comprises:

[0006] Obtaining a sample image, the sample image comprising a sample color image and a sample depth image;

[0007] Inputting the sample color image and the sample depth image into a target detection model to obtain a sample color image mask and a sample depth image mask;

[0008] Calculating a first loss value according to the sample color image mask, the sample depth image mask and a fusion mask; the fusion mask is obtained by fusing a depth image mask obtained by the preset detection model and a color image mask corresponding to the sample color image;

[0009] If the first loss value does not satisfy a first preset condition, the target detection model is continuously trained using other sample images until the first loss value satisfies the first preset condition.

[0010] Preferably, the fusion mask is obtained by fusing a depth image mask obtained by the preset detection model and a color image mask corresponding to the sample color image, comprising:

[0011] inputting the sample depth image into the preset detection model to obtain a depth mask;

[0012] determining whether the sample color image has a corresponding semantic segmentation ground truth label;

[0013] if the sample color image has the corresponding semantic segmentation ground truth label, taking the semantic segmentation ground truth label as a color mask corresponding to the sample color image;

[0014] fusing the depth mask and the color mask to obtain a fused mask.

[0015] Preferably, after determining whether the sample color image has the corresponding semantic segmentation ground truth label, the method further comprises:

[0016] if the sample color image does not have the corresponding semantic segmentation ground truth label, inputting the sample color image into a semantic segmentation model to obtain a semantic segmentation predicted label;

[0017] taking the semantic segmentation predicted label as the color mask corresponding to the sample color image;

[0018] fusing the depth mask and the color mask to obtain the fused mask.

[0019] Preferably, the first loss value is calculated according to the sample color image mask, the sample depth image mask and the fused mask, comprising:

[0020] calculating a first mask loss value between the sample color image mask and the fused mask, and calculating a second mask loss value between the sample depth image mask and the fused mask;

[0021] weighting the first mask loss value and the second mask loss value to obtain the first loss value.

[0022] In a second aspect, a target detection method is provided, comprising:

[0023] collecting a to-be-detected image by a camera device;

[0024] determining a to-be-detected color image and a to-be-detected depth image corresponding to the to-be-detected image;

[0025] inputting the to-be-detected color image and the to-be-detected depth image into a target detection model trained by the training method of the target detection model to obtain a detection result corresponding to the to-be-detected image.

[0026] Preferably, before collecting the to-be-detected image by the camera device, the method further comprises:

[0027] obtaining updated camera device extrinsic parameters according to a pre-trained camera calibration model;

[0028] Correspondingly, the image to be detected is collected by the camera device, including:

[0029] The camera device is reset according to the camera device external parameter, so that the camera device collects the image to be detected based on the updated camera device external parameter.

[0030] Preferably, the training process of the camera calibration model includes:

[0031] The camera device external parameter is obtained according to the camera calibration model;

[0032] The image to be detected is re-determined according to the camera device external parameter;

[0033] The image to be detected and the re-determined image to be detected are input into the target detection model to obtain a sample color image mask and a sample depth image update mask;

[0034] The second loss value is calculated according to the sample color image mask and the sample depth image update mask;

[0035] The camera calibration model is updated according to the second loss value until the second loss value meets the second preset condition, and the training of the camera calibration model is stopped.

[0036] In a third aspect, a training device of a target detection model is provided, and the device includes:

[0037] The acquisition module is configured to acquire sample images, the sample images including sample color images and sample depth images;

[0038] The determination module is configured to input the sample color images and the sample depth images into the target detection model to obtain sample color image masks and sample depth image masks;

[0039] The calculation module is configured to calculate a first loss value according to the sample color image masks, the sample depth image masks, and a fusion mask; the fusion mask is obtained by fusing a depth image mask obtained according to a preset detection model and a color image mask corresponding to the sample color image;

[0040] The training module is configured to continue training the target detection model using other sample images if the first loss value does not meet the first preset condition, until the first loss value meets the first preset condition.

[0041] In a fourth aspect, a target detection device is provided, and the device includes:

[0042] The acquisition module is configured to collect an image to be detected by a camera device;

[0043] The determination module is configured to determine a color image to be detected and a depth image to be detected corresponding to the image to be detected;

[0044] The detection module is configured to input the color image to be detected and the depth image to be detected into the target detection model trained by the training method of the target detection model to obtain a detection result corresponding to the image to be detected.

[0045] In a fifth aspect, an electronic device is provided, including a processor and a memory for storing a computer program, the processor being configured to invoke and run the computer program stored in the memory to execute the method in the first aspect or the implementation manners thereof.

[0046] In a sixth aspect, a computer readable storage medium is provided for storing a computer program, the computer program enabling a computer to execute the method in the first aspect or the implementation manners thereof.

[0047] In a seventh aspect, a computer program product is provided, including computer program instructions, the computer program instructions enabling a computer to execute the method in the first aspect or the implementation manners thereof.

[0048] In an eighth aspect, a computer program is provided, the computer program enabling a computer to execute the method in the first aspect or the implementation manners thereof.

[0049] According to the technical solution provided in the present application, first, a sample image is acquired, and then a sample color image and a sample depth image in the sample image are input into a target detection model to obtain a sample color image mask and a sample depth image mask; a first loss value is calculated according to the sample color image mask, the sample depth image mask and a fusion mask; if the first loss value does not satisfy a first preset condition, the target detection model is continuously trained using other sample images until the first loss value satisfies the first preset condition. The fusion mask is obtained by fusing a depth image mask obtained according to a preset detection model and a color image mask corresponding to the sample color image. Since the color image and the depth image do not need to be mapped to the same space during the training of the target detection model, the target detection model after the training does not need to map the color image and the depth image to the same space when identifying images, so that the target detection model trained according to the present application can directly detect the corresponding result, and the efficiency of target detection can be improved by using the target detection model in the present application. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0051] Figure 1The flow of the training method of the target detection model provided by the embodiment of the present application Figure 1 ;

[0052] Figure 2 The flow of the training method of the target detection model provided by the embodiment of the present application Figure 2 ;

[0053] Figure 3 The flow of the determination method of the fusion mask provided by the embodiment of the present application Figure 1 ;

[0054] Figure 4 The flow of the determination method of the fusion mask provided by the embodiment of the present application Figure 2 ;

[0055] Figure 5 The flow chart of the target detection method provided by the embodiment of the present application

[0056] Figure 6 The flow chart of the training method of the camera calibration model provided by the embodiment of the present application

[0057] Figure 7 The schematic diagram of the training device of the target detection model provided by the embodiment of the present application

[0058] Figure 8 The schematic diagram of the target detection device provided by the embodiment of the present application

[0059] Figure 9 The schematic block diagram of the electronic device provided by the embodiment of the present application DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0062] Figure 1 and Figure 2 A flowchart of a training method of a target detection model provided by an embodiment of the present application, which can be executed by an electronic device, but is not limited thereto. The method can include the following steps:

[0063] S110: Obtain a sample color image, which includes a sample color image and a sample depth image.

[0064] The embodiment can obtain a sample color image through a multi-camera and obtain laser point cloud data through a laser radar, and then determine a sample depth image based on the sample color image and the laser point cloud data.

[0065] The sample color image and the laser point cloud data can be obtained by existing means, such as in the data collection stage of the autonomous driving scene, various types of sensor data will be collected, including the RGB color image (i.e. the sample color image) collected by the multi-camera, the synchronized LiDAR point cloud data, and various types of conversion parameters. Specifically, the depth map [h, w] corresponding to the RGB color image can be rendered from the laser point cloud data, and further rendered into the RGB depth image [h, w, d] corresponding to the RGB color image through color mapping. The RGB depth image (i.e. the sample depth image) has rich geometric information and semantic information. Illustratively, the sample color image can be a road image containing road surface condition information collected by a multi-camera during the autonomous driving process of a car. The sample color image can include human, building, vehicle and other target objects.

[0066] S120: Input the sample color image and the sample depth image into the target detection model to obtain a sample color image mask and a sample depth image mask.

[0067] The target detection model can be a SAM Learn (Segment Anything Model) learning model. Illustratively, as shown inFigure 2 As shown, in this embodiment, the sample color image and sample depth image are input into the SAM Learn model to obtain the sample color image mask and the sample depth image mask. It can be understood that the SAM Learn model architecture can include three parts: an image encoder; a cue encoder; and a mask decoder. The SAM Learn model can effectively map the input sample color image and sample depth image to masks through embedding, cue embedding, and output tokens, thereby obtaining the sample color image mask and the sample depth image mask. The image mask is generally used to occlude (fully or partially) the image being processed to control the area or process of image processing, thereby improving the accuracy of segmentation.

[0068] It should be noted that the SAM Learn model in computer vision is used to find and segment any object in an image. It has numerous derivative applications, such as general segmentation in open scenes, non-semantic edge detection, interactive segmentation, and so on. It should be pointed out that... Figure 2 The SAM Learn model in this context refers to the general SAM model and does not specifically refer to any particular model within it.

[0069] S130: Calculate the first loss value based on the sample color image mask, the sample depth image mask, and the fusion mask.

[0070] The fusion mask is obtained by fusing the depth map mask obtained from the pre-set detection model and the color map mask corresponding to the sample color image. For example... Figure 2 As shown, the preset detection model in this embodiment corresponds to Figure 2 The SAM Fix model in this paper employs a pre-trained model from existing technologies. For example, the SAM Fix architecture can include three parts: an image encoder; a cue encoder; and a mask decoder. Inputting a sample depth image into the SAM Fix model yields the corresponding depth map mask. Then, the depth map mask and color map mask corresponding to the sample depth image are processed. Figure 2 The semantic segmentation GT (or pseudo GT) in the image is fused to obtain a fused mask. Specifically, in this embodiment, the depth map mask and the color map mask can be fused using processing rules such as clustering, voting, and filtering to obtain a fused mask.

[0071] This embodiment fuses depth map masks and color map masks, making the training of the target detection model clearer and more accurate, and more suitable for subsequent detection processing of the image to be detected.

[0072] In an optional embodiment provided in this application, the step S130 of calculating the first loss value based on the sample color image mask, the sample depth image mask, and the fusion mask may include:

[0073] S1301: Calculate a first mask loss value between the sample color image mask and the fusion mask, and calculate a second mask loss value between the sample depth image mask and the fusion mask.

[0074] S1302: Calculate a first loss value by weighting the first mask loss value and the second mask loss value.

[0075] The weight values corresponding to the first mask loss value and the second mask loss value can be set according to actual requirements, for example, the weight value of the first mask loss value can be 0.6, and the weight value of the second mask loss value can be 0.4.

[0076] S140: If the first loss value does not satisfy the first preset condition, continue to use other sample images to train the target detection model until the first loss value satisfies the first preset condition.

[0077] In this embodiment, the sample color image and the sample depth image are input into the target detection model (SAM Learn model) to obtain a sample color image mask and a sample depth image mask, and then the loss values of the sample color image mask and the sample depth image mask with the fusion mask are calculated. Then, the Dice loss function commonly used in segmentation tasks is used to update the parameters of the SAM Learn model through back propagation. The first preset condition can be a specific numerical value set in advance. When the first loss value is less than the numerical value, the training of the SAM Learn model is stopped. If the first loss value is greater than or equal to the numerical value, the SAM Learn model is continuously trained using other sample images until the first loss value is less than the numerical value, and the training of the SAM Learn model is completed.

[0078] The training method of the target detection model provided in the embodiment of the application first acquires sample images, and then inputs the sample color image and the sample depth image in the sample images into the target detection model to obtain a sample color image mask and a sample depth image mask. The first loss value is calculated according to the sample color image mask, the sample depth image mask and the fusion mask. If the first loss value does not satisfy the first preset condition, the target detection model is continuously trained using other sample images until the first loss value satisfies the first preset condition. The fusion mask is obtained by fusing the depth image mask obtained according to the preset detection model and the color image mask corresponding to the sample color image. Since the color image and the depth image do not need to be mapped to the same space in the training process of the target detection model, the target detection model after training does not need to map the color image and the depth image to the same space when identifying the image. Therefore, the target detection model trained by the application can directly detect the corresponding result, and the efficiency of target detection can be improved by using the target detection model in the application.

[0079] Figure 3 and Figure 4 A flow chart of a method for determining a fusion mask is shown, as Figure 3 and Figure 4 The method can include the following steps:

[0080] S310: inputting a sample depth image into a preset detection model to obtain a depth map mask.

[0081] The preset detection model corresponds to a SAM Fix model in Figure 2 The sample depth image is inputted into the SAM Fix model to obtain a corresponding depth map mask.

[0082] S320: determining whether a sample color image has a corresponding semantic segmentation actual label.

[0083] In this embodiment, if the sample color image has a corresponding semantic segmentation actual label, the method proceeds to step S330 for further execution; if the sample color image does not have a corresponding semantic segmentation actual label, the method proceeds to step S350 for further execution.

[0084] The semantic segmentation actual label is a label that has been annotated, and the content of the label is the annotation result of each target in the sample color image. For example, in the field of vehicle driving, the semantic segmentation actual label corresponding to the sample color image obtained by the vehicle-mounted camera can be people, vehicles, trees, etc., i.e., the classification result of each target in the color image.

[0085] S330: if the corresponding semantic segmentation actual label exists, the semantic segmentation actual label is taken as the color map mask corresponding to the sample color image.

[0086] S340: fusing the depth map mask and the color map mask to obtain a fusion mask.

[0087] As shown in Figure 4 if the corresponding semantic segmentation actual label (GT) exists, the semantic segmentation actual label is taken as the color map mask corresponding to the sample color image, and then the depth map mask and the color map mask (GT) are fused to obtain a fusion mask.

[0088] S350: if the corresponding semantic segmentation actual label does not exist, the sample color image is inputted into a semantic segmentation model to obtain a semantic segmentation prediction label.

[0089] The semantic segmentation model in this embodiment can be any segmentation model that can realize pixel-level classification of an image in the prior art. If the sample color image does not have a pre-labeled semantic segmentation actual label, the semantic segmentation model can be used to segment the sample color image to obtain a semantic segmentation result corresponding to the sample color image, and the semantic segmentation result is used as a semantic segmentation prediction label for subsequent processing.

[0090] As shown in FIG. 6, Figure 4 In another embodiment, before the sample color image is input into the semantic segmentation model, a closed-type semantic segmentation model or an open-type semantic segmentation model can be selected according to the openness of the segmentation category, and then the sample color image is segmented based on the determined semantic segmentation model, and the prediction result output by the segmentation model is used as a semantic segmentation prediction label.

[0091] Further, if the sample color image is segmented by the closed-type semantic segmentation model, the segmentation result obtained is a label in the closed set; if the sample color image is segmented by the open-type semantic segmentation model, the segmentation result obtained can not be a label in the closed set.

[0092] It can be understood that the openness of the segmentation category can be pre-set according to different standards. For example, the closed-type semantic segmentation model is mainly trained with predefined categories, and the prediction result cannot be generalized to unknown categories, so the segmentation result obtained is a label in the closed set. On the contrary, if the scene is understood in an open vocabulary way, there are thousands of categories in the prediction result.

[0093] S360: The semantic segmentation prediction label is used as a color image mask corresponding to the sample color image.

[0094] S370: The depth image mask and the color image mask are fused to obtain a fusion mask.

[0095] The determination method of the fusion mask provided in this embodiment is based on the depth image mask obtained by the preset detection model, the color image mask obtained by the semantic segmentation model, and the fusion of the depth image mask and the color image mask to obtain the fusion mask. In this way, the fusion mask contains both depth image features and color image features, so that the target detection model can be better trained based on the fusion mask, thereby ensuring the accuracy of the target detection model training.

[0096] As shown in FIG. 6, Figure 5 Figure 5 A flowchart of a target detection method provided in this embodiment, which can include the following steps:

[0097] S510: An image to be detected is collected by a camera device. ​

[0098] In the embodiment, before the image to be detected is collected by the camera device, the method further includes: obtaining the updated camera device external parameter according to the camera calibration model; correspondingly, collecting the image to be detected by the camera device includes: resetting the camera device according to the updated camera device external parameter, so that the camera device collects the image to be detected based on the reset camera device external parameter.

[0099] The updated camera device external parameter refers to the updated camera device external parameter obtained according to the camera calibration model. It can be understood that the camera device internal parameter is a parameter related to the characteristics of the camera device, such as the focal length, pixel size, etc. of the camera device; and the camera device external parameter is a parameter of the camera device in the world coordinate system, such as the position and rotation direction of the camera device.

[0100] Specifically, the image to be detected (including the captured RGB image and the point cloud data captured by the laser radar) is collected by the camera device, and the camera device external parameter RT (i.e. the pose R, T of the camera device) is also collected. Without loss of generality, the camera calibration model here can be composed of an Encoder-Decoder structure. The input camera device external parameter RT is sent into the Encoder-Decoder structure by using a high-dimensional embedding coding method, and the camera device external parameter RT that can be directly used is output. As known from the foregoing, when the external parameter and the laser point cloud are given, the RGB depth map can be rendered.

[0101] In actual scenarios, the camera device external parameter and the internal parameter often produce offset errors due to calibration errors, sensor synchronization, ego motion, etc. In most cases, manually adjusting the camera device external parameter is time-consuming and laborious. Inaccurate camera device external parameters mean that the mapping relationship between the point cloud and the image has different degrees of deviation, which will affect the accuracy of some detection methods that use the camera device external parameter projection process. Figure 6 As shown in the figure, to solve this problem, another embodiment of the present application provides a camera calibration model training method, which can realize automatic calibration of the updated camera device external parameter through the trained camera calibration model. The training method includes:

[0102] S610: Obtain the updated camera device external parameter according to the camera calibration model.

[0103] As shown in the figure, the updated camera device external parameter is obtained through the camera calibration model (CalibNet external parameter RT correction network). Figure 6

[0104] S620: Redetermine the image to be detected according to the updated camera device external parameter.

[0105] ​In this embodiment, after obtaining the updated external parameters of the camera device, the depth image to be detected can be re-determined based on the updated external parameters of the camera device, the laser point cloud, and the color image to be detected.

[0106] In this embodiment, the CalibNet model (a geometrically supervised deep network) is used to automatically estimate the six-degree-of-freedom rigid body transformation between the lidar and camera equipment in real time. This reduces the need for calibration targets, thereby significantly saving calibration workload.

[0107] S630: Input the color image to be detected and the redefined depth image to be detected into the target detection model to obtain the sample color image mask and the sample depth image update mask.

[0108] Specifically, the color image to be detected and the redefined depth image to be detected are fed into the object detection model (through...). Figure 1 The process uses a pre-trained SAM model to obtain sample color image masks (segmentation mask 1) and sample depth image update masks (segmentation mask 2). Using segmentation mask 1 as the ground truth, segmentation mask 2 and its Dice loss function are calculated, gradients are backpropagated for optimization, and the model parameters of the object detection model are updated online.

[0109] S640: Calculate the second loss value based on the sample color image mask and the sample depth image updated mask.

[0110] S650: Update the camera calibration model based on the second loss value until the second loss value meets the second preset condition, then stop training the camera calibration model.

[0111] Wherein, the second loss value corresponds to Figure 6 In the Loss1 parameter, after obtaining Loss1, it is determined whether Loss1 is less than a preset value. If it is less than the preset value, training of the camera calibration model is stopped; if it is greater than or equal to the preset value, the updated camera device extrinsic parameters of the camera calibration model are obtained again, and then the process is repeated. Figure 6 The calculation of Loss1 continues until Loss1 is less than a preset value, thus completing the training of the camera calibration model.

[0112] S520: Determine the color image and depth image to be detected corresponding to the image to be detected.

[0113] S530: Input the color image to be detected and the depth image to be detected into the target detection model to obtain the detection result corresponding to the image to be detected.

[0114] The detection result corresponding to the to-be-detected image is a process of detecting a target object in a 3D space from data (such as point cloud data, a depth image, a color image, and various sensor data) and classifying and positioning the target object. The attributes of the target object include a center point coordinate (cx, cy, cz), a size (w, l, h), an orientation angle, and the like, which are not specifically limited in this embodiment.

[0115] The target detection method provided in this embodiment first acquires updated camera external parameters according to the camera calibration model, collects a to-be-detected image through the camera, then determines a to-be-detected color image and a to-be-detected depth image corresponding to the to-be-detected image, inputs the to-be-detected color image and the to-be-detected depth image into the trained target detection model, and obtains a detection result corresponding to the to-be-detected image. Since the camera calibration model in this embodiment can automatically confirm the camera external parameters, the camera set according to the camera external parameters can successfully map the color image and the point cloud data into the depth image, so as to input the color image and the depth image into the target detection model to obtain the corresponding detection result, thereby improving the accuracy of target detection through this embodiment.

[0116] In one application scenario provided in this application, first, the updated camera external parameters are acquired through the camera calibration model (CalibNet external parameter RT correction network), then the camera is configured based on the camera external parameters, so that the camera acquires the color image captured by the multi-view (generally 4 or 6 cameras) sensor and the point cloud data scanned by the laser radar at a moment in the vehicle driving process, and then the corresponding depth image is determined based on the color image and the point cloud data. Finally, the color image and the depth image are jointly input into the target detection model to obtain the corresponding detection result.

[0117] It should be noted that the frequency of executing the camera calibration model update is lower than that of the target detection model, and a specified time interval can be set for online update, or the motion conditions and environmental conditions triggering the online update parameters can be set according to the motion state and environmental perception analysis of the ego vehicle.

[0118] Figure 7 A schematic diagram of a training device 700 of a target detection model provided in this embodiment is shown in FIG. 7. Figure 7 As shown in the figure, the device 700 includes:

[0119] The acquisition module 701 is configured to acquire a sample image, and the sample image includes a sample color image and a sample depth image.

[0120] The determination module 702 is configured to input the sample color image and the sample depth image into the target detection model to obtain a sample color image mask and a sample depth image mask.

[0121] The computing module 703 is configured to calculate a first loss value according to the sample color image mask, the sample depth image mask, and the fusion mask; the fusion mask is obtained by fusing a depth map mask obtained by the preset detection model and a color map mask corresponding to the sample color image.

[0122] The training module 704 is configured to, if the first loss value does not satisfy the first preset condition, continue to train the target detection model using other sample images until the first loss value satisfies the first preset condition.

[0123] In some possible implementation manners, the apparatus further includes a fusion module 705, which is specifically configured to: input the sample depth image into the preset detection model to obtain a depth map mask; determine whether the sample color image has a corresponding semantic segmentation actual label; if the sample color image has the corresponding semantic segmentation actual label, take the semantic segmentation actual label as a color map mask corresponding to the sample color image; and fuse the depth map mask and the color map mask to obtain a fusion mask.

[0124] In some possible implementation manners, the fusion module 705 is specifically configured to: if the sample color image does not have the corresponding semantic segmentation actual label, input the sample color image into a semantic segmentation model to obtain a semantic segmentation predicted label; take the semantic segmentation predicted label as a color map mask corresponding to the sample color image; and fuse the depth map mask and the color map mask to obtain the fusion mask.

[0125] In some possible implementation manners, the computing module 703 is specifically configured to: calculate a first mask loss value between the sample color image mask and the fusion mask, and calculate a second mask loss value between the sample depth image mask and the fusion mask; and calculate the first loss value by weighting the first mask loss value and the second mask loss value.

[0126] It should be understood that the apparatus embodiment and the training method embodiment of the target detection model can correspond to each other, and similar descriptions can be referred to the training method embodiment of the target detection model. Specifically, Figure 7 The apparatus 700 shown can perform the training method embodiments of the target detection model described above, and the foregoing and other operations and / or functions of each module in the apparatus 700 are respectively used to implement the corresponding processes in the training method embodiments of the target detection model described above. For brevity, details are not described herein.

[0127] Figure 8 A schematic diagram of a target detection apparatus 800 provided by an embodiment of the present application is shown in FIG. 8. Figure 8 As shown in the figure, the apparatus 800 includes:

[0128] The acquisition module 801 is configured to acquire a to-be-detected image by using a camera device.

[0129] The determining module 802 is configured to determine a to-be-detected color image and a to-be-detected depth image corresponding to a to-be-detected image.

[0130] The detecting module 803 is configured to input the to-be-detected color image and the to-be-detected depth image into the target detection model to obtain a detection result corresponding to the to-be-detected image.

[0131] In some implementable manners, the apparatus further includes an obtaining module 804, which is specifically configured to obtain the updated camera external parameter according to the camera calibration model.

[0132] The collecting module 801 is specifically configured to reset the camera device according to the updated camera external parameter, so that the camera device collects the to-be-detected image based on the updated camera external parameter.

[0133] In some implementable manners, the apparatus further includes a training module 805, which is configured to obtain the updated camera external parameter according to the camera calibration model; re-determine the to-be-detected depth image according to the camera external parameter; input the to-be-detected color image and the re-determined to-be-detected depth image into the target detection model to obtain a sample color image mask and a sample depth image update mask; calculate a second loss value according to the sample color image mask and the sample depth image update mask; update the camera calibration model according to the second loss value, until the second loss value meets a second preset condition, and stop training of the camera calibration model.

[0134] It should be understood that the apparatus embodiment and the target detection method embodiment can correspond to each other, and similar descriptions can be referred to the target detection method embodiment. Specifically, Figure 8 The apparatus 800 shown can perform the target detection method embodiments described above, and the foregoing and other operations and / or functions of each module in the apparatus 800 are respectively for realizing the corresponding processes in the target detection method described above. For the sake of brevity, they will not be described here again.

[0135] Figure 9 is a schematic block diagram of an electronic device 900 provided by an embodiment of the present application. As shown in Figure 9 The electronic device 900 can include a processor 901 and a memory 902. The electronic device 900 can also include one or more of a multimedia component 903, an input / output (I / O) interface 904, and a communication component 905.

[0136] The processor 901 is configured to control overall operations of the electronic device 900 to complete all or part of the steps of the above-described methods. The memory 902 is configured to store various types of data to support operations of the electronic device 900, which can include, for example, instructions for operating any application or method on the electronic device 900, and application-related data, such as contact data, transmitted and received messages, pictures, audio, video, and the like. The memory 902 can be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk. The multimedia component 903 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 902 or transmitted through the communication component 905. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 904 provides an interface between the processor 901 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 905 is configured to perform wired or wireless communication between the electronic device 900 and other devices. The wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, and the like, or a combination of one or more of them, is not limited herein. Therefore, the communication component 905 can include, for example, a Wi-Fi module, a Bluetooth module, an NFC module, and the like.

[0137] In an exemplary embodiment, the electronic device 900 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the above-described methods.

[0138] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described method. For example, the computer-readable storage medium can be the above-described memory 902 including program instructions, which can be executed by the processor 901 of the electronic device 900 to complete the above-described method.

[0139] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described method.

[0140] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a programmable device, and the computer program has code portions for executing the above-described method when executed by the programmable device.

[0141] In another exemplary embodiment, a computer program is also provided, which causes a computer to execute the above-described method.

[0142] Those of ordinary skill in the art can realize that the modules and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0143] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiments is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple devices or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0144] The modules described as separated parts can or can not be physically separated, and the parts shown as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. For example, the functional modules in the embodiments of the present application can be integrated in one processing module, or can be physically separated, or two or more modules can be integrated in one module.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a target detection model, the method comprising: The method comprises: acquiring sample images, the sample images comprising sample color images and sample depth images; the sample depth images being obtained based on the sample color images and laser point cloud data; inputting the sample color images and the sample depth images into a target detection model to obtain sample color image masks and sample depth image masks; calculating a first loss value according to the sample color image masks, the sample depth image masks and a fusion mask; the fusion mask being fused according to a depth image mask obtained by a preset detection model and a color image mask corresponding to the sample color images; the preset detection model and the target detection model being different models; if the first loss value does not satisfy a first preset condition, continuing to train the target detection model using other sample images until the first loss value satisfies the first preset condition; wherein the fusion mask is fused according to a depth image mask obtained by a preset detection model and a color image mask corresponding to the sample color images, comprising: inputting the sample depth images into the preset detection model to obtain a depth image mask; determining whether the sample color images exist corresponding semantic segmentation actual labels; if the corresponding semantic segmentation actual labels exist, taking the semantic segmentation actual labels as the color image masks corresponding to the sample color images; fusing the depth image mask and the color image mask to obtain the fusion mask.

2. The method of claim 1, wherein, After the determination of whether the sample color images exist corresponding semantic segmentation actual labels, the method further comprises: if the corresponding semantic segmentation actual labels do not exist, inputting the sample color images into a semantic segmentation model to obtain semantic segmentation predicted labels; taking the semantic segmentation predicted labels as the color image masks corresponding to the sample color images; fusing the depth image mask and the color image mask to obtain the fusion mask.

3. The method of claim 1, wherein, The calculation of the first loss value according to the sample color image masks, the sample depth image masks and the fusion mask comprises: calculating a first mask loss value between the sample color image masks and the fusion mask, and calculating a second mask loss value between the sample depth image masks and the fusion mask; weighting the first mask loss value and the second mask loss value to obtain the first loss value.

4. A target detection method characterized by, The method comprises: acquiring a to-be-detected image by a camera device; determining a to-be-detected color image and a to-be-detected depth image corresponding to the to-be-detected image; inputting the to-be-detected color image and the to-be-detected depth image into the target detection model of any one of claims 1-3 to obtain a detection result corresponding to the to-be-detected image.

5. The method of claim 4, wherein, Before the acquisition of the to-be-detected image by the camera device, the method further comprises: acquiring updated camera device external parameters according to a pre-trained camera calibration model; The acquisition of the to-be-detected image by the camera device comprises: resetting the camera device according to the camera device external parameters, so that the camera device acquires the to-be-detected image based on the updated camera device external parameters.

6. The method of claim 5, wherein, The training process of the camera calibration model comprises: The external parameters of the camera device are obtained based on the camera calibration model; The depth image to be detected is re-determined based on the external parameters of the camera device; The color image to be detected and the redefined depth image to be detected are input into the target detection model to obtain the sample color image mask and the sample depth image update mask; Calculate the second loss value based on the sample color image mask and the sample depth image update mask; The camera calibration model is updated based on the second loss value until the second loss value meets the second preset condition, at which point the training of the camera calibration model is stopped. 7.A device for training a target detection model, characterized in that, The device includes: The acquisition module is used to acquire sample images, which include sample color images and sample depth images; the sample depth images are obtained based on the sample color images and laser point cloud data. The determination module is used to input the sample color image and the sample depth image into the target detection model to obtain the sample color image mask and the sample depth image mask; The calculation module is used to calculate a first loss value based on the sample color image mask, the sample depth image mask, and the fusion mask; the fusion mask is obtained by fusing the depth map mask obtained by the preset detection model and the color map mask corresponding to the sample color image; the preset detection model and the target detection model are different models; The training module is configured to continue training the target detection model using other sample images if the first loss value does not meet the first preset condition, until the first loss value meets the first preset condition. The device further includes a fusion module, which is used for: The sample depth image is input into the preset detection model to obtain a depth map mask; Determine whether the sample color image has a corresponding semantic segmentation actual label; If a corresponding semantic segmentation actual label exists, then the semantic segmentation actual label is used as the color image mask corresponding to the sample color image; The depth map mask and the color map mask are fused to obtain the fused mask.

8. A target detection apparatus characterized by comprising: The device includes: The acquisition module is used to acquire images to be detected through a camera device; The determination module is used to determine the color image to be detected and the depth image to be detected corresponding to the image to be detected; The detection module is used to input the color image to be detected and the depth image to be detected into the target detection model according to any one of claims 1-3, and obtain the detection result corresponding to the image to be detected.

9. An electronic device, comprising: include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1-6.

Images