Target detection model training method, target detection method, device and equipment

Abstract

This application provides a training method, object detection method, apparatus, and device for an object detection model. The method includes: acquiring a set of sample images using a camera device; determining the visually centered bird's-eye view (BEV) fusion features and semantic segmentation features corresponding to the set of sample images; inputting the BEV fusion features and semantic segmentation features into an object detection model to obtain an object prediction result; calculating a first loss value based on the object prediction result and the actual detection result corresponding to the set of sample images; if the first loss value does not meet a preset condition, continuing to train the object detection model using other set of sample images until the first loss value meets the preset condition. This application can be used to improve the efficiency of object detection.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to a training method for an object detection model, an object detection method, an apparatus, and a device. Background Technology

[0002] 3D object detection is widely used in autonomous driving mobile platforms such as safe cruise control, intelligent vehicles, and drones, and is a key technology for achieving intelligent environmental perception. 3D object detection needs to handle changes in target distance and occlusion in the environment, and also needs to meet the high real-time requirements of mobile platforms. Multimodal fusion methods have significant advantages over single-modal methods. Since images contain rich semantic information and point clouds contain depth information, utilizing multimodal fusion methods to improve detection accuracy is currently a mainstream approach.

[0003] The data captured by the camera is an RGB image, while the data captured by the LiDAR is a LiDAR point cloud. The former belongs to two-dimensional space, and the latter belongs to three-dimensional space. The ultimate goal of 3D object detection is obviously to obtain the state and attributes of the object in its three-dimensional space. If an image modality method is used, it will involve a transformation process from two-dimensional space to three-dimensional space. Therefore, the efficiency of current object detection methods is relatively low. Summary of the Invention

[0004] This application provides a training method for an object detection model, an object detection method, an apparatus, and a device to improve the efficiency of object detection.

[0005] Firstly, a training method for an object detection model is provided, comprising: acquiring a set of sample images through a camera device; determining the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to the set of sample images; inputting the BEV fusion features and semantic segmentation features into the object detection model to obtain the object prediction result; calculating a first loss value based on the object prediction result and the actual detection result corresponding to the set of sample images; if the first loss value does not meet the preset conditions, then continuing to train the object detection model using other set of sample images until the first loss value meets the preset conditions.

[0006] Preferably, determining the BEV fusion features corresponding to the sample image group includes: extracting image data features corresponding to each sample image in the sample image group through the image feature extractor in the target detection model; the sample image group includes sample images corresponding to multiple viewpoints; and determining the BEV fusion features corresponding to the sample image group based on the image data features corresponding to the sample image group and the image data features corresponding to the sample image group acquired at two times prior to the sample image group.

[0007] Preferably, determining the semantic segmentation features corresponding to the sample image group includes: inputting the sample image group into the Open Vocabulary Segmentation (OVSeg) model to obtain the target semantic center point and target feature vector corresponding to the sample image group; and determining the target semantic center point and target feature vector as the semantic segmentation features corresponding to the sample image group.

[0008] Preferably, the method further includes: acquiring the intrinsic and extrinsic parameters of the camera device; determining the 3D grid point projection reference point based on the intrinsic and extrinsic parameters; and inputting the BEV fusion feature, semantic segmentation feature, and 3D grid point projection reference point into the target detection model to obtain the target prediction result.

[0009] Preferably, obtaining the internal and external parameters of the camera device includes: obtaining the external parameters of the camera device based on the camera calibration model.

[0010] Preferably, the training process of the camera calibration model includes: obtaining updated camera device extrinsic parameters based on the camera calibration model; redetermining the depth image based on the updated camera device extrinsic parameters; inputting the color image and the redetermined depth image into the target detection model to obtain a sample color image mask and a sample depth image update mask; calculating a second loss value based on the sample color image mask and the sample depth image update mask; updating the camera calibration model based on the second loss value until the second loss value meets a second preset condition, at which point the training of the camera calibration model is stopped.

[0011] Secondly, a target detection method is provided, comprising: acquiring a group of images to be identified through a camera device; inputting the group of images to be identified into a target detection model trained by the above-mentioned target detection model training method, and obtaining the target prediction result corresponding to the group of images to be identified.

[0012] Thirdly, a training device for an object detection model is provided, comprising:

[0013] The acquisition module is used to acquire sample image groups through a camera device;

[0014] The determination module is used to determine the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to the sample image group;

[0015] The prediction module is used to input BEV fusion features and semantic segmentation features into the target detection model to obtain target prediction results;

[0016] The calculation module is used to calculate the first loss value based on the target prediction results and actual detection results corresponding to the sample image group;

[0017] The training module is used to continue training the object detection model with other sample image groups if the first loss value does not meet the preset conditions, until the first loss value meets the preset conditions.

[0018] Fourthly, a target detection device is provided, comprising: an acquisition module for acquiring a group of images to be identified through a camera device; and a detection module for inputting the group of images to be identified into a target detection model trained by the above-mentioned target detection model training method to obtain a target prediction result corresponding to the group of images to be identified.

[0019] Fifthly, an electronic device is provided, comprising: a processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory, and performing the methods as described in the first aspect or its various implementations.

[0020] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0021] In a seventh aspect, a computer program product is provided, including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0022] Eighthly, a computer program is provided that causes a computer to perform the methods described in the first aspect or its various implementations.

[0023] The technical solution provided in this application first acquires a group of sample images using a camera device. Then, it determines the visual-centric bird's-eye view BEV fusion features and semantic segmentation features corresponding to each group of sample images. These features are then input into the object detection model to obtain the object prediction result. A first loss value is calculated based on the object prediction result and the actual detection result corresponding to the sample image group. If the first loss value does not meet the preset conditions, other sample image groups are used to train the object detection model until the first loss value meets the preset conditions. Since this application uses BEV fusion features and semantic segmentation features in the training process of the object detection model, it leverages the advantages of BEV fusion features in visual tasks and adds semantic segmentation features to train the object detection model. The semantic segmentation features enhance the object detection model's ability to represent BEV fusion features, thereby improving the accuracy of object detection model training and ultimately increasing the efficiency of object detection. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a training method for an object detection model provided in an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of the BEV fusion feature acquisition process provided in an embodiment of this application;

[0027] Figure 3 The flowchart of a training method for an object detection model provided in this application embodiment Figure 1 ;

[0028] Figure 4 The flowchart of a training method for an object detection model provided in this application embodiment Figure 2 ;

[0029] Figure 5 The flowchart of the training method for the camera calibration model provided in the embodiments of this application Figure 1 ;

[0030] Figure 6 The flowchart of the training method for the camera calibration model provided in the embodiments of this application Figure 2 ;

[0031] Figure 7 A schematic diagram of a training device for an object detection model provided in an embodiment of this application;

[0032] Figure 8 A schematic diagram of a target detection device provided in an embodiment of this application;

[0033] Figure 9 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0036] Figure 1 The flowchart illustrates a training method for an object detection model provided in this application embodiment. This method may include the following steps:

[0037] S110: Acquire sample image groups through camera equipment.

[0038] The sample image set includes sample images from multiple perspectives. For example, in the field of autonomous driving, the sample image set obtained by the multi-camera device installed on the vehicle includes 6 sample images, namely the front view image, the rear view image, the left front view image, the left rear view image, the right front view image, and the right rear view image.

[0039] S120: Determine the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to the sample image groups respectively.

[0040] Among them, the BEV (Bird's-Eye View) perspective can provide a 360-degree all-around view of the vehicle's surroundings and can easily represent the height and three-dimensional spatial information of objects in the scene, which is beneficial for the vehicle to accurately perceive the scene in complex environments.

[0041] In this embodiment, it is assumed that the current input is a set of sample images acquired by six cameras (containing six RGB images from different viewpoints). The current sample image set is then fed into an image feature extractor to obtain the image features of each image. Spatiotemporal fusion is then performed to obtain the BEV fusion features for the current moment. Without loss of generality, the image feature extractor can use common backbone networks such as ResNet, EfficientNet, and SwingT. Spatiotemporal fusion can employ mainstream algorithm models such as BEVDet4D and BEVFormer. The BEV decoder can use a deformable transformer decoder-like scheme.

[0042] like Figure 2 As shown, in an optional embodiment provided in this application, acquiring the BEV fusion features corresponding to the sample image group through a camera device may include: extracting the image data features corresponding to each sample image in the sample image group through the image feature extractor in the target detection model; the sample image group includes sample images corresponding to multiple viewpoints; and then determining the BEV fusion features corresponding to the sample image group based on the image data features corresponding to the sample image group and the image data features corresponding to the sample image group at two moments before the acquisition time of the sample image group.

[0043] For example, if the current time is t, the image data features corresponding to the sample image groups obtained at time t-1 and t-2 are extracted from the data storage. Then, spatiotemporal fusion is performed based on the image data features corresponding to the three sample image groups to obtain the BEV fusion features corresponding to the sample image groups.

[0044] In one optional embodiment provided in this application, acquiring the semantic segmentation features corresponding to each sample image group through a camera device may include: inputting the sample image group into an Open Vocabulary Segmentation (OVSeg) model to obtain the target semantic center point and target feature vector corresponding to the sample image group; and then determining the target semantic center point and target feature vector as the semantic segmentation features corresponding to the sample image group. The OVSeg model can understand and segment images using any user-defined vocabulary, and allows the SAM (Segment Anything Model) model to know the category to be segmented, thus completing fine-grained open language segmentation.

[0045] It should be noted that the large-scale computer vision model SAM is used to find and segment any object in an image, and it has numerous derivative applications, such as general segmentation of open scenes, non-semantic edge detection, interactive segmentation, and so on. This application adds the existing SAM-derived OVSeg model to perform open-category semantic segmentation of images to obtain the target semantic center point and target feature vector corresponding to the sample image group. Specifically, a height set can be predefined as prior knowledge, and each class in the target category corresponds to a discrete height set (containing three elements, representing the sampled values ​​in the height direction). For the category clustering of the OVSeg segmentation results, the target category and its height set are matched. Combining the extracted information with the predefined height, the target semantic center point and target feature vector are calculated.

[0046] S130: Input the BEV fusion features and semantic segmentation features into the target detection model to obtain the target prediction results.

[0047] Target prediction refers to the process of detecting, classifying, and locating a target object in 3D space from data (such as point clouds, depth maps, RGB images, and other sensor data). The target object's attributes include: center point coordinates (cx, cy, cz), size (w, l, h), and orientation angle.

[0048] S140: Calculate the first loss value based on the target prediction results and actual detection results corresponding to the sample image group.

[0049] The actual detection results corresponding to the sample image group are the already labeled detection results. The accuracy of the target detection model can be verified by calculating the first loss value of the target prediction result and the actual detection result. The smaller the first loss value, the higher the corresponding accuracy; the larger the first loss value, the lower the corresponding accuracy. At this time, it is necessary to continue training the target detection model.

[0050] S150: If the first loss value does not meet the preset conditions, continue to train the target detection model using other sample image groups until the first loss value meets the preset conditions.

[0051] The preset condition is a specific numerical value. In this embodiment, if the first loss value is less than the value, it means that the accuracy of the target detection model has reached the expected level, and the training of the target detection model is completed. If the first loss value is greater than or equal to the value, it means that the accuracy of the target detection model has not reached the expected level, and other sample image groups need to be used to train the target detection model until the first loss value meets the preset condition.

[0052] This application provides a method for training an object detection model. First, a set of sample images is acquired using a camera device. Then, the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to each set of sample images are determined. The BEV fusion features and semantic segmentation features are then input into the object detection model to obtain the object prediction result. A first loss value is calculated based on the object prediction result and the actual detection result corresponding to the set of sample images. If the first loss value does not meet the preset conditions, other sets of sample images are used to train the object detection model until the first loss value meets the preset conditions.

[0053] Since the data used in the training process of the object detection model in this application are BEV fusion features and semantic segmentation features, that is, by leveraging the advantages of BEV fusion features in visual tasks and adding semantic segmentation features to train the object detection model, the semantic segmentation features improve the object detection model's ability to represent BEV fusion features. Thus, this application can improve the training accuracy of the object detection model, and consequently, the object detection model in this application can improve the efficiency of object detection.

[0054] Figure 3 and Figure 4 A flowchart of another method for training an object detection model provided in an embodiment of this application is shown. The method may include the following steps:

[0055] S310: Acquire sample image groups and internal and external parameters of the camera device through the camera device, and determine the 3D grid point projection reference point based on the internal and external parameters.

[0056] Specifically, acquiring the intrinsic and extrinsic parameters of the camera equipment includes: acquiring the intrinsic parameters of the camera equipment and acquiring the extrinsic parameters of the camera equipment based on the camera calibration model. The camera calibration model is a network model trained on data. The extrinsic parameters of the camera equipment can be obtained through this model, allowing the camera equipment to be configured accordingly. The intrinsic parameters of the camera equipment are parameters related to the camera equipment's own characteristics, such as its focal length and pixel size; the extrinsic parameters are parameters of the camera equipment in the world coordinate system, such as its position and rotation direction.

[0057] In real-world scenarios, calibration errors, sensor synchronization issues, and vehicle movement often cause offset errors in the extrinsic parameters of the camera equipment. Manually adjusting these parameters is often time-consuming and labor-intensive. Inaccurate extrinsic parameters mean varying degrees of deviation in the mapping relationship between the point cloud and the image, which can affect the accuracy of some detection methods that utilize the projection process of extrinsic parameters. Figure 5 and Figure 6 A flowchart illustrating a training method for a camera calibration model provided in this application embodiment is included, and the method may include the following steps:

[0058] S510: Obtain updated camera device external parameters based on the camera calibration model.

[0059] correspond Figure 6 The updated camera device extrinsic parameters (RT update) are obtained through the camera calibration model (CalibNet extrinsic parameter RT correction network).

[0060] S520: Re-determines the depth image based on the updated camera device external parameters.

[0061] In this embodiment, after obtaining the updated external parameters of the camera device, a color image (RGB image) is acquired by the camera device with the updated external parameters. Then, a depth map is rendered from the laser point cloud data based on various conversion parameters, and an RGB depth image is further rendered through color mapping.

[0062] S530: Input the color image and the redefined depth image into the target detection model to obtain the sample color image mask and the sample depth image update mask.

[0063] Specifically, the color image and the redefined depth image are fed into the object detection model (through...). Figure 6 The process involves using a pre-trained SAM model to obtain sample color image masks (segmentation mask 1) and sample depth image update masks (segmentation mask 2). Using segmentation mask 1 as the sample label, segmentation mask 2 and its Dice loss function are calculated, gradients are backpropagated for optimization, and the model parameters of the object detection model are updated online.

[0064] S540: Calculate the second loss value based on the sample color image mask and the sample depth image updated mask.

[0065] S550: Update the camera calibration model based on the second loss value until the second loss value meets the second preset condition, then stop training the camera calibration model.

[0066] Wherein, the second loss value corresponds to Figure 6 In the Loss1 parameter, after obtaining Loss1, it is determined whether Loss1 is less than a preset value. If it is less than the preset value, training of the camera calibration model is stopped; if it is greater than or equal to the preset value, the camera device extrinsic parameters of the camera calibration model are reacquired, and then the process is repeated. Figure 6 The calculation of Loss1 continues until Loss1 is less than a preset value, thus completing the training of the camera calibration model.

[0067] In this embodiment, the camera calibration model (CalibNet) can be composed of an Encoder-Decoder structure. The input extrinsic parameters are encoded, represented by high-dimensional embeddings, and fed into the Encoder-Decoder structure, outputting updated camera extrinsic parameters that can be directly used. As mentioned earlier, given the camera extrinsic parameters and laser point cloud, an RGB depth map can be rendered. The RGB image and RGB depth map are then fed into the SAM model to obtain segmentation mask 1 and segmentation mask 2, respectively. Using segmentation mask 1 as the sample label, the loss function of segmentation mask 2 is calculated, and gradients are backpropagated for optimization. Only the CalibNet network is learnable in the online update module, which is equivalent to implicitly correcting the camera extrinsic parameters through the loss of the mask.

[0068] This embodiment addresses extrinsic parameter errors caused by vehicle body movement, sensor synchronization issues, and calibration errors by updating the camera's extrinsic parameters online using a camera calibration model. Therefore, this embodiment improves the accuracy of determining the camera's extrinsic parameters, thereby enhancing the accuracy of the target detection model training.

[0069] It should be noted that the frequency of online updating of the camera device's extrinsic parameters in this embodiment is lower than that of the target detection model. A specified time interval can be set for online updating, or the motion and environmental conditions that trigger online updating of extrinsic parameters can be set according to the vehicle's motion status and environmental perception analysis. Furthermore, the online updated extrinsic parameters can be used for other downstream tasks.

[0070] S320: Determine the BEV fusion features and semantic segmentation features corresponding to the sample image groups respectively.

[0071] like Figure 4 As shown, the sample image group is input into the image feature extractor and the OVSeg model respectively. The sample image group is processed by the image feature extractor to obtain image feature data, and then spatiotemporal fusion is performed to obtain BEV fusion features. The sample image group is input into the OVSeg model to obtain semantic segmentation features, which include the target semantic center point and the target feature vector.

[0072] S330: Input the BEV fusion features, semantic segmentation features, and 3D grid point projection reference points into the target detection model to obtain the target prediction results.

[0073] like Figure 4 As shown, in this embodiment, the BEV fusion features, semantic segmentation features, and 3D grid point projection reference points are input into the BEV decoder to obtain the target prediction result.

[0074] The input to the BEV decoder can include a key, a value (i.e., BEV fusion features), and queries (semantic query, raster sampling position query, and query to be learned). After obtaining the BEV fusion features at the current time step through the pre-processing network, these features are used as the key and value inputs to the BEV encoder, and the 3D raster point projections are used as reference points to set the n queries to be learned.

[0075] S340: Calculate the first loss value based on the target prediction results and actual detection results corresponding to the sample image group.

[0076] S350: If the first loss value does not meet the preset conditions, continue to train the target detection model using other sample image groups until the first loss value meets the preset conditions.

[0077] It should be noted that steps S340 and S350 in this embodiment are different from those in the previous embodiment. Figure 1 The corresponding steps are described in the same way, and will not be repeated here in this embodiment.

[0078] This application provides another method for training a target detection model. BEV fusion features, semantic segmentation features, and 3D grid point projection reference points are input into the target detection model to obtain target prediction results. A first loss value is calculated based on the target prediction results and actual detection results corresponding to the sample image group. If the first loss value does not meet a preset condition, other sample image groups are used to train the target detection model until the first loss value meets the preset condition. Since the 3D grid point projection reference points in this embodiment are determined based on a camera calibration model, and this camera calibration model updates the external parameters of the camera equipment online, it can address external parameter errors caused by vehicle body movement, sensor synchronization, and calibration errors. Therefore, this embodiment can improve the accuracy of target detection model training. Furthermore, this application utilizes the advantages of BEV fusion features in visual tasks and adds semantic segmentation features to train the target detection model. The semantic segmentation features enhance the target detection model's ability to represent BEV fusion features, thereby improving the accuracy of target detection model training and ultimately increasing the efficiency of target detection.

[0079] This application provides a target detection method, which acquires a group of images to be identified using a camera device; then, the group of images to be identified is input into the pre-trained target detection model to obtain the target prediction result corresponding to the group of images to be identified.

[0080] In one application scenario provided in this application, firstly, a color image captured by a multi-view camera (generally 4 or 6 views) at a certain moment during the vehicle's movement is acquired through a camera device. The color image is the image to be identified. Then, the color image is input into the pre-trained target detection model to obtain the target prediction result corresponding to the group of images to be identified.

[0081] Figure 7 This is a schematic diagram of a training device 700 for an object detection model provided in an embodiment of this application. Figure 7 As shown, the device 700 includes:

[0082] Acquisition module 701 is used to acquire a group of sample images through a camera device;

[0083] The determination module 702 is used to determine the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to the sample image groups respectively;

[0084] Prediction module 703 is used to input BEV fusion features and semantic segmentation features into the target detection model to obtain target prediction results;

[0085] Calculation module 704 is used to calculate the first loss value based on the target prediction result and the actual detection result corresponding to the sample image group;

[0086] The training module 705 is used to continue training the target detection model using other sample image groups if the first loss value does not meet the preset conditions, until the first loss value meets the preset conditions.

[0087] In some implementations, the determining module 702 is specifically used to: extract image data features corresponding to each sample image in the sample image group through the image feature extractor in the target detection model; the sample image group includes sample images corresponding to multiple viewpoints; and determine the BEV fusion features corresponding to the sample image group based on the image data features corresponding to the sample image group and the image data features corresponding to the sample image group at two moments before the sample image group.

[0088] In some implementations, the determining module 702 is specifically used to: input the sample image group into the Open Vocabulary Segmentation (OVSeg) model to obtain the target semantic center point and target feature vector corresponding to the sample image group; and determine the target semantic center point and target feature vector as the semantic segmentation features corresponding to the sample image group.

[0089] In some implementations, the acquisition module 701 is further configured to: acquire the intrinsic and extrinsic parameters of the camera device; the determination module 702 is further configured to: determine the 3D grid point projection reference point based on the intrinsic and extrinsic parameters; and the prediction module 703 is further configured to: input the BEV fusion feature, semantic segmentation feature, and 3D grid point projection reference point into the target detection model to obtain the target prediction result.

[0090] In some implementations, the acquisition module 701 is specifically used to: acquire the external parameters of the camera device based on the camera calibration model.

[0091] In some implementations, the training module 705 is further configured to: obtain updated camera device extrinsic parameters based on the camera calibration model; redetermine the depth image based on the camera device extrinsic parameters; input the color image and the redetermined depth image into the target detection model to obtain a sample color image mask and a sample depth image update mask; calculate a second loss value based on the sample color image mask and the sample depth image update mask; update the camera calibration model based on the second loss value until the second loss value satisfies a second preset condition, and then stop training the camera calibration model.

[0092] It should be understood that the device embodiments and the training method embodiments of the object detection model can correspond to each other, and similar descriptions can be found in the training method embodiments of the object detection model. To avoid repetition, further details are omitted here. Specifically, Figure 7The apparatus 700 shown can execute the above-described training method embodiment for the target detection model, and the aforementioned and other operations and / or functions of each module in the apparatus 700 are respectively for implementing the corresponding process in the above-described training method for the target detection model. For the sake of brevity, they will not be described in detail here.

[0093] Figure 8 This is a schematic diagram of a target detection device 800 provided in an embodiment of this application. Figure 8 As shown, the device 800 includes:

[0094] The acquisition module 801 is used to acquire a group of images to be identified through a camera device;

[0095] The detection module 802 is used to input the image group to be identified into the above target detection model to obtain the target prediction result corresponding to the image group to be identified.

[0096] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, each step of the target detection model training method embodiment in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the target detection model training method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above-described target detection model training method embodiment.

[0097] Figure 9 This is a schematic block diagram of the electronic device 900 provided in an embodiment of this application. Figure 9 As shown, the electronic device 900 may include a processor 901 and a memory 902. The electronic device 900 may also include one or more of a multimedia component 903, an input / output (I / O) interface 904, and a communication component 905.

[0098] The processor 901 controls the overall operation of the electronic device 900 to complete all or part of the steps in the above-described target detection model training method. The memory 902 stores various types of data to support the operation of the electronic device 900. This data may include, for example, instructions for any application or method operating on the electronic device 900, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 903 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 902 or transmitted via communication component 905. The audio component also includes at least one speaker for outputting audio signals. I / O interface 904 provides an interface between processor 901 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 905 is used for wired or wireless communication between the electronic device 900 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 905 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0099] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the training method of the target detection model described above.

[0100] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the target detection model training method described above. For example, the computer-readable storage medium may be the memory 902 including program instructions described above, which may be executed by the processor 901 of the electronic device 900 to complete the target detection model training method described above.

[0101] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described training method for the target detection model.

[0102] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the training method of the target detection model described above when executed by the programmable device.

[0103] In another exemplary embodiment, a computer program is also provided, which causes a computer to perform the training method of the object detection model as described above.

[0104] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0106] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0107] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for an object detection model, characterized in that, The method includes: The sample image group and the internal and external parameters of the camera device are acquired by the camera device, and the 3D grid point projection reference point is determined according to the internal and external parameters of the camera device. Determine the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to the sample image group; The BEV fusion features, the semantic segmentation features, and the 3D grid point projection reference points are input into the target detection model to obtain the target prediction result; The first loss value is calculated based on the target prediction result and the actual detection result corresponding to the sample image group; If the first loss value does not meet the preset condition, then the target detection model is trained using other sample image groups until the first loss value meets the preset condition.

2. The method according to claim 1, characterized in that, The step of determining the visually centered bird's-eye view BEV fusion features corresponding to the sample image group includes: The image feature extractor in the target detection model extracts the image data features corresponding to each sample image in the sample image group; the sample image group includes sample images corresponding to multiple viewpoints. Based on the image data features corresponding to the sample image group and the image data features corresponding to the sample image group acquired at two times prior to the sample image group, the BEV fusion features corresponding to the sample image group are determined.

3. The method according to claim 1, characterized in that, Determining the semantic segmentation features corresponding to the sample image group includes: The sample image group is input into the Open Vocabulary Segmentation (OVSeg) model to obtain the target semantic center point and target feature vector corresponding to the sample image group; The target semantic center point and the target feature vector are determined as the semantic segmentation features corresponding to the sample image group.

4. The method according to claim 1, wherein obtaining the internal and external parameters of the camera device includes: The external parameters of the camera device are obtained based on the camera calibration model.

5. The method according to claim 4, characterized in that, The training process of the camera calibration model includes: The updated external parameters of the camera device are obtained based on the camera calibration model. The depth image is re-determined based on the updated camera device external parameters; The color image and the redefined depth image are input into the target detection model to obtain the sample color image mask and the sample depth image update mask; Calculate the second loss value based on the sample color image mask and the sample depth image update mask; The camera calibration model is updated based on the second loss value until the second loss value meets the second preset condition, at which point the training of the camera calibration model is stopped.

6. A target detection method, characterized in that, The method includes: The image group to be identified is acquired through a camera device; The image group to be identified is input into the target detection model according to any one of claims 1-5 to obtain the target prediction result corresponding to the image group to be identified.

7. A training device for an object detection model, characterized in that, include: The acquisition module is used to acquire a sample image group and the internal and external parameters of the camera device through the camera device, and to determine the 3D grid point projection reference point based on the internal and external parameters of the camera device; The determination module is used to determine the visually centered bird's-eye view BEV fusion features and semantic segmentation features corresponding to the sample image group; The prediction module is used to input the BEV fusion features, the semantic segmentation features, and the 3D grid point projection reference points into the target detection model to obtain the target prediction result; The calculation module is used to calculate the first loss value based on the target prediction result and the actual detection result corresponding to the sample image group; The training module is configured to continue training the target detection model using other sample image groups if the first loss value does not meet the preset conditions, until the first loss value meets the preset conditions.

8. A target detection device, characterized in that, The device includes: The acquisition module is used to acquire a group of images to be identified through a camera device; The detection module is used to input the image group to be identified into the target detection model according to any one of claims 1-5 to obtain the target prediction result corresponding to the image group to be identified.

9. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1-6.

Images