Training method, detection method, system and storage medium of target detection model

Abstract

The application relates to a target detection model training method, a detection method, a detection system and a storage medium. The target detection model comprises a decoder and an encoder, the training method comprises the following steps: acquiring a sample bird's-eye view and point cloud information corresponding to the sample bird's-eye view, wherein the sample bird's-eye view comprises label information of a detection object; based on a spatial transformation network of the encoder, the sample bird's-eye view is spatially encoded with the point cloud information as a sample to obtain a high-dimensional feature vector, and the high-dimensional feature vector comprises spatial information and pixel information; the high-dimensional feature vector is decoded through a detection head of the decoder to obtain detection information of the detection object; and the parameters of the target detection model are adjusted according to the detection information and the label information. The above method trains the target detection model by using the sample bird's-eye view and the point cloud information, and improves the detection precision of the target detection model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method, detection method, system, and storage medium for an object detection model. Background Technology

[0002] In existing detection model training methods, when collecting training samples, multiple front views of the vehicle are often obtained first, and then information is labeled for each front view. The labeling and alignment process takes a lot of time, which increases the cost of training the detection model. Summary of the Invention

[0003] Therefore, it is necessary to provide a training method, detection method, system, and storage medium for a target detection model that can reduce training costs and improve detection accuracy, in order to address the above-mentioned technical problems.

[0004] On the one hand, a training method for an object detection model is provided, the object detection model including a decoder and an encoder, the method including:

[0005] Obtain a bird's-eye view of the sample and the point cloud information corresponding to the bird's-eye view of the sample. The bird's-eye view of the sample includes the annotation information of the detected object.

[0006] Based on the encoder spatial transformation network, point cloud information is used as samples to spatially encode the sample bird's-eye view to obtain a high-dimensional feature vector, which includes spatial information and pixel information.

[0007] The high-dimensional feature vector is decoded by the detector head of the decoder to obtain the detection information of the detected object;

[0008] The parameters of the target detection model are adjusted based on the detection and annotation information.

[0009] In some embodiments, the step of spatially encoding a sample bird's-eye view using point cloud information as samples based on the encoder's spatial transformation network to obtain a high-dimensional feature vector includes:

[0010] The point cloud information is input into the point cloud coding network to obtain a point cloud pseudo-image;

[0011] Based on the spatial transformation network, the sample bird's-eye view is spatially encoded using point cloud pseudo-images as samples to obtain high-dimensional feature vectors.

[0012] In some embodiments, the step of spatially encoding the sample bird's-eye view using point cloud pseudo-images as samples based on a spatial transformation network to obtain a high-dimensional feature vector includes:

[0013] High-dimensional feature maps are obtained by extracting high-dimensional features from the bird's-eye view of the sample through a multi-layer large convolutional kernel network of the encoder.

[0014] Based on the spatial transformation network, the high-dimensional feature map is spatially encoded using point cloud pseudo-images as samples to obtain high-dimensional feature vectors.

[0015] In some embodiments, the step of spatially encoding a high-dimensional feature map using a point cloud pseudo-image as a sample, based on a spatial transformation network, to obtain a high-dimensional feature vector includes:

[0016] The high-dimensional feature map and point cloud pseudo-map are input into the spatial transformation network to obtain the spatial transformation matrix;

[0017] High-dimensional feature vectors are obtained by processing the high-dimensional feature map using a spatial transformation matrix.

[0018] In some embodiments, the step of obtaining a sample bird's-eye view and the point cloud information corresponding to the sample bird's-eye view includes:

[0019] Obtain multiple sample ring views of the vehicle;

[0020] Obtain a bird's-eye view of the samples based on multiple sample ring views.

[0021] In some embodiments, the step of obtaining a bird's-eye view of a sample based on multiple sample ring views includes:

[0022] Based on the vehicle's body posture information, multiple sample surround views are stitched together to obtain a bird's-eye view based on the samples; and / or

[0023] Multiple sample ring views are input into the subpixel generation network to obtain a sample bird's-eye view.

[0024] In some embodiments, the detection information includes detection bounding box information and detection category information, and the annotation information includes ground truth bounding box information and ground truth category information. The step of adjusting the parameters of the target detection model based on the detection information and annotation information includes:

[0025] The parameters of the object detection model are adjusted based on the first loss value between the detected bounding box information and the ground truth bounding box information, and / or the second loss value between the detected class information and the ground truth class information.

[0026] On the other hand, embodiments of this application also provide a detection method, which performs detection using a trained target detection model, the target detection model including a decoder and an encoder, and the method includes:

[0027] Obtain actual bird's-eye view;

[0028] A spatial transformation network based on an encoder is used to spatially encode the actual bird's-eye view to obtain a high-dimensional feature vector.

[0029] The high-dimensional feature vector is decoded by the detector head of the decoder to obtain the detection result of the object.

[0030] In some embodiments, the step of spatially encoding the actual bird's-eye view based on the encoder's spatial transformation network to obtain a high-dimensional feature vector includes:

[0031] High-dimensional feature maps are obtained by extracting high-dimensional features from actual bird's-eye views through a multi-layer large convolutional kernel network of the encoder.

[0032] High-dimensional feature maps are spatially encoded using a spatial transformation network to obtain high-dimensional feature vectors.

[0033] In some embodiments, the step of spatially encoding a high-dimensional feature map using a spatial transformation network to obtain a high-dimensional feature vector includes:

[0034] The high-dimensional feature map is input into the spatial transformation network to obtain the spatial transformation matrix;

[0035] High-dimensional feature vectors are obtained by processing the high-dimensional feature map using a spatial transformation matrix.

[0036] In some embodiments, the step of obtaining an actual bird's-eye view includes:

[0037] Obtain multiple actual surround view images of the vehicle;

[0038] Obtain an actual bird's-eye view based on multiple actual panoramic views.

[0039] In some embodiments, the step of obtaining an actual bird's-eye view based on multiple actual panoramic views includes:

[0040] Based on the vehicle's body posture information, multiple actual surround view images are transformed and stitched together to obtain an actual bird's-eye view; and / or

[0041] Multiple actual panoramic views are input into the subpixel generation network to obtain the actual bird's-eye view.

[0042] On the other hand, a detection system is provided, including:

[0043] The server includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. When the first processor executes the first computer program, it implements the steps of the training method described above.

[0044] The vehicle includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor. When the second processor executes the second computer program, it implements the steps of the detection method described above.

[0045] On another front, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described detection method.

[0046] The training method, detection method, system, and storage medium of the aforementioned object detection model utilize sample bird's-eye view images for training, reducing the workload of information annotation and alignment. Furthermore, this scheme also uses point cloud information to compensate for the missing depth information in the sample bird's-eye view images, improving the detection accuracy of the object detection model. Attached Figure Description

[0047] Figure 1 This diagram illustrates the application environment of the training method and detection method for the target detection model provided in the embodiments of this application.

[0048] Figure 2 This is a flowchart illustrating the training method for the target detection model provided in this application embodiment.

[0049] Figure 3 This is a schematic diagram illustrating a training method for the target detection model provided in an embodiment of this application.

[0050] Figure 4 This is a schematic flowchart of the detection method provided in the embodiments of this application.

[0051] Figure 5 This is a schematic diagram of a scenario for the detection method provided in the embodiments of this application.

[0052] Figure 6 This is an internal structure diagram of the server provided in an embodiment of this application.

[0053] Figure 7 This is an internal structural diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The training and detection methods for the target detection model provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 1 communicates with server 2 via a network. Terminal 1 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, vehicle terminals, and robots, etc., with operating systems and imaging devices. Server 2 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0056] In some embodiments, the object detection model can be deployed on server 2. During the training process of the object detection model, terminal 1 provides training data to server 2, such as sample loop views and point cloud information. Server 2 then trains the object detection model based on the training data. Similarly, during the actual detection process using the object detection model, terminal 1 provides actual input data to server 2, such as actual loop views. Server 2 then performs detection based on the actual input data using the object detection model and sends the detection results back to terminal 1.

[0057] In some embodiments, terminal 1 may also download a pre-trained object detection model from server 2. During the detection process, after acquiring the actual bird's-eye view, terminal 1 obtains the detection result through the processing of the object detection model downloaded thereon.

[0058] like Figure 2 As shown, this application provides a method for training an object detection model. This object detection model can perform relevant detections on objects. In a vehicle driving scenario, the objects to be detected include vehicles, parking spaces, the ground, and people. The types of detection performed on the objects include: determining the category of the object, determining the bounding box of the object, such as determining whether the object is a vehicle or the ground, and determining the bounding box of the vehicle's position. Figure 3 As shown, the target detection model includes an encoder and a decoder.

[0059] Next, this training method will be applied to Figure 1 Taking the server in the example, this training method includes the following steps:

[0060] Step S101: Obtain the sample bird's-eye view and the point cloud information corresponding to the sample bird's-eye view. The sample bird's-eye view includes the annotation information of the detected object.

[0061] Among them, the sample bird's-eye view is an image obtained from an aerial perspective, which contains rich information such as color, texture, and semantics.

[0062] In vehicle driving scenarios, multiple sample surround views can be acquired using the vehicle's multi-camera system to obtain a sample bird's-eye view (BEV). In some embodiments, based on the vehicle's body posture information, multiple sample surround views can be transformed to obtain multiple sample top views, and then these multiple sample top views can be stitched together to obtain the sample bird's-eye view. In some embodiments, such as... Figure 3 As shown, multiple sample ring views can also be input into the subpixel generation network to obtain a high-resolution sample bird's-eye view.

[0063] The objects to be detected in the sample bird's-eye view can be labeled in advance to obtain labeling information. For example, the category of the object to be detected can be labeled, and bounding boxes can be used to label the object to be detected. Since in this embodiment, only the sample bird's-eye view needs to be labeled, and it is not necessary to label the multiple panoramic views that make up the sample bird's-eye view separately, the cost of training the object detection model is reduced by reducing the amount of data preprocessing.

[0064] In some embodiments, point cloud information can be obtained by scanning the scene with LiDAR. Point cloud information includes the three-dimensional coordinates and reflection intensity of points. It should be noted that although sample bird's-eye views contain rich information, they record information by projecting the physical world onto a two-dimensional camera plane, thus lacking depth information. If only sample bird's-eye views are used to train an object detection model, the model will have poor localization capabilities due to the lack of explicit depth information, resulting in insufficient accuracy in the bounding boxes of identified objects. Point cloud information, on the other hand, expresses the information of objects in three-dimensional space, containing detailed depth information. Therefore, point cloud information corresponding to sample bird's-eye views can improve the detection accuracy of object detection models.

[0065] Step S102: Based on the spatial transformation network of the encoder, the sample bird's-eye view is spatially encoded using point cloud information as samples to obtain a high-dimensional feature vector, which includes spatial information and pixel information.

[0066] Spatial Transform Networks (STNs) are neural networks with spatial transformation feature extractors. They can adaptively transform and align information spatially according to classification or tasks, such as translation, scaling, rotation, and other geometric transformations. Using point cloud information as samples, the high-dimensional feature vector obtained by spatially encoding the bird's-eye view of the samples using a spatial transformation network not only contains spatial information, such as the world coordinates of pixels, but also pixel information, such as color.

[0067] In some embodiments, the step of spatially encoding a sample bird's-eye view using point cloud information as samples based on an encoder spatial transformation network to obtain a high-dimensional feature vector includes:

[0068] The point cloud information is input into the point cloud coding network to obtain a point cloud pseudo-image.

[0069] Based on the spatial transformation network, the sample bird's-eye view is spatially encoded using point cloud pseudo-images as samples to obtain high-dimensional feature vectors.

[0070] like Figure 3 As shown, by using a point cloud encoding network, each point in the point cloud information can be projected onto real-world coordinates, thus obtaining a point cloud pseudo-image.

[0071] In some embodiments, high-dimensional features can first be extracted from the sample bird's-eye view using a multi-layer large convolutional kernel network of the encoder to obtain a high-dimensional feature map. Then, based on a spatial transformation network, spatial encoding is performed on the high-dimensional feature map using point cloud pseudo-images as samples to obtain a high-dimensional feature vector.

[0072] In some embodiments, such as Figure 3 As shown, high-dimensional feature maps and point cloud pseudo-maps can also be input into a spatial transformation network to obtain a spatial transformation matrix. The high-dimensional feature map is then processed using this spatial transformation matrix to obtain high-dimensional feature vectors.

[0073] Step S103: The high-dimensional feature vector is decoded by the detection head of the decoder to obtain the detection information of the object.

[0074] The detection head decodes the high-dimensional feature vector to obtain the corresponding input data according to its detection task, and then determines the detection information of the object to be detected based on the input data. In this embodiment, since point cloud information with depth information is added, not only two-dimensional bounding box detection tasks can be realized, but also three-dimensional bounding box detection tasks can be realized.

[0075] In some embodiments, different types of detection heads can be selected depending on the detection task. For example, if the detection task is the 3D bounding box of a car, the detection head can be set to decode the high-dimensional feature vector to obtain the car's center coordinates, length, width, and height information, etc. If the detection task is the 2D bounding box of a car, the detection head can be set to decode the high-dimensional feature vector to obtain the car's planar horizontal and vertical coordinates, width, and length information, etc. It should be noted that, in order to apply multiple detection heads to multiple detection tasks, the input size of the detection head can be pre-aligned with the output size of the encoder.

[0076] Step S104: Adjust the parameters of the target detection model based on the detection information and annotation information.

[0077] The detection information may include detection bounding box information and / or detection category information. The annotation information may include ground truth bounding box information and / or ground truth category information.

[0078] When the detection information includes detection bounding box information and the annotation information includes ground truth bounding box information, the parameters of the target detection model can be adjusted based on the first loss value between the detection bounding box information and the ground truth bounding box information.

[0079] When the detection information includes detection category information and the annotation information includes true category information, the parameters of the target detection model can be adjusted based on the second loss value between the detection category information and the true category information.

[0080] Detection information can include detection bounding box information and detection category information. When annotation information can include ground truth bounding box information and ground truth category information, the parameters of the target detection model can be adjusted by combining the first loss value between the detection bounding box information and the ground truth bounding box information and the second loss value between the detection category information and the ground truth category information.

[0081] The first and second loss values ​​can be calculated using the appropriate loss function selected based on the detected object, and no specific limitations are made here.

[0082] The target detection model training method provided in this application uses sample bird's-eye view images to train the target detection model, reducing the workload of information annotation and annotation alignment. Furthermore, this method also uses point cloud information to compensate for the missing depth information in the sample bird's-eye view images, improving the detection accuracy of the target detection model.

[0083] like Figure 4 As shown, this application provides a detection method that uses the trained target detection model described above for detection. Next, this detection method will be applied to... Figure 1 Taking the terminal in the example, this detection method includes the following steps:

[0084] Step S201: Obtain the actual bird's-eye view.

[0085] The terminal can acquire a real bird's-eye view by capturing multiple actual surround view images using multiple cameras mounted on it. In some embodiments, based on the vehicle's body posture information, the multiple actual surround view images can be transformed into multiple surround view images, and then the surround view images can be stitched together to obtain a real bird's-eye view. In some embodiments, such as Figure 5 As shown, multiple actual panoramic views can also be input into the subpixel generation network to obtain a high-resolution actual bird's-eye view.

[0086] Step S202: Based on the spatial transformation network of the encoder, the actual bird's-eye view is spatially encoded to obtain a high-dimensional feature vector.

[0087] In some embodiments, the step of spatially encoding the actual bird's-eye view based on the encoder's spatial transformation network to obtain a high-dimensional feature vector includes:

[0088] High-dimensional feature maps are obtained by extracting high-dimensional features from actual bird's-eye views through a multi-layered large convolutional kernel network of the encoder.

[0089] High-dimensional feature maps are spatially encoded using a spatial transformation network to obtain high-dimensional feature vectors.

[0090] Specifically, high-dimensional feature maps can be spatially encoded using transformation information generated by a spatial transformation network to obtain high-dimensional feature vectors. These high-dimensional feature vectors include both spatial and pixel information.

[0091] In some embodiments, the step of spatially encoding a high-dimensional feature map using a spatial transformation network to obtain a high-dimensional feature vector includes:

[0092] The high-dimensional feature map is input into the spatial transformation network to obtain the spatial transformation matrix.

[0093] High-dimensional feature vectors are obtained by processing the high-dimensional feature map using a spatial transformation matrix.

[0094] Step S203: The high-dimensional feature vector is decoded by the detection head of the decoder to obtain the detection result of the object.

[0095] The detection head decodes the high-dimensional feature vector to obtain the corresponding input data according to its detection task, and then determines the detection information of the object to be detected based on the input data. In this embodiment, since point cloud information with depth information is added, the detection head can not only perform two-dimensional bounding box detection tasks, but also three-dimensional bounding box detection tasks.

[0096] In the detection method provided in this embodiment, the spatial transformation network in the trained target detection model is used to spatially encode the actual bird's-eye view, resulting in a high-dimensional feature vector containing spatial and pixel information, which can achieve detection for various tasks.

[0097] It should be understood that, although Figure 2-5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-5 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0098] In some embodiments, a detection system is provided. The detection system includes a server and a vehicle. The internal structure diagram of the server can be as follows: Figure 6As shown, the server includes a first processor, a first memory, a first network interface, and a database connected via a first system bus. The first processor provides computing and control capabilities. The first memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a first computer program, and the database. The internal memory provides an environment for the operation of the operating system and the first computer program in the non-volatile storage medium. The server's database stores data for training an object detection model and data for detection. The server's network interface communicates with external terminals via a network connection. When the first computer program is executed by the first processor, it implements a method for training an object detection model.

[0099] Vehicle internal structure diagrams can be as follows Figure 7 As shown, the vehicle includes a second processor, a second memory, a second network interface, a display screen, an input device, a camera device, and a lidar connected via a second system bus. The second processor provides computing and control capabilities. The vehicle's memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a second computer program. The internal memory provides an environment for the operation of the operating system and the second computer program in the non-volatile storage medium. The camera device includes multiple cameras for capturing a panoramic view of the scene. The lidar is used to acquire point cloud information. The vehicle's network interface is used for communication with external terminals via a network connection. The second computer program, when executed by the second processor, implements a detection method. The vehicle's display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the vehicle's exterior, or an external keyboard, touchpad, or mouse.

[0100] Those skilled in the art will understand that Figure 6 and Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the server and vehicle to which the present application is applied. Specific servers and vehicles may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] This application provides a server, including a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. In one aspect, when the first processor executes the first computer program, it performs the following steps:

[0102] The process involves acquiring a sample bird's-eye view and its corresponding point cloud information, including the annotation information of the target object. Based on a spatial transformation network of the encoder, the sample bird's-eye view is spatially encoded using the point cloud information as samples to obtain a high-dimensional feature vector. This high-dimensional feature vector is then decoded by the decoder's detection head to obtain the detection information of the target object. Based on the detection and annotation information, the parameters of the target detection model are adjusted.

[0103] In some embodiments, when the first processor executes the first computer program, it further performs the following steps:

[0104] Point cloud information is input into a point cloud encoding network to obtain a point cloud pseudo-image. Based on a spatial transformation network, the point cloud pseudo-image is used as a sample to perform spatial encoding on the sample bird's-eye view, resulting in a high-dimensional feature vector.

[0105] In some embodiments, when the first processor executes the first computer program, it further performs the following steps:

[0106] High-dimensional features are extracted from the sample bird's-eye view using a multi-layered large convolutional kernel network of the encoder, resulting in a high-dimensional feature map. Based on a spatial transformation network, the high-dimensional feature map is spatially encoded using point cloud pseudo-images as samples, yielding a high-dimensional feature vector.

[0107] In some embodiments, when the first processor executes the first computer program, it further performs the following steps:

[0108] The high-dimensional feature map and point cloud pseudo-image are input into the spatial transformation network to obtain the spatial transformation matrix. The high-dimensional feature map is then processed using the spatial transformation matrix to obtain high-dimensional feature vectors.

[0109] In some embodiments, when the first processor executes the first computer program, it further performs the following steps:

[0110] Obtain multiple sample surround views of the vehicle. Based on these multiple sample surround views, obtain a sample bird's-eye view.

[0111] In some embodiments, when the first processor executes the first computer program, it further performs the following steps:

[0112] Based on the vehicle's body posture information, multiple sample surround views are stitched together to obtain a sample bird's-eye view. And / or multiple sample surround views are input into a subpixel generation network to obtain a sample bird's-eye view.

[0113] In some embodiments, when the first processor executes the first computer program, it further performs the following steps:

[0114] The parameters of the object detection model are adjusted based on the first loss value between the detected bounding box information and the ground truth bounding box information, and / or the second loss value between the detected class information and the ground truth class information.

[0115] On the other hand, embodiments of this application provide a vehicle including a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor. In one aspect, when the second processor executes the second computer program, it performs the following steps:

[0116] Obtain the actual bird's-eye view. Based on the spatial transformation network of the encoder, the actual bird's-eye view is spatially encoded to obtain a high-dimensional feature vector. The high-dimensional feature vector is then decoded by the detection head of the decoder to obtain the detection result of the object.

[0117] In some embodiments, when the second processor executes the second computer program, it further performs the following steps:

[0118] High-dimensional features are extracted from the actual bird's-eye view using a multi-layered large convolutional kernel network of the encoder, resulting in a high-dimensional feature map. This high-dimensional feature map is then spatially encoded using a spatial transformation network to obtain a high-dimensional feature vector.

[0119] In some embodiments, when the second processor executes the second computer program, it further performs the following steps:

[0120] The high-dimensional feature map is input into a spatial transformation network to obtain a spatial transformation matrix. The high-dimensional feature map is then processed using the spatial transformation matrix to obtain high-dimensional feature vectors.

[0121] In some embodiments, when the second processor executes the second computer program, it further performs the following steps:

[0122] Obtain multiple actual surround view images of the vehicle. Based on these images, obtain an actual bird's-eye view.

[0123] In some embodiments, when the second processor executes the second computer program, it further performs the following steps:

[0124] Based on the vehicle's body attitude information, multiple actual surround view images are transformed and stitched together to obtain an actual bird's-eye view. Alternatively, multiple actual surround view images can be input into a subpixel generation network to obtain the actual bird's-eye view.

[0125] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the following steps:

[0126] The process involves acquiring a sample bird's-eye view and its corresponding point cloud information, including the annotation information of the target object. Based on a spatial transformation network of the encoder, the sample bird's-eye view is spatially encoded using the point cloud information as samples to obtain a high-dimensional feature vector. This high-dimensional feature vector is then decoded by the decoder's detection head to obtain the detection information of the target object. Based on the detection and annotation information, the parameters of the target detection model are adjusted.

[0127] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0128] Point cloud information is input into a point cloud encoding network to obtain a point cloud pseudo-image. Based on a spatial transformation network, the point cloud pseudo-image is used as a sample to perform spatial encoding on the sample bird's-eye view, resulting in a high-dimensional feature vector.

[0129] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0130] High-dimensional features are extracted from the sample bird's-eye view using a multi-layered large convolutional kernel network of the encoder, resulting in a high-dimensional feature map. Based on a spatial transformation network, the high-dimensional feature map is spatially encoded using point cloud pseudo-images as samples, yielding a high-dimensional feature vector.

[0131] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0132] The high-dimensional feature map and point cloud pseudo-image are input into the spatial transformation network to obtain the spatial transformation matrix. The high-dimensional feature map is then processed using the spatial transformation matrix to obtain high-dimensional feature vectors.

[0133] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0134] Obtain multiple sample surround views of the vehicle. Based on these multiple sample surround views, obtain a sample bird's-eye view.

[0135] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0136] Based on the vehicle's body posture information, multiple sample surround views are stitched together to obtain a sample bird's-eye view. And / or multiple sample surround views are input into a subpixel generation network to obtain a sample bird's-eye view.

[0137] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0138] The parameters of the object detection model are adjusted based on the first loss value between the detected bounding box information and the ground truth bounding box information, and / or the second loss value between the detected class information and the ground truth class information.

[0139] On the other hand, when a computer program is executed by a processor, it performs the following steps:

[0140] Obtain the actual bird's-eye view. Based on the spatial transformation network of the encoder, the actual bird's-eye view is spatially encoded to obtain a high-dimensional feature vector. The high-dimensional feature vector is then decoded by the detection head of the decoder to obtain the detection result of the object.

[0141] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0142] High-dimensional features are extracted from the actual bird's-eye view using a multi-layered large convolutional kernel network of the encoder, resulting in a high-dimensional feature map. This high-dimensional feature map is then spatially encoded using a spatial transformation network to obtain a high-dimensional feature vector.

[0143] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0144] The high-dimensional feature map is input into a spatial transformation network to obtain a spatial transformation matrix. The high-dimensional feature map is then processed using the spatial transformation matrix to obtain high-dimensional feature vectors.

[0145] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0146] Obtain multiple actual surround view images of the vehicle. Based on these images, obtain an actual bird's-eye view.

[0147] In some embodiments, when a computer program is executed by a processor, it further performs the following steps:

[0148] Based on the vehicle's body attitude information, multiple actual surround view images are transformed and stitched together to obtain an actual bird's-eye view. Alternatively, multiple actual surround view images can be input into a subpixel generation network to obtain the actual bird's-eye view.

[0149] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A training method for an object detection model, characterized in that, The target detection model includes a decoder and an encoder, and the method includes: Obtain a sample bird's-eye view as training data and point cloud information corresponding to the sample bird's-eye view, wherein the sample bird's-eye view includes annotation information of the detected object; Based on the spatial transformation network of the encoder, the sample bird's-eye view is spatially encoded using point cloud information as samples to obtain a high-dimensional feature vector. The high-dimensional feature vector includes spatial information and pixel information, including: The point cloud information is input into the point cloud encoding network to obtain a point cloud pseudo-image; the sample bird's-eye view is subjected to high-dimensional feature extraction through the multi-layer large convolutional kernel network of the encoder to obtain a high-dimensional feature map; based on the spatial transformation network, the high-dimensional feature map is spatially encoded using the point cloud pseudo-image as a sample to obtain the high-dimensional feature vector. The high-dimensional feature vector is decoded by the detection head of the decoder to obtain the detection information of the detected object; The parameters of the target detection model are adjusted based on the detection information and the annotation information.

2. The training method for the target detection model according to claim 1, characterized in that, The step of spatially encoding the high-dimensional feature map using the point cloud pseudo-image as a sample based on the spatial transformation network to obtain the high-dimensional feature vector includes: The high-dimensional feature map and the point cloud pseudo-map are input into the spatial transformation network to obtain the spatial transformation matrix; The high-dimensional feature map is processed by the spatial transformation matrix to obtain the high-dimensional feature vector.

3. The training method for the target detection model according to claim 1, characterized in that, The step of obtaining the sample bird's-eye view includes: Obtain multiple sample ring views of the vehicle; Based on the multiple sample ring views, obtain the sample bird's-eye view.

4. The training method for the target detection model according to claim 3, characterized in that, The step of obtaining the bird's-eye view of the sample based on the multiple sample ring views includes: Based on the vehicle's body posture information, the multiple sample panoramic views are transformed and stitched together to obtain the bird's-eye view based on the samples; and / or The multiple sample ring views are input into the subpixel generation network to obtain the sample bird's-eye view.

5. The training method for the target detection model according to claim 1, characterized in that, The detection information includes detection bounding box information and detection category information, and the annotation information includes ground truth bounding box information and ground truth category information. The step of adjusting the parameters of the target detection model based on the detection information and the annotation information includes: The parameters of the target detection model are adjusted based on a first loss value between the detected bounding box information and the ground truth bounding box information, and / or a second loss value between the detected category information and the ground truth category information.

6. A detection method, characterized in that, Detection is performed using a target detection model trained by any one of the training methods of claims 1-5, wherein the target detection model includes a decoder and an encoder, and the method includes: Obtain actual bird's-eye view; Based on the spatial transformation network of the encoder, the actual bird's-eye view is spatially encoded to obtain a high-dimensional feature vector. The high-dimensional feature vector is decoded by the detection head of the decoder to obtain the detection result of the object.

7. The detection method according to claim 6, characterized in that, The step of spatially encoding the actual bird's-eye view based on the encoder to obtain a high-dimensional feature vector includes: The encoder's multi-layer large convolutional kernel network is used to extract high-dimensional features from the actual bird's-eye view to obtain a high-dimensional feature map. The high-dimensional feature map is spatially encoded using the spatial transformation network to obtain the high-dimensional feature vector.

8. The detection method according to claim 7, characterized in that, The step of spatially encoding the high-dimensional feature map using the spatial transformation network to obtain the high-dimensional feature vector includes: The high-dimensional feature map is input into the spatial transformation network to obtain the spatial transformation matrix; the high-dimensional feature map is then processed using the spatial transformation matrix to obtain the high-dimensional feature vector.

9. The detection method according to claim 6, characterized in that, The steps for obtaining the actual bird's-eye view include: Obtain multiple actual surround view images of the vehicle; The actual bird's-eye view is obtained based on the multiple actual panoramic views.

10. The detection method according to claim 9, characterized in that, The step of obtaining the actual bird's-eye view based on the multiple actual panoramic views includes: Based on the vehicle's body posture information, the multiple actual surround view images are transformed and stitched together to obtain the actual bird's-eye view; and / or the multiple actual surround view images are input into a subpixel generation network to obtain the actual bird's-eye view.

11. A detection system, characterized in that, include: The server includes a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor. When the first processor executes the first computer program, it implements the steps of the training method according to any one of claims 1-5. A vehicle, the vehicle including a second memory, a second processor and a second computer program stored in the second memory and executable on the second processor, wherein the second processor, when executing the second computer program, implements the steps of the detection method according to any one of claims 6 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 6 to 10.

Images