A model training method, object detection method, device, equipment and medium

Abstract

The embodiment of the present application provides a model training method, an object detection method, an apparatus, a device and a medium, and relates to the field of image processing technology. The model training method includes: obtaining a sample image group; inputting each sample image in the sample image group into an object detection model to be trained to obtain a prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module; according to the prediction result of the sample image group, the true value information and the position represented by the current position feature of each candidate sampling point, the model loss of the object detection model is determined; when it is determined based on the model loss that the object detection model has not converged, the model parameters of the object recognition model are adjusted, and the step of obtaining the sample image group is returned. Through the present application, different scenes can be adapted to perform multi-camera object detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a model training method, object detection method, device, equipment and medium. Background Art

[0002] Object detection can identify specific objects in images or videos (including the category and location of the object). Single-camera object detection (object detection through images taken by a single camera) cannot meet the object detection requirements of complex scenes (such as traffic scenes). For example, it is impossible to uniformly detect the entire scene; and due to occlusion, the accuracy of single-camera object detection is insufficient. Therefore, there is a need for multi-camera object detection.

[0003] Multi-camera object detection detects objects using images from multiple cameras of the same scene (the image from one camera is acquired by that camera) and outputs a unified detection result from a bird's-eye view of the scene.

[0004] However, the relevant technology is suitable for object detection in fixed scenes (the size, position, resolution of the bird's-eye view and the fixed installation of the camera) and cannot be adapted to object detection with multiple cameras in different scenes, and has high limitations. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a model training method, object detection method, device, equipment and medium to adapt to different scenes for multi-camera object detection. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0007] Acquire a sample image group; wherein the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size;

[0008] Input each sample image in the sample image group into the object detection model to be trained to obtain the prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, the feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position-aware feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image; the fusion module is used to fuse, for each candidate sampling point, the position-aware feature of the candidate sampling point with respect to each sample image to obtain the fusion feature of the candidate sampling point; the prediction module is used to generate a position offset and a category prediction result corresponding to each candidate sampling point based on the current position feature and the fusion feature of each candidate sampling point used to characterize the position, to obtain the prediction result; The position perception feature of a candidate sampling point with respect to any sample image is: a feature obtained by fusing a first feature content with respect to a reference point corresponding to the candidate sampling point in a feature map of the sample image and a second feature content with respect to a reference point corresponding to the candidate sampling point in a position coding feature of the feature map of the sample image, wherein the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from a sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, and the position coding feature of the feature map of any sample image is a feature obtained by at least performing position coding on the coordinates of each pixel in the feature map of the sample image after performing designated coordinate mapping; performing the designated coordinate mapping on each pixel in the feature map of any sample image includes: mapping each pixel in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to the second mapping relationship corresponding to the sample image;

[0009] Determining a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point;

[0010] When it is determined based on the model loss that the object detection model has not converged, the model parameters of the object recognition model are adjusted, and the process returns to the step of obtaining the sample image group.

[0011] In a second aspect, an embodiment of the present application provides an object detection method, the method comprising:

[0012] Acquire a target image group; wherein the target image group includes target images captured by each camera set up in the target detection scene at the same time;

[0013] Input each target image in the target image group into a designated object detection model to obtain a prediction result of the target image group; wherein the designated object detection model is a model obtained by adjusting the first mapping relationship corresponding to each sample image to the first designated relationship corresponding to each target image, and adjusting the second mapping relationship corresponding to each sample image used for designated coordinate mapping to the second designated relationship corresponding to each target image, with respect to the object detection model trained based on the model training method; wherein the first designated relationship corresponding to any target image is used to map any candidate sampling point from a bird's-eye view of a target detection scene to the target image, and the second designated relationship corresponding to any target image is used to map each pixel point in a feature map of the target image to a spatial coordinate system of a physical space indicated by the bird's-eye view of the target detection scene; the bird's-eye view of the target detection scene is a bird's-eye view of a physical space of a designated spatial size;

[0014] For each candidate sampling point, a predicted position is generated according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and the category prediction result of the candidate sampling point in the prediction result of the target image group is used as the obtained category prediction result corresponding to the predicted position.

[0015] In a third aspect, an embodiment of the present application provides a model training device, the device comprising:

[0016] A first acquisition module is used to acquire a sample image group; wherein the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size;

[0017] The first input module is used to input each sample image in the sample image group into the object detection model to be trained to obtain the prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, the feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position perception feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image; the fusion module is used to fuse, for each candidate sampling point, the position perception features of the candidate sampling point with respect to each sample image to obtain the fusion feature of the candidate sampling point; the prediction module is used to generate, based on the current position feature and fusion feature of each candidate sampling point used to characterize the position, a position offset and a category prediction result corresponding to each candidate sampling point to obtain the prediction result. measurement result; the position perception feature of any candidate sampling point with respect to any sample image is: a feature obtained by fusing a first feature content with respect to a reference point corresponding to the candidate sampling point in a feature map of the sample image and a second feature content with respect to a reference point corresponding to the candidate sampling point in a position coding feature of the feature map of the sample image, the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from a sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, the position coding feature of the feature map of any sample image is a coordinate after each pixel point in the feature map of the sample image is mapped to a specified coordinate, at least a feature obtained by position coding; the mapping of each pixel point in the feature map of any sample image to the specified coordinate includes: mapping each pixel point in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to a second mapping relationship corresponding to the sample image;

[0018] A determination module, configured to determine a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point;

[0019] An adjustment module is used to adjust the model parameters of the object recognition model when it is determined based on the model loss that the object detection model has not converged, and return to the step of obtaining the sample image group.

[0020] In a fourth aspect, an embodiment of the present application provides an object detection device, the device comprising:

[0021] A second acquisition module is used to acquire a target image group; wherein the target image group includes target images captured by each camera set up in the target detection scene at the same time;

[0022] A second input module is used to input each target image in the target image group into a designated object detection model to obtain a prediction result of the target image group; wherein the designated object detection model is a model obtained by adjusting the first mapping relationship corresponding to each sample image to the first designated relationship corresponding to each target image, and adjusting the second mapping relationship corresponding to each sample image used for the designated coordinate mapping to the second designated relationship corresponding to each target image for the object detection model trained based on the model training method; wherein the first designated relationship corresponding to any target image is used to map any candidate sampling point from the bird's-eye view of the target detection scene to the target image, and the second designated relationship corresponding to any target image is used to map each pixel point in the feature map of the target image to the spatial coordinate system of the physical space indicated by the bird's-eye view of the target detection scene; the bird's-eye view of the target detection scene is a bird's-eye view of a physical space of a specified space size;

[0023] A generating module is used to generate a predicted position for each candidate sampling point according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and use the category prediction result of the candidate sampling point in the prediction result of the target image group as the obtained category prediction result corresponding to the predicted position.

[0024] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0025] Memory, used to store computer programs;

[0026] The processor is used to implement any of the above-mentioned model training methods or object detection methods when executing the program stored in the memory.

[0027] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned model training methods or object detection methods.

[0028] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned model training methods or object detection methods.

[0029] Beneficial effects of the embodiments of the present application:

[0030] In the model training method provided in the embodiment of the present application, the object detection model to be trained includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, so that prediction is performed on the sample image group through each module, and the model loss of the object detection model is determined according to the prediction result of the sample image group, the true value information and the position represented by the current position feature of each candidate sampling point, so as to adjust the model parameters of the model, that is, to train the model; and the position perception feature generated by the visual change module is: the first feature content of the reference point corresponding to the candidate sampling point in the feature map of the sample image and the second feature content of the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image are fused, so that the global correspondence between the image pixels and the bird's-eye view can be supplemented on the basis of the features of the feature map, that is, the correspondence between the image pixels and the physical space indicated by the sample bird's-eye view can be supplemented, and the physical space is a specified space size, so that the difference between different detection scenes can be reduced by unifying the prediction space, that is, unifying to the physical space of the specified space size. It can be seen that through the model training method provided in the present application, a model for multi-camera object detection in different scenes can be trained, that is, the trained model can adapt to different scenes.

[0031] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0033] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of an object detection method provided in an embodiment of the present application;

[0035] Figure 3 Another schematic diagram of a flow chart of an object detection method provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of a process for generating a position coding feature provided in an embodiment of the present application;

[0037] Figure 5a It is a schematic diagram of a feature fusion method in related technologies;

[0038] Figure 5b A schematic diagram of a method for fusing location awareness features provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of a unified prediction space for different maps provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of the structure of an object detection device provided in an embodiment of the present application;

[0042] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0044] As one of the basic tasks of computer vision, object detection can identify and locate specific objects in images or videos. Single-camera object detection takes the image of a single camera as input and outputs the location (such as a bounding box) and category of the object on the image. Single-camera object detection has been widely used in intelligent transportation scenarios. However, with the increasing number of vehicles, traffic congestion and blockage are becoming more and more serious. Single-camera object detection cannot meet the increasingly complex needs of urban management. The main problem is that the viewing angle of a single camera is limited. Each camera only detects objects within a single viewing angle, and cannot perform unified detection on complex scenes such as entire intersections or urban roads. In addition, due to occlusion, the lack of accuracy of single-camera object detection further brings difficulties to the formulation of object detection strategies for complex scenes.

[0045] Therefore, there is a need for multi-camera object detection. Multi-camera object detection takes images collected by multiple cameras in the same scene at the same time as input, and outputs a unified object detection result under the bird's-eye view of the scene. Compared with single-camera object detection, multi-camera object detection has the following advantages: 1) Multi-cameras have a wider viewing angle, which can fully cover complex scenes such as intersections and perform unified object detection; 2) The same object is generally not blocked under the viewing angle of multiple cameras, which can improve the occlusion problem; 3) There is no difference in viewing angle between near and far in the bird's-eye view, which is more conducive to the development of subsequent applications.

[0046] However, the introduction of images from more cameras also increases the demand for object detection algorithms, mainly including 1) higher data calibration costs. As the number of cameras increases, the number of images that need to be calibrated rises sharply; at the same time, multiple cameras also need to perform time synchronization, joint calibration and other steps. 2) Higher computing power costs. Object detection algorithms process images from multiple cameras at the same time, and the structure of object detection algorithms (such as object detection models) is more complex; 3) High dependence on camera calibration. The input of multi-camera object detection is the image from the perspective of each camera, and the output is the result from a bird's-eye view. Perspective conversion is required through the internal and external parameters of multiple cameras.

[0047] There are two mainstream methods for multi-camera object detection: one is post-fusion, that is, each camera performs single-camera object detection separately, and the detection results are mapped to the bird's-eye view through the camera's internal and external parameters to fuse the detection results. However, the fusion logic of post-fusion is complex and the maintenance cost is high. At the same time, the fusion result is greatly affected by the camera's object detection result, and the performance ceiling is not high. The second is front fusion, that is, the image features of each camera are extracted through a shared backbone network, and the features are directly fused, and then the unified detection results in the scene are output based on the fused features. The front fusion method can simplify the overall process of multi-camera object detection, and at the same time, it is data-driven (that is, the detection results are only affected by the images of multiple cameras), and the performance ceiling is higher. The front fusion solution is widely used in the field of autonomous driving (multiple cameras are installed on the vehicle side). When it is migrated to traffic scenes (multiple cameras are installed on the road side), the model generalization problem is more serious. The main reason is that traffic scenes are more fragmented, such as 1) the number of cameras in different traffic scenes is different; 2) the order of cameras in different traffic scenes is different; 3) the size, location, and map resolution of intersections in different traffic scenes are also different; 4) the camera installations in different traffic scenes are different (the camera external parameters are different, such as installation angles, installation positions, etc.). In the face of the differences in different traffic scenes, the current single model of the multi-camera object detection algorithm is usually only applicable to traffic scenes with a fixed number and order of cameras; the detection area is also relatively fixed; and it is highly dependent on camera parameters (including internal and external parameters). The current multi-camera object detection cannot be effectively generalized to different scenes, which further increases the cost. For example, every time the scene switches, the multi-camera object detection model needs to be retrained. How to adapt to different scenes for multi-camera object detection is an urgent problem to be solved in the current application of multi-camera object detection in traffic scenes.

[0048] There are several specific solutions for multi-camera object detection in related calculations:

[0049] 1. Using the post-fusion solution, first input each camera image into the object detection model of a single camera to obtain the detection result of a single camera; then the image matching module uses the surf feature (Speeded Up Robust Features, a local feature detector and descriptor in computer vision) and the random sampling consensus algorithm to calculate the mapping matrix between the two cameras; finally, through the object redistribution module, the detection results of different cameras are fused using the mapping matrix. The shortcomings of this method are as follows: 1) The overall framework adopts a post-fusion method, relying on traditional feature point matching to obtain the mapping matrix, and combining logic to fuse the pre-generated object detection results of each single camera. However, as the number of cameras increases, the fusion logic will become more complicated. 2) The output detection results are still the detection results on each camera image. The multi-camera is mainly to supplement the occluded objects under the single camera. Each camera still performs object detection in a single viewing range, and cannot perform unified detection on complex scenes such as the entire intersection or urban roads.

[0050] 2. Using the front fusion solution, first extract features from the images of each camera, and then map the image features of each camera to the bird's-eye view for feature fusion through the internal and external parameters of the camera. Finally, perform object detection based on the fused features and output the detection results of the object from the bird's-eye view. This solution is mainly used in vehicle-side scenarios (driving scenarios). The driving scenario uses the vehicle itself as the observation center, and the generalization problem is relatively small (that is, the order, number, and installation of the cameras usually do not change). It is not optimized for the fragmented characteristics of traffic scenes; at the same time, when constructing the mapping relationship between the image space and the bird's-eye view space, a dense feature method is used. As the scene range increases, the scale of the image features will also increase significantly, thereby increasing the computational pressure.

[0051] 3. Using the pre-fusion solution, first extract features from images of multiple cameras, then initialize a set of reference positions for the object positions, and use the inverse projection of the image space and the bird's-eye view space to map the reference positions to the image features of each camera to obtain the features of the corresponding positions, then perform feature fusion, and finally predict the object detection results under the bird's-eye view based on the fused features. This solution uses sparse sampling to interact with image features and bird's-eye view space, which is more efficient. However, it is also not optimized for the fragmented characteristics of traffic scenes. Its single model will significantly reduce the performance of object detection when the number / order of cameras is adjusted, or the map / intersection position changes significantly, or the camera installation changes (position or angle). In addition, the sparse sampling method gradually constructs the correspondence between the image space and the bird's-eye view space through continuous iterations of the network training process, which is more difficult to learn and more dependent on camera parameters.

[0052] In response to the above problems, the present application provides a model training method, an object detection method, an apparatus, a device and a medium, and provides a set of multi-camera object detection solutions suitable for traffic scenes with good generalization. Through technical means such as fusion feature decoupling, prediction space unification, and image position encoding, multi-camera object detection can be applied to cameras of any number and order, and complex scenes of different sizes, positions, and resolutions. In addition, the dependence on camera calibration parameters (external parameters) is reduced, and the general ability of the algorithm is further improved. The present application adopts a front fusion solution, which directly fuses the image features of multiple cameras through the model, and the framework is simpler. The present application outputs a unified global object detection result under a bird's-eye view, which is more conducive to the development of subsequent applications. Since traffic scenes are usually large in scope, the sparse candidate sampling point method adopted in the present application is more efficient, and the computational complexity of the algorithm will not increase significantly with the increase in scope. Through the present application, different scenes can be adapted for multi-camera object detection.

[0053] The following first introduces a model training method provided by this application.

[0054] Among them, a model training method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is used to train an object detection model to be trained. In a specific application, the electronic device can be a mobile phone, a computer, etc., and the present application does not limit the specific form of the electronic device.

[0055] A model training method provided in an embodiment of the present application may include the following steps:

[0056] Acquire a sample image group; wherein the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size;

[0057] Input each sample image in the sample image group into the object detection model to be trained to obtain the prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, the feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position-aware feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image; the fusion module is used to fuse, for each candidate sampling point, the position-aware feature of the candidate sampling point with respect to each sample image to obtain the fusion feature of the candidate sampling point; the prediction module is used to generate a position offset and a category prediction result corresponding to each candidate sampling point based on the current position feature and the fusion feature of each candidate sampling point used to characterize the position, to obtain the prediction result; The position perception feature of a candidate sampling point with respect to any sample image is: a feature obtained by fusing a first feature content with respect to a reference point corresponding to the candidate sampling point in a feature map of the sample image and a second feature content with respect to a reference point corresponding to the candidate sampling point in a position coding feature of the feature map of the sample image, wherein the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from a sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, and the position coding feature of the feature map of any sample image is a feature obtained by at least performing position coding on the coordinates of each pixel in the feature map of the sample image after performing designated coordinate mapping; performing the designated coordinate mapping on each pixel in the feature map of any sample image includes: mapping each pixel in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to the second mapping relationship corresponding to the sample image;

[0058] Determining a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point;

[0059] When it is determined based on the model loss that the object detection model has not converged, the model parameters of the object recognition model are adjusted, and the process returns to the step of obtaining the sample image group.

[0060] In the model training method provided in the embodiment of the present application, the object detection model to be trained includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, so that prediction is performed on the sample image group through each module, and the model loss of the object detection model is determined according to the prediction result of the sample image group, the true value information and the position represented by the current position feature of each candidate sampling point, so as to adjust the model parameters of the model, that is, to train the model; and the position perception feature generated by the visual change module is: the first feature content of the reference point corresponding to the candidate sampling point in the feature map of the sample image and the second feature content of the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image are fused, so that the global correspondence between the image pixels and the bird's-eye view can be supplemented on the basis of the features of the feature map, that is, the correspondence between the image pixels and the physical space indicated by the sample bird's-eye view can be supplemented, and the physical space is a specified space size, so that the difference between different detection scenes can be reduced by unifying the prediction space, that is, unifying to the physical space of the specified space size. It can be seen that through the model training method provided in the present application, a model for multi-camera object detection in different scenes can be trained, that is, the trained model can adapt to different scenes.

[0061] A model training method provided in an embodiment of the present application is introduced below in conjunction with the accompanying drawings.

[0062] like Figure 1 As shown, a model training method provided in an embodiment of the present application may include the following steps:

[0063] S101: Acquire a sample image group;

[0064] The sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size;

[0065] The embodiment of the present application is to train an object detection model that can adapt to different scenes to perform multi-camera object detection. The object detection model can perform multi-camera object detection tasks for images collected by each camera set up in the same detection scene at the same time, and output the object detection results in the bird's-eye view of the detection scene. Therefore, a sample image group can be obtained first, and the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and the object position and object category truth information in the sample bird's-eye view of the sample detection scene at the same time; each sample image can be used for the training of the object detection model, and the truth information can be used for the calculation of the model loss of the subsequent object detection model.

[0066] Optionally, in order to avoid the differences in detection scenes, the different resolutions of the bird's-eye view of the detection scene, etc., the present application unifies the prediction space as an area with a fixed range at the center of the intersection, that is, the sample bird's-eye view is a bird's-eye view of a physical space of a specified spatial size. Among them, the pixel size and longitude and latitude range of the sample bird's-eye view can be used to calculate the mapping relationship between the pixel coordinate system of the sample bird's-eye view and the spatial coordinate system of the physical space (when the object detection model is applied to other detection scenes, it is necessary to switch the mapping relationship to the mapping relationship between the pixel coordinate system of the bird's-eye view in other detection scenes and the spatial coordinate system of the physical space), and the pixel coordinates of the sample bird's-eye view are converted into coordinates in the spatial coordinate system of the physical space using the true value of the pixel coordinates of the center point of the intersection. The subsequently trained object detection model can output object detection results for a physical space centered on the intersection and having a specified spatial size (such as 200 meters × 200 meters).

[0067] In addition, in order to avoid the result deviation caused by different intersection center point positions, the present application adopts a random offset strategy, that is, the intersection center point will randomly change within a smaller range to ensure the robustness of the trained object detection model (that is, the trained object detection model, for different intersection positions that change within this smaller range, the output object detection results are less different); the unified prediction space size in the present application (the size is the specified space size) can ensure that, during the training process of the object detection model, the output detection results are under a prediction space of uniform size (no matter how the bird's-eye view changes, the distribution of samples in the output prediction results and the detection range of candidate sampling points can remain stable, all under this prediction space), effectively improving the generalization ability of the object detection model.

[0068] S102: Inputting each sample image in the sample image group into the object detection model to be trained to obtain a prediction result of the sample image group;

[0069] The object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module. The feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image. The perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position-aware feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image. The fusion module is used to fuse, for each candidate sampling point, the position-aware features of the candidate sampling point with respect to each sample image to obtain a fused feature of the candidate sampling point. The prediction module is used to generate a position offset and a category prediction result corresponding to each candidate sampling point based on the current position feature and fused feature of each candidate sampling point used to characterize the position, to obtain the prediction result. The position-aware feature of any candidate sampling point with respect to any sample image is : a feature obtained by fusing a first feature content of a reference point corresponding to the candidate sampling point in the feature map of the sample image and a second feature content of a reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image, wherein the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from the sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, and the position coding feature of the feature map of any sample image is a feature obtained by mapping the coordinates of each pixel in the feature map of the sample image to a specified coordinate, at least after position coding; performing the specified coordinate mapping on each pixel in the feature map of any sample image includes: mapping each pixel in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to the second mapping relationship corresponding to the sample image;

[0070] In order to train an object detection model that can adapt to different scenes and perform multi-camera object detection, each sample image in the sample image group can be input into the object detection model to be trained to obtain the prediction result of the sample image group. Among them, the object detection model in the present application includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module; the feature extraction module can extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module can generate the position perception features of each candidate sampling point with respect to each sample image based on the feature map of each sample image for each candidate sampling point belonging to the sample bird's-eye view; the fusion module can fuse the position perception features of the candidate sampling point with respect to each sample image to obtain the fusion features of the candidate sampling point; thus, the prediction module generates the position offset and category prediction results corresponding to each candidate sampling point based on the current position features and fusion features used to characterize the position of each candidate sampling point.

[0071] It should be noted that the processing logic of the perspective transformation module can be understood as software logic, and the software logic can be written into the object detection model to realize the function of the perspective transformation module. The feature extraction module, fusion module and prediction module can be the feature extraction network, feature fusion network and prediction network (including regression network and classification network) of the object detection model itself.

[0072] In the present application, the interaction between image features and the bird's-eye view space is performed by means of candidate sampling points, and the position perception feature of any candidate sampling point with respect to any sample image is a feature obtained by fusing the first feature content with respect to the reference point corresponding to the candidate sampling point in the feature map of the sample image, and the second feature content with respect to the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image. That is, the present application not only considers the first feature content with respect to the reference point corresponding to the candidate sampling point in the feature map of the sample image (the first feature content can be understood as: the feature content obtained by mapping the candidate sampling point to the reference point in the feature map of the sample image through the first mapping relationship, and using the reference point to perform feature sampling on the feature map of the sample image), but also considers the second feature content with respect to the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image (the second feature content can be understood as: the feature content obtained by sampling the position coding feature of the feature map of the sample image using the reference point corresponding to the candidate sampling point), and the first feature content and the second feature content are fused to obtain the position perception feature of the candidate sampling point with respect to the sample image.

[0073] Furthermore, the reference point corresponding to any candidate sampling point is a mapping point of the candidate sampling point from the sample bird's-eye view to the sample image based on the first mapping relationship corresponding to the sample image; the position coding feature of the feature map of any sample image is the coordinate after each pixel in the feature map of the sample image is mapped to the specified coordinate, at least the feature obtained after position coding, when each pixel in the feature map of any sample image is mapped to the specified coordinate, each pixel in the feature map of the sample image can be mapped to the spatial coordinate system of the physical space indicated by the sample bird's-eye view based on the second mapping relationship corresponding to the sample image, thereby equivalent to using the mapping relationship between the feature map of a sample image under the sample detection scene and the physical space indicated by the sample bird's-eye view, obtaining the second feature content through the mapping relationship, and then combining the first feature content obtained through the first mapping relationship as the position perception feature of the candidate sampling point under the sample image, which can characterize the perceived position of the candidate sampling point with respect to the sample image in the sample bird's-eye view, and then combining the fusion feature formed by the position perception features of the candidate sampling point with respect to each sample image, and the current position feature of the candidate sampling point, the position offset and category prediction result corresponding to the candidate sampling point can be predicted.

[0074] In addition, the first feature content is the feature content of the reference point in the feature map of the sample image that maps the candidate sampling point to the feature map of the sample image through a first mapping relationship, which is equivalent to establishing a local connection between the candidate sampling point and the reference point in the feature map of the bird's-eye view space and the sample image. The first feature content can be understood as the feature of the local connection obtained by the interaction of the image features of the candidate sampling point and the sample image with the bird's-eye view space. The second feature content is the feature content of the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image (the reference point at this time is obtained by mapping the candidate sampling point to the position coding feature, and the mapping relationship used may be the same as or different from the first mapping relationship). The position coding feature of the feature map of the sample image is the coordinates obtained by mapping each pixel point in the feature map of the sample image to the spatial coordinate system of the physical space indicated by the sample bird's-eye view using the second mapping relationship. At least the feature obtained after position coding, through the second mapping relationship, is equivalent to establishing a connection between the bird's-eye view space and each pixel point in the feature map of the sample image, that is, establishing a global connection between the bird's-eye view space and the feature map of the sample image; therefore, the second feature content can be understood as the feature of the global connection obtained by the interaction of the image feature of the candidate sampling point, the sample image and the bird's-eye view space.

[0075] Since the first mapping relationship and the second mapping relationship can also be configured according to different cameras and different detection scenarios, after the object detection model is trained, it is only necessary to configure the first mapping relationship and the second mapping relationship according to the actual detection scenario to obtain an object detection model adapted to the actual detection scenario. It can be seen that this application can adapt to different scenarios for multi-camera object detection.

[0076] In one implementation, the present application may also generate a multi-scale feature map for each sample image to provide features of more scales for training the object detection model to be trained; the feature extraction of each sample image to obtain the feature map of each sample image includes:

[0077] Perform feature extraction on each sample image to generate feature maps of different resolutions of the sample image, and obtain a multi-scale feature map of the sample image; wherein the feature map of each resolution of the sample image is a feature map of one scale;

[0078] The perspective transformation module generates, for each candidate sampling point belonging to the sample bird's-eye view, a position-aware feature of the candidate sampling point with respect to each sample image based on a feature map of each sample image, including:

[0079] For each candidate sampling point belonging to the sample bird's-eye view image, based on the multi-scale feature map of each sample image, a position-aware feature of each scale of the candidate sampling point with respect to each sample image is generated.

[0080] In the present application, feature extraction can be performed on each sample image to generate feature maps of different resolutions of the sample image. The feature map of each resolution can be a feature map of one scale, so as to obtain a multi-scale feature map of the sample image. For the feature map of each scale of the sample image, the position perception feature of the candidate sampling point with respect to the scale of the sample image can be generated in the above manner, thereby obtaining the position perception feature of the candidate sampling point with respect to each scale of the sample image; then, the position perception feature of the candidate sampling point with respect to each scale of the sample image can be fused to obtain the position perception feature of the candidate sampling point with respect to the sample image.

[0081] It can be seen that the present application can also generate a multi-scale feature map for each sample image, and based on the multi-scale feature map of each sample image, generate the position-aware features of the candidate sampling point at each scale of each sample image, so as to provide features of more scales, train the object detection model to be trained, and obtain a more accurate object detection model.

[0082] Exemplarily, in one implementation, a method for determining a position coding feature of a feature map of any sample image includes:

[0083] Perform the specified coordinate mapping on each pixel point in the feature map of the sample image to obtain the target coordinate of each pixel point;

[0084] Normalize the target coordinates of each pixel;

[0085] The normalized target coordinates are position-encoded to obtain position-encoded features of the feature map of the sample image.

[0086] When generating the position coding features of the feature map of any sample image, each pixel point in the feature map of the sample image can be first mapped to a specified coordinate to obtain the target coordinates of each pixel point in the spatial coordinate system of the physical space indicated by the sample bird's-eye view; the target coordinates of each pixel point can be first normalized, and then the normalized target coordinates can be position-coded to obtain the position coding features of the feature map of the sample image.

[0087] Among them, if each sample image has a multi-scale feature map of the sample image, for the feature map of any scale of the sample image, the position coding feature of the scale of the sample image can be generated in the above-mentioned manner; of course, before the feature map of any scale is mapped to the specified coordinates, the feature map of the scale can be normalized first, so that the feature map of any scale can be adapted to the second mapping relationship for the specified coordinate mapping. Among them, the second mapping relationship is the mapping relationship of the pixel points in the feature map of the sample image to the spatial coordinate system of the physical space indicated by the sample bird's-eye view. The second mapping relationship is specifically: the mapping relationship between the image coordinate system of the sample image and the image coordinate system of the sample bird's-eye view, and the mapping relationship between the image coordinate system of the sample bird's-eye view and the spatial coordinate system of the physical space indicated by the sample bird's-eye view. Through these two mapping relationships, the image coordinate system of the sample image can be mapped to the spatial coordinate system of the physical space indicated by the sample bird's-eye view; therefore, in order to realize the mapping of each pixel point in the feature map of any scale of the sample image to the spatial coordinate system of the physical space indicated by the sample bird's-eye view, the feature map of the sample image of each scale can be normalized first; after normalization, in the feature map of the sample image of this scale, the coordinates of each pixel point are the coordinates under the normalized coordinate system, the normalized coordinate system can be understood as the image coordinate system of the sample image obtained after normalization, and the second mapping relationship can be understood as the coordinate mapping of the normalized image coordinate system and the spatial coordinate system of the physical space indicated by the sample bird's-eye view. Therefore, after the feature maps of each scale are normalized, the normalized coordinates of the same pixel points at different scales are the same as the coordinates of the same pixel points in the sample image in the normalized coordinate system. For example, if the size of the sample image is w×h, the size of the feature map of one scale is 0.5w×0.5h, the coordinates of a pixel point on the feature map of this scale are (0.25w, 0.25h), and after normalization, the coordinates of the pixel point are (0.5, 0.5); the coordinates of the point corresponding to the pixel point on the sample image are (0.5w, 0.5h). The coordinates of the point corresponding to the pixel point on the sample image in the normalized coordinate system are (0.5, 0.5); it can be seen that for any sample image, the same pixel point in the feature map of each scale of the sample image has the same coordinates as the corresponding point in the sample image in the normalized coordinate system, and the size of the scale does not affect the normalized coordinates of the same pixel point. Therefore, the second mapping relationship can be directly used to map the specified coordinates of each pixel point in the feature map of various scales of the sample image, so as to efficiently obtain the position coding features of the feature map of each scale of the sample image.

[0088] In the present application, in order to avoid the situation where different candidate sampling points correspond to the same object, the fusion module is further used to perform a specified interaction process on each candidate sampling point based on the fusion feature of each candidate sampling point; wherein the specified interaction process is used to make any candidate sampling point correspond to at most one object.

[0089] The fusion module may also perform specified interactive processing on each candidate sampling point so that any candidate sampling point corresponds to at most one object; illustratively, the interactive processing of the self-attention mechanism may be performed on each candidate sampling point to suppress the situation where multiple candidate sampling points correspond to the same object; thereby, for multiple candidate sampling points corresponding to the same object, the fusion features of one of the candidate sampling points are amplified, and the fusion features of the remaining candidate sampling points are suppressed. The specific implementation process may be similar to that of the prior art and will not be described in detail herein.

[0090] In addition, the fusion module can also fuse the obtained fusion features of the candidate sampling point with the current position features of the candidate sampling point used to characterize the position, and then input them into the prediction module for prediction, which is reasonable.

[0091] S103: determining a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point;

[0092] After the object detection model outputs the prediction results of the sample image group, the model loss of the object detection model can be determined based on the prediction results of the sample image group, the true value information, and the position represented by the current position characteristics of each candidate sampling point. Among them, the position prediction results and category prediction results corresponding to each candidate sampling point can be determined based on the prediction results of the sample image group and the position represented by the current position characteristics of each candidate sampling point, and then the model loss of the object detection model is calculated using the object position and object category in the true value information.

[0093] Specifically, determining the model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point includes:

[0094] For each candidate sampling point, according to the position offset of the candidate sampling point in the prediction result of the sample image group, the position represented by the current position feature of the candidate sampling point is adjusted to obtain a predicted position, and the category prediction result of the candidate sampling point in the prediction result of the sample image group is used as the category prediction result corresponding to the obtained predicted position;

[0095] Based on the obtained predicted positions and corresponding category prediction results, and the true value information, the model loss of the object detection model is determined.

[0096] For each candidate sampling point, the position represented by the current position feature of the candidate sampling point can be adjusted according to the position offset of the candidate sampling point in the prediction result of the sample image group. The adjusted position is the sum of the position represented by the current position feature of the candidate sampling point and the position offset (which may include size and direction). A predicted position is obtained. The predicted position can be understood as the position prediction result of the candidate sampling point. Combined with the category prediction result of the candidate sampling point in the prediction result of the sample image group, the predicted position and the corresponding category prediction result are obtained.

[0097] After obtaining each predicted position and the corresponding category prediction result, the model loss of the object detection model can be determined in combination with the true value information. Exemplarily, the model loss of the object detection model can be calculated using a predetermined loss function. The specific implementation method can be similar to the prior art and will not be elaborated here.

[0098] It should be noted that the prediction result of the sample image group includes the position offset and the category prediction result corresponding to each candidate sampling point. The position offset of any candidate sampling point can be understood as the offset between the true position prediction result of the candidate sampling point and the position represented by the current position feature of the candidate sampling point. According to the position offset of the candidate sampling point and the position represented by the current position feature of the candidate sampling point, the true position prediction result of the candidate sampling point can be obtained; the category prediction result corresponding to each candidate sampling point can be understood as the predicted category of an object corresponding to the candidate sampling point, such as trees, stones, vehicles, etc. Of course, since the number of candidate sampling points may be greater than the number of objects in the sample bird's-eye view (for example, the number of candidate sampling points may be 900, and the number of objects in the sample bird's-eye view may be 20), the category prediction result corresponding to the candidate sampling point may also be empty, that is, the candidate sampling point does not correspond to an object, and the present application does not limit this.

[0099] In addition, the category prediction result corresponding to each candidate sampling point can be the category with the highest confidence among the categories to which the candidate sampling point belongs. For example, when the object detection model predicts the category of a candidate sampling point, the confidence that the candidate sampling point belongs to the category of vehicle is 80%, the confidence that it belongs to stone is 10%, and the confidence that it belongs to tree is 10%. Then the predicted category prediction result of the candidate sampling point is vehicle, and the confidence is 80%.

[0100] S104: When it is determined based on the model loss that the object detection model has not converged, the model parameters of the object recognition model are adjusted, and the process returns to the step of obtaining the sample image group.

[0101] After determining the model loss of the object detection model, you can judge whether the object detection model has converged based on the model loss. If it has converged, such as: the model loss is less than the preset threshold, then the trained object detection model is directly obtained; if it has not converged, such as: the model loss is not less than the preset threshold, then you need to continue training the object detection model, and you can first adjust the model parameters of the object detection model, such as: adjusting the parameters of the feature extraction module, perspective transformation module, fusion module and prediction module, and re-acquire the sample image group for training.

[0102] In addition, before returning to the step of obtaining the sample image group, the method further includes:

[0103] The position features for representing the positions of the candidate sampling points are adjusted.

[0104] In the present application, before the object detection model is trained, each candidate sampling point may have an initial position. After a round of object detection model training is completed, the position features of each candidate sampling point used to characterize the position may be adjusted, that is, the initial position of each candidate sampling point is adjusted (for example: based on the position offset output by the object detection model and the position features of the candidate sampling point before the object detection model training, the position features of the candidate sampling point are adjusted; in each round of object detection model training process, the position features of the candidate sampling point may be adjusted according to the position offset output by the training round and the position features of the candidate sampling point before the training round), and then the object detection model is continued to be trained to obtain more accurate positions of each candidate sampling point.

[0105] Of course, the position features of each candidate sampling point may not be adjusted. In each round of object detection model training, the positions represented by the position features of each candidate sampling point are their corresponding initial positions, and this application does not limit this.

[0106] In the technical solution of the present application, the operations of acquiring, storing, using, processing, transmitting, providing and disclosing the sample images, true value information, feature maps, position coding features, first mapping relationships, second mapping relationships, etc., are all carried out with the user's authorization.

[0107] In the model training method provided in the embodiment of the present application, the object detection model to be trained includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, so that prediction is performed on the sample image group through each module, and the model loss of the object detection model is determined according to the prediction result of the sample image group, the true value information and the position represented by the current position feature of each candidate sampling point, so as to adjust the model parameters of the model, that is, to train the model; and the position perception feature generated by the visual change module is: the first feature content of the reference point corresponding to the candidate sampling point in the feature map of the sample image and the second feature content of the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image are fused, so that the global correspondence between the image pixels and the bird's-eye view can be supplemented on the basis of the features of the feature map, that is, the correspondence between the image pixels and the physical space indicated by the sample bird's-eye view can be supplemented, and the physical space is a specified space size, so that the difference between different detection scenes can be reduced by unifying the prediction space, that is, unifying to the physical space of the specified space size. It can be seen that through the model training method provided in the present application, a model for multi-camera object detection in different scenes can be trained, that is, the trained model can adapt to different scenes.

[0108] Optionally, in another embodiment of the present application, for each candidate sampling point, fusing the position perception features of the candidate sampling point with respect to each sample image to obtain the fused features of the candidate sampling point includes:

[0109] For each candidate sampling point, determining a fusion weight of the position-aware feature of the candidate sampling point with respect to each sample image;

[0110] Based on the obtained fusion weight, each position perception feature of the candidate sampling point with respect to each sample image is fused to obtain a fusion feature of the candidate sampling point;

[0111] The step of determining the fusion weight of the position-aware feature of the candidate sampling point with respect to each sample image includes:

[0112] For the position-aware feature of the candidate sampling point with respect to any sample image, input the target image feature corresponding to the position-aware feature into the fully connected layer to obtain the fully connected layer encoding result corresponding to the position-aware feature; wherein the target image feature is a feature map used to generate the first feature in the position-aware feature;

[0113] The fully connected layer encoding result corresponding to the position-aware feature is normalized to obtain a fusion weight of the position-aware feature of any sample image.

[0114] In the present application, for each candidate sampling point, when the position perception features of the candidate sampling point with respect to each sample image are fused (i.e., the position perception features of the candidate sampling point under multiple cameras are fused), the fusion weight of the position perception features of the candidate sampling point with respect to each sample image can be first determined, and then based on the obtained fusion weight, the position perception features of the candidate sampling point with respect to each sample image are fused to obtain the fused features of the candidate sampling point.

[0115] Among them, in order to decouple the relationship between the fusion weight and the number and order of cameras, the present application, for the position perception feature of the candidate sampling point with respect to any sample image, inputs the target image feature corresponding to the position perception feature into the fully connected layer for encoding, obtains the fully connected layer encoding result corresponding to the position perception feature, and then normalizes the fully connected layer encoding result to obtain the fusion weight of the position perception feature of any sample image; the target image feature is the feature map used to produce the first feature in the position perception feature, that is, the feature map of the sample image, and the fusion weight of the position perception feature of the candidate sampling point with respect to each sample image is obtained after the image feature of the sample image is encoded by the fully connected layer and normalized, that is, the corresponding fusion weight of the position perception feature of the candidate sampling point with respect to the sample image can be generated according to the image feature of the sample image, and the generation process of the fusion weight has nothing to do with the number and order of the camera itself. Therefore, when fusing the position perception feature of the candidate sampling point with respect to each sample image, the present application can be applicable to detection scenarios with any number of cameras and camera order; the object detection model obtained by subsequent training can also be applicable to detection scenarios with any number of cameras and camera order.

[0116] Based on the above model training method, the present application embodiment provides an object detection method, such as Figure 2 As shown, the method includes:

[0117] S201: Acquire a target image group; wherein the target image group includes target images captured by various cameras set up in a target detection scene at the same time;

[0118] S202: Input each target image in the target image group into a designated object detection model to obtain a prediction result of the target image group; wherein the designated object detection model is a model obtained by adjusting the first mapping relationship corresponding to each sample image to the first designated relationship corresponding to each target image, and adjusting the second mapping relationship corresponding to each sample image used for designated coordinate mapping to the second designated relationship corresponding to each target image, with respect to the object detection model trained based on the model training method; wherein the first designated relationship corresponding to any target image is used to map any candidate sampling point from a bird's-eye view of a target detection scene to the target image, and the second designated relationship corresponding to any target image is used to map each pixel point in a feature map of the target image to a spatial coordinate system of a physical space indicated by the bird's-eye view of the target detection scene; the bird's-eye view of the target detection scene is a bird's-eye view of a physical space of a designated spatial size;

[0119] S203: For each candidate sampling point, a predicted position is generated according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and the category prediction result of the candidate sampling point in the prediction result of the target image group is used as the obtained category prediction result corresponding to the predicted position.

[0120] It can be understood that the target images in the target image group are collected by each camera set up in the target detection scene at the same time, and the sample images contained in the sample image group are collected by each camera set up in the sample detection scene at the same time. The present application only needs to use the object detection model trained by the sample image group, adjust the first mapping relationship in the object detection model to the first specified relationship, and adjust the second mapping relationship to the second specified relationship, and control the bird's-eye view of the target detection scene to be a bird's-eye view of the physical space of the specified space size (that is, the size of the unified prediction space is the specified space size), so as to realize object detection of multiple cameras in the target detection scene; and according to the position offset of the candidate sampling point in the prediction result of the target image group, and the position represented by the current position feature of the candidate sampling point, when generating a predicted position, the position represented by the current position feature of the candidate sampling point can be directly added to the position offset of the candidate sampling point in the prediction result of the target image group to generate a predicted position, and the present application does not limit this.

[0121] The object detection method provided in the embodiment of the present application adjusts the first mapping relationship corresponding to each sample image to the first specified relationship corresponding to each target image, and adjusts the second mapping relationship corresponding to each sample image used for the specified coordinate mapping to the second specified relationship corresponding to each target image, based on the object detection model trained by the model training method. The first specified relationship corresponding to any target image is used to map any candidate sampling point from the bird's-eye view of the target detection scene to the target image, and the second specified relationship corresponding to any target image is used to map each pixel point in the feature map of the target image to the spatial coordinate system of the physical space indicated by the bird's-eye view of the target detection scene; and the bird's-eye view of the target detection scene is a bird's-eye view of the physical space of the specified spatial size; thereby, the adjusted specified object detection model can adapt to the target detection scene for object detection. The target image group formed by the target images collected by each camera set up in the target detection scene at the same time can be obtained and input into the designated object detection model to obtain the prediction result of the target image group; then, for each candidate sampling point, a predicted position is generated according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and the category prediction result of the candidate sampling point in the prediction result of the target image group is used as the category prediction result corresponding to the predicted position, that is, the object detection result under the bird's-eye view of the target detection scene is obtained. The object detection model obtained by the model training method of the present application only needs to adjust the first mapping relationship to the first specified relationship and the second mapping relationship to the second specified relationship according to the target detection scene, so as to obtain the designated object detection model adapted to the target detection scene; the designated object detection model supplements the correspondence between the image pixels and the physical space indicated by the bird's-eye view of the target detection scene, and the physical space is of a designated space size, so that the differences between different detection scenes are reduced by unifying the prediction space, that is, unifying to the physical space of the designated space size. It can be seen that this application can adapt to different scenes to perform multi-camera object detection.

[0122] The following is a detailed introduction to an object detection method provided by the present application in conjunction with a specific embodiment.

[0123] This application provides a multi-camera object detection method that can cope with the fragmented characteristics of traffic scenes, and has the following characteristics: the object detection model trained by the model training method can be applied to detection scenes with any number and order of cameras, and can maintain good performance even when factors such as the resolution of the bird's-eye view of the detection scene, the size or position of the intersection, and the camera setup (camera external parameters such as position or angle) change. The overall process of the solution of this application is as follows: Figure 3 As shown:

[0124] It can be seen that the specified object detection model of this application includes:

[0125] Feature extraction module: Through a shared backbone network, the image features of the target images captured by each camera (camera 1, camera 2, ... camera n) set up in the target detection scenario at the same time are extracted respectively to obtain the feature map of each target image. The feature map contains high-level semantic information and is the basis for subsequent object detection.

[0126] Perspective transformation module: maps a series of candidate sampling points of the bird's-eye view of the target detection scene to the corresponding position sampling image features of the feature map of each target image through the internal and external parameters / projection matrix of each camera (the sampled image features are the first feature content). Among them, the perspective transformation module can establish a local connection between the feature map of each target image and the bird's-eye view of the target detection scene to obtain the first feature content; at the same time, the global connection between the feature map of the target image and the bird's-eye view of the target detection scene is established through the position coding feature of the feature map of the target image to obtain the second feature content; that is, the position perception features (including the first feature content and the second feature content) of each candidate sampling point in the bird's-eye view of the target detection scene with respect to each target image are generated.

[0127] Fusion module: The position perception features of each candidate sampling point generated by the perspective transformation module with respect to each target image are fused in a way that decouples the relationship between the fusion weight and the number and order of cameras (i.e., the position perception features of the candidate sampling point from multiple cameras are fused); at the same time, the candidate sampling points are subjected to designated interaction processing to suppress the situation where multiple candidate sampling points are located on the same object. Of course, the candidate sampling point also contains the position feature of its current position, which can also be fused with the fusion feature of the candidate sampling point and input into the prediction module for object detection.

[0128] Prediction module: Based on the features of the candidate sampling points after fusion interaction in the fusion module (including the fusion features and position features of the candidate sampling points), the regression network and the classification network are used to predict the position prediction results of each candidate sampling point in a unified prediction space (that is, the position offset, and the true prediction position can be determined based on the position offset and the position features of the candidate sampling points) and the category prediction results to achieve multi-camera object detection; of course, in order to adapt to different scenes and bird's-eye views of different sizes, the present application can output the prediction results of a unified prediction space.

[0129] The specific implementation process of each module in this application is as follows:

[0130] Feature extraction module: The convolutional neural network commonly used in deep learning is used to extract high-level semantic features of the target image, which usually includes a backbone network and a feature pyramid network. The backbone network includes but is not limited to VGG (Visual Geometry Group Network), ResNet (residual neural network), DarkNet (dark network), DenseNet (densely connected network), etc. The network parameters are shared among multiple cameras, that is, the target images of multiple cameras are synchronously input into the backbone network with the same network parameters for feature extraction. When training the backbone network, the model parameters trained on a large-scale general data set can be used for fine-tuning to improve the feature extraction capability and convergence speed of the feature extraction module.

[0131] The feature pyramid network usually uses FPN (Feature Pyramid Network), which can generate feature maps of different resolutions for the target image of a single camera, facilitating multi-scale object detection. After passing through the feature extraction module, the target image group will obtain a set of multi-camera multi-scale image features (that is, a multi-scale feature map is generated for the target image of each camera).

[0132] Perspective transformation module: The input of the perspective transformation module is the multi-camera and multi-scale image features output by the feature extraction module, the preset candidate sampling points, and the internal and external parameters / projection matrix of each camera. The output is the position perception features of each candidate sampling point with respect to each sample image.

[0133] Each candidate sampling point is a set of n×d (n is the number, d is the vector dimension, such as 256) pre-set fixed number of learnable feature vectors, whose dimensions can be changed to n×2 / n×3 (2D coordinates x, y or 3D coordinates x, y, z on the bird's-eye view) through a simple fully connected layer. Each candidate sampling point can be considered as a possible object in the bird's-eye view space. Each candidate sampling point samples the position-aware features of multiple cameras and continuously interacts with other candidate sampling points (suppressing the situation where multiple candidate sampling points are located on the same object), and finally predicts the category and position of the object.

[0134] Camera internal and external parameters / projection matrix: The camera's internal and external parameters or the projection matrix of the image and the bird's-eye view are the bridges for constructing the mapping relationship between the feature space (the feature map of the image) and the output space (the bird's-eye view). The camera's internal parameters include fx, fy (the focal lengths in the x and y directions, respectively), cx, cy (the coordinates of the camera's optical center in the image coordinate system), and in some cases, distortion parameters. The camera's projection matrix can usually be expressed as a 3×3 matrix K: K= , which can be used to realize the conversion between the camera coordinate system and the image coordinate system. The camera extrinsic parameters can describe the position and posture of the camera in the world coordinate system, mainly including the rotation matrix R and the translation vector T. Usually the two can be expressed as a 4×4 matrix P: , which can be used to realize the conversion between the camera coordinate system and the world coordinate system. Combined with the camera's internal and external parameters, the world coordinate system can be mapped to the image coordinate system. The complete mapping formula is: , (x, y, z) are the coordinates in the world coordinate system, and (u, v) are the coordinates in the image coordinate system. The camera intrinsic parameters can be determined when the camera leaves the factory, and the camera extrinsic parameters are related to the camera installation and need to be calibrated on site. Of course, the correspondence between the image space (the camera's image coordinate system) and the bird's-eye view space (the bird's-eye view image coordinate system) can also be directly calibrated to obtain the corresponding projection matrix.

[0135] Through the internal and external parameters / projection matrix of each camera, the preset candidate sampling points on the bird's-eye view can be mapped to the corresponding position (i.e., reference point) of the target image of each camera, and the first feature content corresponding to the candidate sampling point can be obtained by bilinear interpolation sampling. This method constructs a local connection between the image space and the bird's-eye view space. With the continuous iteration of the object detection model training, the object detection model gradually learns the position correspondence between the image space and the bird's-eye view space, so that regression prediction can be performed in the bird's-eye view space based on the image features (first feature content) corresponding to the candidate sampling points under multiple cameras. However, in this way, the image features themselves do not perceive the position of the prediction results, which increases the difficulty of the task. When the object detection model is migrated to a new scene application, the correspondence between the image space and the bird's-eye view space changes, and the object detection model cannot directly rely on the new correspondence to build the connection between the image space and the bird's-eye view space (the projection matrix can only provide point-to-point mapping, and cannot directly give a global position mapping relationship, so it is difficult to directly guide the motion interaction of the candidate sampling points on the bird's-eye view according to the image features). This solution is essentially overfitting the camera parameters.

[0136] Therefore, in order to reduce the model's dependence on the projection matrix, this application generates a position coding feature in the physical space indicated by the corresponding bird's-eye view for each pixel of the image feature, and generates a position coding feature for each candidate sampling point, thereby increasing the global perception of each candidate sampling point. The generation process of the position coding feature is as follows Figure 4As shown: where m and n are the length and width dimensions of the feature map of the target image, so that the 2-dimensional representation of the x, y pixel coordinates (px, py) of any position on the feature map of the target image, m×n×2 (image pixels) represents the pixel coordinates of any position on the feature map of the target image. v-map represents the projection matrix between the image space and the bird's-eye view space (because the feature map is normalized, the projection matrix between the feature map space and the bird's-eye view space is the same as the projection matrix between the image space and the bird's-eye view space), and the physical relationship refers to the mapping relationship between the image coordinate system of the bird's-eye view and the spatial coordinate system of the physical space (it can be understood as the correspondence between the distance between the pixels in the bird's-eye view and the distance in the corresponding physical space; for example: the correspondence between the distance between two pixels and the actual distance in meters).

[0137] For each pixel coordinate on the feature map of the target image, the coordinates of each pixel coordinate on the feature map in the spatial coordinate system of the physical space indicated by the bird's-eye view are mapped from the camera's image coordinate system -> the bird's-eye view's image coordinate system -> the physical space's spatial coordinate system, i.e., m×n×2 (map meters); the obtained coordinates are then normalized and sent to the position encoding network for position encoding; each candidate sampling point can be sampled from the position encoding result to obtain the position encoding feature m×n×256 position encoding feature (second feature content) corresponding to the candidate sampling point, and then added to the original image feature m×n×256 image feature (first feature content) of the candidate sampling point in the feature map to obtain the m×n×256 position perception feature of the candidate sampling point. In this way, the global correspondence between image features and physical space is explicitly constructed, and the connection between feature space (feature map of the target image) and prediction space (predicted physical space) has been established without the need for a learning process. After obtaining the position-aware features of the candidate sampling points (i.e., knowing the positions of the candidate sampling points in the physical space indicated by the bird's-eye view), the candidate sampling points can be guided to move in the physical space indicated by the bird's-eye view (i.e., specifying interactive processing to suppress the situation where multiple candidate sampling points are located to the same object).

[0138] Fusion module: After each candidate sampling point is mapped to the corresponding position (reference point) of the multi-camera image, it can be sampled to obtain multi-view and multi-scale position perception features; the fusion module can fuse the multi-camera and multi-scale position perception features of each candidate sampling point to obtain the fusion features of each candidate sampling point. Existing fusion methods include Figure 5a As shown, the fusion method in this application is as follows Figure 5b As shown, n is the number of candidate sampling points, 256 is the feature dimension, fc represents the fully connected layer, cams is the number of cameras, and levels is the number of scales.

[0139] The mainstream fusion method of feature overlap in the prior art is weighted fusion (the fused features are not the location-aware features in this application), such as Figure 5a As shown in the figure, for each candidate sampling point, the feature vector n×256 of the candidate sampling point is used to convert its dimension into cams×levels, that is, n×number of cameras×number of scales, through the fully connected layer fc, as the fusion weight of the features of n candidate sampling points on multiple cameras (the number of cameras is cams) and multiple scales (the number of scales is levels). Among them, cams is a preset fixed value, which limits the output dimension of the fully connected layer, so this method cannot support any number of camera inputs. In addition, the feature vector of the candidate sampling point is essentially a multi-camera fusion feature. The fusion weights of each single camera are generated from the fused results. The fusion module must couple the relationship between multiple cameras in learning, including the arrangement order, etc. Although the robustness of the model to the order can be enhanced by the training strategy of randomly swapping the camera order, the model still essentially learns the camera order. For example, if cams is 3, the feature vectors of the candidate sampling points are converted into w1, w2, and w3 (the scale number is not considered at this time) through the fully connected layer, and used as the fusion weights of camera 1, camera 2, and camera 3 respectively. When the camera order changes to camera 1, camera 3, and camera 2, w1, w2, and w3 will be used as the fusion weights of camera 1, camera 3, and camera 2 respectively, resulting in the inability of the fusion method to adapt to the change in the camera order; and when the number of cameras changes, the feature vectors of the candidate sampling points need to be converted into the dimension of the changed number of cameras through the fully connected layer, resulting in the inability of the fusion method to adapt to the change in the number of cameras.

[0140] Therefore, this application adjusts the fusion method to Figure 5b The structure shown in FIG. 1 uses the feature maps of each camera obtained by back-projecting the candidate sampling point. The multi-scale feature map of each camera, n×levels×256, is encoded by the fully connected layer fc and then normalized to generate the fusion weight of the candidate sampling point on the position perception feature of the camera, n×levels×1, thereby avoiding the limitation of the number of cameras. Among them, the multi-scale feature map of each camera (n×number of cameras×number of scales×256) is normalized by the softmax function to generate the fusion weight of n candidate sampling points on the multi-camera multi-scale position perception features (n×number of cameras×number of scales×1). At the same time, the fusion weight is essentially determined by the strength of the feature map of each camera, and the relationship between different cameras is not coupled. Changing the order of cameras has no effect on the result of the generated fusion weight. Through such structural adjustment, the relationship with the number and order of cameras is decoupled from the root, which is more universal and can be easily applied to different traffic intersection scenes. Finally, the multi-camera multi-scale fusion features (fusion features) obtained by sampling each candidate sampling point are added to its own high-dimensional features (position features) to obtain new high-dimensional features of the candidate sampling point.

[0141] After the fusion features of the candidate sampling points are obtained, they are combined with the position features of the candidate sampling points and input into the prediction module for object detection.

[0142] In addition, candidate sampling points will also interact with each other, usually through a transformer-structured multi-head attention network (a network with a self-attention mechanism), which can increase the global perception of candidate sampling points and reduce the problem of multiple candidate sampling points detecting the same object.

[0143] Prediction module: Finally, based on the location perception features of the candidate sampling points and the location features of the candidate sampling points, the prediction module predicts the location offset and category confidence of each candidate sampling point through the regression network and the classification network, and obtains the location offset and category prediction results of each candidate sampling point. When regressing the position, the bird's-eye view map is usually used as the prediction space of the position. However, due to the limitations of the actual situation, different intersections have different sizes, and there are obvious differences in the distribution of vehicle targets between large intersections and small intersections; at the same time, more differences will be introduced when the bird's-eye view map is cropped, such as the location of the intersection in the map, the range of map cropping, and the scale of the map and the physical space. These differences will significantly affect the distribution of objects in the bird's-eye view map space, resulting in the inability of candidate sampling points trained on the sample detection scene to effectively perceive objects in the target detection scene. For example, for the same number of candidate sampling points, their distribution is relatively dense for small intersection maps, and the object detection model will learn that the distance between every two candidate sampling points is about 1-2 vehicles; however, in larger intersection maps, the distribution of candidate sampling points becomes sparse, and the distance between two candidate sampling points is about 5-10 vehicles. This huge difference will inevitably lead to abnormalities in object detection.

[0144] Therefore, if Figure 6As shown, in order to avoid the huge differences between different maps, the application unifies the prediction space as an area with a fixed range at the center of the intersection (the area formed by each candidate sampling point in the figure, that is, the above-mentioned specified space size), such as an area of ​​200 meters × 200 meters. Specifically, the pixel size and longitude and latitude range of the map are used to calculate the mapping relationship between pixels and meters (that is, the mapping relationship between the pixel coordinate system of the map and the spatial coordinate system of the physical space). Using the true value of the pixel coordinate of the center point of the intersection, the pixel coordinates are converted into meter coordinates (that is, the pixel coordinate system is converted into the spatial coordinate system of the physical space), and the object detection model directly outputs the object detection results within the range of 200 meters × 200 meters with the intersection as the center. In addition, since the intersection scene does not have a self-vehicle center like the vehicle side, the intersection center is set artificially. In order to avoid the result deviation caused by different center point positions, the application adopts a random offset strategy, that is, when the object detection model is trained, the center point of the intersection will randomly change within a smaller range, thereby ensuring the robustness of the object detection model. By unifying the size of the prediction space, the object detection model ensures the uniform size of the output space during the learning process. No matter how the map changes, the distribution of its samples and the perception range of the candidate sampling points can remain stable, effectively improving the generalization ability of the object detection model.

[0145] The object detection method provided by this application can train an object detection model that can adapt to detection scenarios with any number of cameras and camera order. This application is a front-fusion solution with end-to-end training and prediction. It directly inputs multi-camera images to output the global detection results of the bird's-eye view (object detection results of multiple cameras). The overall structure is simple and efficient, without complex logical fusion steps. This application uses sparse candidate sampling point sampling to construct the connection between image features and the bird's-eye view space (global connection and local connection). The increase in the perception range will not bring more calculations.

[0146] In this application, the perspective conversion module adds position encoding features to the multi-camera image features, explicitly constructs the connection between the feature map and the bird's-eye view space, reduces the dependence of the object detection model on camera parameters, and improves the generalization ability in different scenarios. The fusion weights are decoupled in the fusion module, and the relationship between the weights and the number and order of cameras is decoupled, so that the model can be applied to traffic scenes with any number of cameras and any order. The prediction space is unified from the bird's-eye view map to a fixed range (meters) at the intersection center. Combined with the center point offset training strategy, the object detection model can be applied to scenes of different sizes / intersection locations / map resolutions. This application is specially designed for the fragmented characteristics of traffic roadside erection scenes. The object detection model structure can support any number and order of cameras, while reducing the dependence on camera parameters. When migrating the object detection model to a new scene, it only needs to adjust its mapping relationship. The object detection model can maintain good algorithm performance and better generalization ability. This application can greatly reduce the data cost, computing power cost and camera calibration cost of the object detection solution for multiple cameras in traffic scenes.

[0147] Based on the above method embodiment, the present application also provides a model training device, such as Figure 7 As shown, the device comprises:

[0148] The first acquisition module 710 is used to acquire a sample image group; wherein the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size;

[0149] The first input module 720 is used to input each sample image in the sample image group into the object detection model to be trained to obtain the prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, the feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position perception feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image; the fusion module is used to fuse, for each candidate sampling point, the position perception features of the candidate sampling point with respect to each sample image to obtain the fusion feature of the candidate sampling point; the prediction module is used to generate the position offset and category prediction result corresponding to each candidate sampling point based on the current position feature and fusion feature used to characterize the position of each candidate sampling point, to obtain the Prediction result; the position perception feature of any candidate sampling point with respect to any sample image is: a feature obtained by fusing a first feature content with respect to a reference point corresponding to the candidate sampling point in a feature map of the sample image and a second feature content with respect to a reference point corresponding to the candidate sampling point in a position coding feature of the feature map of the sample image, wherein the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from a sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, and the position coding feature of the feature map of any sample image is a coordinate after each pixel point in the feature map of the sample image is mapped to a specified coordinate, at least a feature obtained through position coding; the specified coordinate mapping of each pixel point in the feature map of any sample image includes: mapping each pixel point in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to a second mapping relationship corresponding to the sample image;

[0150] A determination module 730, configured to determine a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point;

[0151] The adjustment module 740 is used to adjust the model parameters of the object recognition model when it is determined based on the model loss that the object detection model has not converged, and return to the step of obtaining the sample image group.

[0152] In the model training device provided in the embodiment of the present application, the object detection model to be trained includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, so that prediction is performed on the sample image group through each module, and the model loss of the object detection model is determined according to the prediction result of the sample image group, the true value information and the position represented by the current position feature of each candidate sampling point, so as to adjust the model parameters of the model, that is, to train the model; and the position perception feature generated by the visual change module is: the first feature content of the reference point corresponding to the candidate sampling point in the feature map of the sample image and the second feature content of the reference point corresponding to the candidate sampling point in the position coding feature of the feature map of the sample image are fused. In this way, the global correspondence between the image pixels and the bird's-eye view can be supplemented on the basis of the features of the feature map, that is, the correspondence between the image pixels and the physical space indicated by the sample bird's-eye view can be supplemented, and the physical space is a specified space size, so that the difference between different detection scenes can be reduced by unifying the prediction space, that is, unifying to the physical space of the specified space size. It can be seen that through the model training method provided in the present application, a model for multi-camera object detection in different scenes can be trained, that is, the trained model can adapt to different scenes.

[0153] Based on the above method embodiment, the present application also provides an object detection device, such as Figure 8 As shown, the device comprises:

[0154] The second acquisition module 810 is used to acquire a target image group; wherein the target image group includes target images captured by each camera set up in the target detection scene at the same time;

[0155] The second input module 820 is used to input each target image in the target image group into a designated object detection model to obtain a prediction result of the target image group; wherein the designated object detection model is a model obtained by adjusting the first mapping relationship corresponding to each sample image to the first designated relationship corresponding to each target image, and adjusting the second mapping relationship corresponding to each sample image used for the designated coordinate mapping to the second designated relationship corresponding to each target image for the object detection model trained based on the model training method; wherein the first designated relationship corresponding to any target image is used to map any candidate sampling point from the bird's-eye view of the target detection scene to the target image, and the second designated relationship corresponding to any target image is used to map each pixel point in the feature map of the target image to the spatial coordinate system of the physical space indicated by the bird's-eye view of the target detection scene; the bird's-eye view of the target detection scene is a bird's-eye view of a physical space of a specified space size;

[0156] The generating module 830 is used to generate a predicted position for each candidate sampling point according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and use the category prediction result of the candidate sampling point in the prediction result of the target image group as the obtained category prediction result corresponding to the predicted position.

[0157] The object detection device provided in the embodiment of the present application adjusts the first mapping relationship corresponding to each sample image to the first specified relationship corresponding to each target image, and adjusts the second mapping relationship corresponding to each sample image used for the specified coordinate mapping to the second specified relationship corresponding to each target image, based on the object detection model trained by the model training method. The first specified relationship corresponding to any target image is used to map any candidate sampling point from the bird's-eye view of the target detection scene to the target image, and the second specified relationship corresponding to any target image is used to map each pixel point in the feature map of the target image to the spatial coordinate system of the physical space indicated by the bird's-eye view of the target detection scene; and the bird's-eye view of the target detection scene is a bird's-eye view of the physical space of the specified spatial size; thereby, the adjusted specified object detection model can adapt to the target detection scene to perform object detection. The target image group formed by the target images collected by each camera set up in the target detection scene at the same time can be obtained and input into the designated object detection model to obtain the prediction result of the target image group; then, for each candidate sampling point, a predicted position is generated according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and the category prediction result of the candidate sampling point in the prediction result of the target image group is used as the category prediction result corresponding to the predicted position, that is, the object detection result under the bird's-eye view of the target detection scene is obtained. The object detection model obtained by the model training method of the present application only needs to adjust the first mapping relationship to the first specified relationship and the second mapping relationship to the second specified relationship according to the target detection scene, so as to obtain the designated object detection model adapted to the target detection scene; the designated object detection model supplements the correspondence between the image pixels and the physical space indicated by the bird's-eye view of the target detection scene, and the physical space is of a designated space size, so that the differences between different detection scenes are reduced by unifying the prediction space, that is, unifying to the physical space of the designated space size. It can be seen that this application can adapt to different scenes to perform multi-camera object detection.

[0158] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0159] Memory 901, used for storing computer programs;

[0160] The processor 902 is used to implement any of the model training methods or object detection methods when executing the program stored in the memory 901.

[0161] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 communicate with each other via the communication bus.

[0162] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0163] The communication interface is used for communication between the above electronic device and other devices.

[0164] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0165] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0166] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods or object detection methods are implemented.

[0167] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method or object detection method in the above-mentioned embodiments.

[0168] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0169] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0170] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample image group; wherein the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size; Input each sample image in the sample image group into the object detection model to be trained to obtain the prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, the feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position-aware feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image; the fusion module is used to fuse, for each candidate sampling point, the position-aware feature of the candidate sampling point with respect to each sample image to obtain the fusion feature of the candidate sampling point; the prediction module is used to generate a position offset and a category prediction result corresponding to each candidate sampling point based on the current position feature and the fusion feature of each candidate sampling point used to characterize the position, to obtain the prediction result; The position perception feature of a candidate sampling point with respect to any sample image is: a feature obtained by fusing a first feature content with respect to a reference point corresponding to the candidate sampling point in a feature map of the sample image and a second feature content with respect to a reference point corresponding to the candidate sampling point in a position coding feature of the feature map of the sample image, wherein the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from a sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, and the position coding feature of the feature map of any sample image is a feature obtained by at least performing position coding on the coordinates of each pixel in the feature map of the sample image after performing designated coordinate mapping; performing the designated coordinate mapping on each pixel in the feature map of any sample image includes: mapping each pixel in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to the second mapping relationship corresponding to the sample image; Determining a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point; When it is determined based on the model loss that the object detection model has not converged, the model parameters of the object recognition model are adjusted, and the process returns to the step of obtaining the sample image group.

2. The method according to claim 1, characterized in that: The determining of the model loss of the object detection model according to the prediction result of the sample image group, the true value information and the position represented by the current position feature of each candidate sampling point comprises: For each candidate sampling point, according to the position offset of the candidate sampling point in the prediction result of the sample image group, the position represented by the current position feature of the candidate sampling point is adjusted to obtain a predicted position, and the category prediction result of the candidate sampling point in the prediction result of the sample image group is used as the category prediction result corresponding to the obtained predicted position; Based on the obtained predicted positions and corresponding category prediction results, and the true value information, the model loss of the object detection model is determined.

3. The method according to claim 1, characterized in that: Before returning to the step of obtaining the sample image group, the method further includes: The position features for representing the positions of the candidate sampling points are adjusted.

4. The method according to any one of claims 1 to 3, characterized in that: The method for determining the position encoding features of the feature map of any sample image includes: Perform the specified coordinate mapping on each pixel point in the feature map of the sample image to obtain the target coordinate of each pixel point; Normalize the target coordinates of each pixel; The normalized target coordinates are position-encoded to obtain position-encoded features of the feature map of the sample image.

5. The method according to any one of claims 1 to 3, characterized in that: The step of fusing, for each candidate sampling point, the position perception features of the candidate sampling point with respect to each sample image to obtain the fused features of the candidate sampling point includes: For each candidate sampling point, determining a fusion weight of the position-aware feature of the candidate sampling point with respect to each sample image; Based on the obtained fusion weight, each position perception feature of the candidate sampling point with respect to each sample image is fused to obtain a fusion feature of the candidate sampling point; The step of determining the fusion weight of the position-aware feature of the candidate sampling point with respect to each sample image includes: For the position-aware feature of the candidate sampling point with respect to any sample image, input the target image feature corresponding to the position-aware feature into the fully connected layer to obtain the fully connected layer encoding result corresponding to the position-aware feature; wherein the target image feature is a feature map used to generate the first feature in the position-aware feature; The fully connected layer encoding result corresponding to the position-aware feature is normalized to obtain a fusion weight of the position-aware feature of any sample image.

6. The method according to any one of claims 1 to 3, characterized in that: The step of extracting features from each sample image to obtain a feature map of each sample image includes: Perform feature extraction on each sample image to generate feature maps of different resolutions of the sample image, and obtain a multi-scale feature map of the sample image; wherein the feature map of each resolution of the sample image is a feature map of one scale; The perspective transformation module generates, for each candidate sampling point belonging to the sample bird's-eye view, a position-aware feature of the candidate sampling point with respect to each sample image based on a feature map of each sample image, including: For each candidate sampling point belonging to the sample bird's-eye view image, based on the multi-scale feature map of each sample image, a position-aware feature of each scale of the candidate sampling point with respect to each sample image is generated.

7. The method according to any one of claims 1 to 3, characterized in that: The fusion module is further used to perform a specified interaction process on each candidate sampling point based on the fusion feature of each candidate sampling point; wherein the specified interaction process is used to make any candidate sampling point correspond to at most one object.

8. An object detection method, characterized in that: The method comprises: Acquire a target image group; wherein the target image group includes target images captured by each camera set up in the target detection scene at the same time; Input each target image in the target image group into a designated object detection model to obtain a prediction result of the target image group; wherein the designated object detection model is an object detection model trained based on the method described in any one of claims 1 to 7, and the first mapping relationship corresponding to each sample image is adjusted to the first designated relationship corresponding to each target image, and the second mapping relationship corresponding to each sample image used for the designated coordinate mapping is adjusted to the second designated relationship corresponding to each target image; wherein the first designated relationship corresponding to any target image is used to map any candidate sampling point from a bird's-eye view of the target detection scene to the target image, and the second designated relationship corresponding to any target image is used to map each pixel point in the feature map of the target image to the spatial coordinate system of the physical space indicated by the bird's-eye view of the target detection scene; the bird's-eye view of the target detection scene is a bird's-eye view of a physical space of a specified spatial size; For each candidate sampling point, a predicted position is generated according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and the category prediction result of the candidate sampling point in the prediction result of the target image group is used as the obtained category prediction result corresponding to the predicted position.

9. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire a sample image group; wherein the sample image group includes sample images collected by each camera set up in the sample detection scene at the same time, and true value information, wherein the true value information is the object position and object category in the sample bird's-eye view of the sample detection scene at the same time, and the sample bird's-eye view is a bird's-eye view of a physical space of a specified space size; The first input module is used to input each sample image in the sample image group into the object detection model to be trained to obtain the prediction result of the sample image group; wherein the object detection model includes a feature extraction module, a perspective transformation module, a fusion module and a prediction module, the feature extraction module is used to extract features from each sample image to obtain a feature map of each sample image; the perspective transformation module is used to generate, for each candidate sampling point belonging to the sample bird's-eye view, a position perception feature of the candidate sampling point with respect to each sample image based on the feature map of each sample image; the fusion module is used to fuse, for each candidate sampling point, the position perception features of the candidate sampling point with respect to each sample image to obtain the fusion feature of the candidate sampling point; the prediction module is used to generate, based on the current position feature and fusion feature of each candidate sampling point used to characterize the position, a position offset and a category prediction result corresponding to each candidate sampling point to obtain the prediction result. measurement result; the position perception feature of any candidate sampling point with respect to any sample image is: a feature obtained by fusing a first feature content with respect to a reference point corresponding to the candidate sampling point in a feature map of the sample image and a second feature content with respect to a reference point corresponding to the candidate sampling point in a position coding feature of the feature map of the sample image, the reference point corresponding to the candidate sampling point is a mapping point obtained by mapping the candidate sampling point from a sample bird's-eye view to the sample image based on a first mapping relationship corresponding to the sample image, the position coding feature of the feature map of any sample image is a coordinate after each pixel point in the feature map of the sample image is mapped to a specified coordinate, at least a feature obtained by position coding; the mapping of each pixel point in the feature map of any sample image to the specified coordinate includes: mapping each pixel point in the feature map of the sample image to a spatial coordinate system of a physical space indicated by the sample bird's-eye view according to a second mapping relationship corresponding to the sample image; A determination module, configured to determine a model loss of the object detection model according to the prediction result of the sample image group, the true value information, and the position represented by the current position feature of each candidate sampling point; An adjustment module is used to adjust the model parameters of the object recognition model when it is determined based on the model loss that the object detection model has not converged, and return to the step of obtaining the sample image group.

10. An object detection device, characterized in that: The device comprises: A second acquisition module is used to acquire a target image group; wherein the target image group includes target images captured by each camera set up in the target detection scene at the same time; A second input module is used to input each target image in the target image group into a designated object detection model to obtain a prediction result of the target image group; wherein the designated object detection model is an object detection model trained based on the method described in any one of claims 1 to 7, and the first mapping relationship corresponding to each sample image is adjusted to the first designated relationship corresponding to each target image, and the second mapping relationship corresponding to each sample image used for the designated coordinate mapping is adjusted to the second designated relationship corresponding to each target image; wherein the first designated relationship corresponding to any target image is used to map any candidate sampling point from a bird's-eye view of a target detection scene to the target image, and the second designated relationship corresponding to any target image is used to map each pixel point in a feature map of the target image to a spatial coordinate system of a physical space indicated by the bird's-eye view of the target detection scene; the bird's-eye view of the target detection scene is a bird's-eye view of a physical space of a specified spatial size; A generating module is used to generate a predicted position for each candidate sampling point according to the position offset of the candidate sampling point in the prediction result of the target image group and the position represented by the current position feature of the candidate sampling point, and use the category prediction result of the candidate sampling point in the prediction result of the target image group as the obtained category prediction result corresponding to the predicted position.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1 to 7, or the method described in claim 8, when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 or the method according to claim 8 is implemented.

Images