Target detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a target detection method and device, electronic equipment and storage medium. The present disclosure relates to the field of automatic driving, in particular to the field of image recognition of automatic driving. The specific implementation scheme is: in response to a start operation of a vehicle, a camera is controlled to capture an image of a road where the vehicle is located at the current time, and the image is an initial image; in a case where a virtual camera is turned on, the initial image is pixel mapped based on a homography matrix to obtain a virtual camera image; and the virtual camera image is input into a first detection model trained in advance to detect the virtual camera image to obtain a first target detection result. According to the disclosure of the present scheme, the system error caused by the difference in camera parameters can be reduced, and the accuracy of target detection can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving, and more particularly to the field of image recognition for autonomous driving, specifically to a target detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Significant differences exist in camera mounting positions and intrinsic parameters across different vehicle models and even within the same model. These variations can cause the same 3D anchor points to map differently onto images from different camera perspectives, introducing systematic errors. These errors can lead to inconsistent predictions from computer vision models under varying conditions, particularly in road structure perception tasks, impacting the model's accuracy and stability. Summary of the Invention

[0003] This disclosure provides a target detection method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a target detection method is provided, comprising:

[0005] In response to the vehicle's start-up operation, the camera is controlled to capture an image of the road where the vehicle is located at the current moment; the image is the initial image.

[0006] With the virtual camera enabled, the initial image is pixel-mapped based on the homography matrix to obtain the virtual camera image;

[0007] The virtual camera image is input into the pre-trained first detection model to perform detection processing on the virtual camera image, and the first target detection result is obtained.

[0008] According to another aspect of this disclosure, a target detection apparatus is provided, comprising:

[0009] The image acquisition module is used to acquire the image of the road where the vehicle is located at the current moment, captured by the camera. The image is the initial image.

[0010] The image processing module is used to perform pixel mapping on the initial image based on the homography matrix to obtain the virtual camera image when the virtual camera is turned on.

[0011] The first detection module is used to input the virtual camera image into the pre-trained first detection model, perform detection processing on the virtual camera image, and obtain the first target detection result.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0018] The solution disclosed herein can reduce systematic errors caused by differences in camera parameters and improve the accuracy of target detection.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a schematic diagram of camera position according to an embodiment of the present disclosure;

[0022] Figure 2 This is a schematic flowchart of a target detection method according to an embodiment of the present disclosure;

[0023] Figure 3 This is a flowchart illustrating the process of determining a homography matrix according to an embodiment of this disclosure;

[0024] Figure 4 This is a flowchart of virtual camera processing according to an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram comparing an initial image and a virtual camera image according to an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of the target detection device according to an embodiment of the present disclosure;

[0027] Figure 7 This is a schematic diagram of a scenario based on an embodiment of the target detection method of this disclosure;

[0028] Figure 8This is a schematic diagram of the structure of an electronic device used to implement the target detection method of the embodiments of this disclosure. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0031] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0032] Before introducing the technical solutions of the embodiments of this disclosure, the technical terms that may be used in this disclosure will be further explained:

[0033] Homography Matrix: In image processing and computer vision, the homography matrix is ​​used to describe the projection mapping relationship between two planes, and is often used in scenarios such as image correction and perspective transformation.

[0034] Object detection is a fundamental task in the field of computer vision, which aims to identify target objects in images or videos and determine their location and category.

[0035] In related technologies, cameras are used in autonomous driving systems to perceive the surrounding environment and generate views of the road structure. However, the camera mounting positions and intrinsic parameters vary significantly across different vehicle models and their different stages. This difference can cause the same 3D anchor point in the same scene to be mapped to different positions on the image from different camera perspectives, leading to systematic errors. These errors can cause inconsistent predictions from computer vision models under different conditions, particularly in road structure perception tasks, affecting the model's accuracy and stability. Therefore, eliminating systematic errors caused by differences in camera parameters under various camera configurations, ensuring consistency in the image-to-3D position mapping under different configurations, and improving the model's generalization ability are key to ensuring the accuracy and stability of environmental perception in autonomous driving systems.

[0036] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, this disclosure proposes a target detection method that uses virtual camera parameters to process the initial image. This method can uniformly process image data under different vehicle models and camera configurations, eliminate system errors caused by differences in camera parameters, and thus ensure the consistency of the image-to-3D position mapping relationship under various configuration conditions, making the environmental perception in the autonomous driving system more accurate and stable.

[0037] According to embodiments of this disclosure, Figure 1 This is a schematic diagram of camera position according to an embodiment of the present disclosure, such as... Figure 1 As shown, vehicle 100 is equipped with multiple cameras (such as cameras 101-106), which are installed in different positions on the vehicle (e.g., at the front, left front, right front, rear, left rear, and right rear). Cameras in different positions can capture images from different angles. It is understandable that... Figure 1 The vehicles shown are merely illustrative, and the number and installation location of the cameras are not limited thereto. This disclosure does not limit the vehicle type or model, nor does it limit the number or installation location of the cameras.

[0038] This disclosure provides a target detection method. Figure 2 This is a schematic flowchart of a target detection method according to an embodiment of the present disclosure. This target detection method can be applied to a target detection device. The target detection device is located in an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, mobile phones, tablets, and vehicle terminals. In some possible implementations, the target detection method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 2As shown, the target detection method includes:

[0039] S201. In response to the vehicle's start-up operation, control the camera to capture an image of the road where the vehicle is located at the current moment. This image is the initial image.

[0040] S202. With the virtual camera enabled, perform pixel mapping on the initial image based on the homography matrix to obtain the virtual camera image;

[0041] S203. Input the virtual camera image into the pre-trained first detection model, perform detection processing on the virtual camera image, and obtain the first target detection result.

[0042] In this embodiment of the disclosure, the road may include, but is not limited to, urban roads: including standard lanes, complex intersections, pedestrian crossings, traffic signs, traffic lights, etc.; highways: including multi-lane highways, toll stations, ramps, diversion zones and viaducts, etc.; rural roads: including narrow roads, unpaved roads and roads in agricultural and forestry areas, etc.; tunnels and underpasses; slopes and mountain roads, etc.

[0043] In this embodiment of the disclosure, a virtual camera refers to a camera model implemented through software simulation, which is used to uniformly process images taken by different vehicles. The virtual camera can perform pixel mapping on initial images from different cameras based on a preset homography matrix, thereby converting these images into standardized images (i.e., virtual camera images) generated under a unified viewpoint and parameters.

[0044] In this embodiment of the disclosure, a homography matrix is ​​a mathematical tool used to describe the perspective transformation relationship between two planes. Specifically, a homography matrix can geometrically transform images from different viewpoints, so that image points that were originally in different positions can correspond to the same plane after the transformation.

[0045] In this embodiment, the first detection model may include, but is not limited to, a perception model, an object detection model, etc. The above is merely illustrative and is not intended to limit all possible types of the first detection model; an exhaustive list is not provided here. It should be noted that this disclosure does not limit the training method or model structure of the first detection model. The first detection model may be based on deep learning algorithms and can automatically identify target objects (such as pedestrians, vehicles, traffic signs, etc.) in an image.

[0046] In some implementations, controlling the camera to capture images of the road where the vehicle is located includes: in response to the driver's or autonomous driving program's activation of the vehicle, triggering the camera to capture images of the area around the vehicle. Thus, the vehicle can periodically capture images of the road where it is located while in motion.

[0047] In some implementations, pixel mapping of the initial image based on the homography matrix includes: First, obtaining intrinsic and extrinsic parameters of the camera from the vehicle's sensor system. Intrinsic parameters typically include parameters such as focal length and optical center, while extrinsic parameters include the camera's rotation matrix and translation vector relative to the vehicle. Based on these intrinsic and extrinsic parameters, a ground hypothesis is applied to calculate the homography matrix. This homography matrix maps ground points in the road image from the original camera's viewpoint to the virtual camera's viewpoint, ensuring consistency across different camera parameters by projecting 3D anchor points under the ground hypothesis onto the 2D image plane. Next, using the calculated homography matrix, the coordinates of each pixel in the initial image are mapped to the standard viewpoint coordinate system of the virtual camera. Specifically, multiplying the 2D coordinates of each pixel in the initial image by the homography matrix yields new coordinates representing the pixel's position in the virtual camera's viewpoint. The transformed pixels are reconstructed in the virtual camera image, forming a more standardized image compared to the original image, which can be used as input for subsequent detection models.

[0048] For example, suppose that after the vehicle starts, the system automatically activates the front-facing camera and begins capturing images of the road ahead. At this point, the image captured by the camera includes a pedestrian crossing a zebra crossing. The virtual camera obtains the current intrinsic parameters (e.g., focal length 35mm, optical center at the image center) and extrinsic parameters (e.g., camera rotation angle relative to the vehicle is 5 degrees, translation vector is 0.5 meters) from the sensor system. Based on these parameters, the system calculates a homography matrix using ground assumptions. Suppose a pedestrian's position in the original image is (1000, 500) pixels; through the homography matrix transformation, this point is mapped to the virtual camera's standard viewpoint, and the new position might become (980, 520). Thus, even if the vehicle's camera parameters differ, the pedestrian's position in the virtual camera's viewpoint remains consistent.

[0049] In some implementations, once the vehicle starts and preset conditions are met (such as seatbelts being fastened and doors being closed), the system automatically triggers the camera to begin operating. The camera captures the road environment in front of the vehicle at the current moment, generates a high-resolution image as the initial image, and stores it in the system memory for subsequent processing.

[0050] In some implementations, the system checks whether the virtual camera function is enabled. If enabled, a homography matrix is ​​calculated based on preset virtual camera parameters (such as position, angle, focal length, etc.). Using the calculated homography matrix, perspective transformation and pixel remapping are performed on the initial image to simulate the image seen from the virtual camera's perspective, i.e., the virtual camera image. This step aims to optimize the object detection effect by adjusting the perspective, especially for object recognition in specific scenes or road conditions.

[0051] In some implementations, a virtual camera image is input into a pre-trained first detection model to obtain a first target detection result, which includes, but is not limited to, information such as the target's category, location (bounding box), and confidence level.

[0052] The technical solution of this disclosure can unify images captured by different vehicles and cameras to the same viewpoint through a virtual camera and homography matrix, reducing deviations caused by parameter differences and ensuring image consistency. Simultaneously, since virtual camera images reduce image distortion and redundant information compared to the initial images, inputting the virtual camera images generated by the virtual camera into a pre-trained model can improve the accuracy of object detection. Using the above image processing method enables the same detection model to be applicable to object detection under different camera configurations, and also helps to improve the model's versatility and stability.

[0053] In some embodiments, before performing pixel mapping on the initial image based on the homography matrix, the target detection method may further include: fine-tuning the initial image to obtain an optimized initial image.

[0054] Here, fine-tuning includes, but is not limited to, cropping, scaling, or other adjustments and optimizations of the initial image to improve image quality or correct certain problems in the image, thereby obtaining an optimized image that is more suitable for subsequent processing. These adjustments and optimizations may include, but are not limited to, translation, rotation, brightness and contrast adjustments, and noise reduction.

[0055] In this way, through fine-tuning, the quality of the initial image used to generate the virtual camera image is improved, eliminating problems such as noise, uneven lighting, and geometric distortion, generating a clearer and more accurate optimized image, thereby improving the image quality of the virtual camera and thus helping to improve the accuracy of target detection.

[0056] In some embodiments, pixel mapping of the initial image based on the homography matrix to obtain a virtual camera image may include:

[0057] The initial image is pixel-mapped based on the homography matrix to obtain the image from the virtual camera's perspective.

[0058] Normalize the image from the virtual camera's perspective to obtain the virtual camera image.

[0059] In this embodiment, pixel mapping refers to using a homography matrix to map pixels in the initial image to new positions from the perspective of a virtual camera.

[0060] In some implementations, after obtaining the homography matrix, each pixel in the initial image can be mapped using the homography matrix to transform it to a new position in the virtual camera's view. This step is typically achieved through matrix multiplication, where the homogeneous coordinates of each pixel are multiplied by the homography matrix to obtain its new coordinates in the virtual camera's view. Since the mapped coordinates may not be integers, interpolation (such as bilinear or bicubic interpolation) is usually required to obtain the final pixel value.

[0061] In some implementations, the mapped image may exhibit uneven pixel value distribution or exceed normal ranges due to factors such as perspective transformation. Therefore, it is necessary to normalize the mapped image to ensure that its pixel values ​​are within a reasonable range. Normalization operations may include adjusting the image's brightness and contrast, or scaling the pixel values ​​to a specific range.

[0062] Suppose we have an initial street scene image and want to simulate a virtual camera view from above. We can follow these steps: Select multiple sets of corresponding points (such as building corners, road intersections, etc.) in both the initial image and the desired virtual camera view; calculate the homography matrix using these points; apply the homography matrix to perform pixel mapping on the initial image to obtain the image from the virtual camera view. Normalize the mapped image, adjusting parameters such as brightness and contrast to obtain the final virtual camera image. In this way, we can transform an ordinary street scene image into a virtual camera image with a bird's-eye view effect using homography matrix and pixel mapping techniques.

[0063] Thus, through the mapping of the homography matrix, the initial image can be easily converted into an image from any virtual camera perspective, enabling free transformation of the viewpoint. Combined with normalization operations, the visual effect of the mapped image can be further improved, making it more suitable for practical applications and thus further enhancing the quality of the generated virtual camera images.

[0064] In some embodiments, such as Figure 3 As shown, the process for determining the homography matrix may include:

[0065] S301. Determine the source projection point of the source image based on the camera parameters;

[0066] S302. Determine the standard projection points of the target image based on the virtual camera parameters;

[0067] S303. Perform Gaussian elimination based on the source projection point and the standard projection point to obtain the homography matrix.

[0068] In some implementations, determining the source projection point of the source image based on camera parameters includes: when the vehicle starts, obtaining the intrinsic and extrinsic parameters of the current camera by reading the camera's configuration file or through online calibration; then, using these camera parameters, mapping the three-dimensional point in the real world to a two-dimensional coordinate position on the source image plane; this two-dimensional coordinate point is the source projection point, representing the position of the three-dimensional point in the source image.

[0069] In some implementations, determining the standard projection point of the target image based on virtual camera parameters includes: acquiring the intrinsic and extrinsic parameters of the current vehicle camera from the vehicle's sensor system; performing standard virtual camera calibration based on the acquired intrinsic and extrinsic parameters; firstly, fine-tuning the rotation angle of the target image to fully align the first coordinate system of the target image with the second coordinate system of the source image; secondly, fine-tuning the translation of the target image to match the position of the target image with that in the source image; thirdly, fine-tuning the intrinsic parameters of the target image to achieve a preset image quality; and finally, determining the standard projection point based on the 3D anchor points of the target image after rotation angle fine-tuning, translation fine-tuning, and intrinsic parameter fine-tuning.

[0070] In some implementations, the homography matrix is ​​obtained by solving Gaussian elimination based on the source projection points and the standard projection points. This includes: first, obtaining several known source projection points from the source image (these points can be two-dimensional coordinates in the source image); simultaneously, obtaining the corresponding points of these points in the standard image, i.e., the standard projection points (these points can also be two-dimensional coordinates, usually corresponding to points in the virtual camera image); then, establishing a system of linear equations about the homography matrix based on the correspondence between the source projection points and the standard projection points; each pair of source projection points and standard projection points provides two equations, and multiple points form multiple equations; solving the system of equations using Gaussian elimination to obtain the parameters of the homography matrix; transforming the system of equations into upper triangular form through row operations using Gaussian elimination, and then solving it back by substitution to finally determine the homography matrix; after solving for all unknowns, the homography matrix is ​​constructed to map the points in the source image to their corresponding positions in the standard image.

[0071] Thus, by accurately calculating the source projection points and standard projection points, and using Gaussian elimination to solve the homography matrix, systematic errors caused by different camera parameters are effectively eliminated, thereby improving the accuracy of image processing after virtual camera calibration. Simultaneously, the adoption of a standard virtual camera calibration method allows the camera parameters of different vehicles to be uniformly mapped to a standard model, reducing errors caused by differences in vehicle camera configurations and improving the adaptability and versatility of the target detection model.

[0072] In some embodiments, determining the standard projection point of the target image based on virtual camera parameters may include:

[0073] The target image is rotated and finely adjusted so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0074] The target image is translated and finely adjusted so that its position matches that of the source image.

[0075] The intrinsic parameters of the target image are fine-tuned to ensure that the imaging quality of the target image reaches the preset value.

[0076] Standard projection points are determined based on the 3D anchor points of the target image after fine-tuning of rotation angle, translation, and intrinsic parameters.

[0077] In this embodiment of the disclosure, virtual camera parameters are parameters used to simulate camera shooting effects, including focal length, optical center position, camera position, and orientation.

[0078] In this embodiment of the disclosure, the standard projection point is the precise location of a specific region or feature point in the target image from the perspective of a virtual camera.

[0079] In this embodiment of the disclosure, 3D anchor points are pixel positions in the image that correspond to fixed points in the real world, used to establish a correspondence between the image and the real world.

[0080] In this embodiment of the disclosure, the preset value can be set or adjusted according to system requirements.

[0081] In some implementations, fine-tuning the rotation angle of the target image may include: analyzing the angular deviation between the target image and the source image, and using an image rotation algorithm (such as affine transformation, perspective transformation, etc.) to fine-tune the target image until their coordinate systems are aligned. This ensures complete alignment of the first coordinate system of the target image with the second coordinate system of the source image.

[0082] In some implementations, fine-tuning the target image by translation includes: adjusting the target image using a translation transformation algorithm based on the positional deviation between the target image and the source image. This typically involves moving the image in the x and y directions. In this way, the position of the target image in image space can be adjusted to match the corresponding position in the source image.

[0083] In some implementations, the target image undergoes intrinsic parameter fine-tuning, which typically involves adjusting camera intrinsic parameters (such as parameters in the camera matrix K). Optimal intrinsic parameter settings can be determined using camera calibration techniques or image quality assessment algorithms. Imaging parameters of the target image, such as focal length and optical center position, are adjusted to achieve preset image quality.

[0084] In some implementations, determining standard projection points based on the 3D anchor points of the target image after rotation angle fine-tuning, translation fine-tuning, and intrinsic parameter fine-tuning includes: First, determining the corresponding position of each 3D anchor point in the target image from the virtual camera's perspective based on the correspondence between the fine-tuned target image and the source image. Then, using these corresponding positions as references, calculating the standard projection points. The standard projection point is the precise position of a specific region or feature point in the image from the virtual camera's perspective. Thus, using the fine-tuned target image, combined with its 3D anchor points (i.e., the pixel positions in the image corresponding to fixed points in the real world), the standard projection points from the virtual camera's perspective are determined.

[0085] In an autonomous driving system, it's necessary to align road images (target images) captured by an onboard camera with a high-precision map (source image). First, the initial image is fine-tuned by rotation to align it with the map's coordinate system. Then, translational adjustments ensure the road image's accurate position on the map. Next, intrinsic parameter adjustments optimize the image quality. Finally, lane markings in the road image (as 3D anchor points) are used to determine the standard projection point from the virtual camera's perspective, thus providing the vehicle with precise navigation information.

[0086] Thus, by fine-tuning the rotation angle and translation, the alignment accuracy between the target image and the source image can be significantly improved, reducing errors caused by image offset or rotation. Intrinsic parameter fine-tuning allows adjustment of the target image's imaging parameters to better meet practical application requirements, improving image clarity and accuracy. Based on the fine-tuned target image and 3D anchor points, a standard projection point can be determined from the virtual camera's perspective, achieving precise projection and providing accurate positional information for subsequent image processing and analysis.

[0087] In some embodiments, fine-tuning the rotation angle of the target image to fully align the first coordinate system of the target image with the second coordinate system of the source image may include:

[0088] Determine the angle by which the target image needs to be rotated relative to the source image based on the target points;

[0089] Rotate the target image by the corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0090] In this embodiment of the disclosure, the target point should be a feature point in the image that is easy to identify and has a relatively fixed position, such as a corner point or an edge intersection point.

[0091] In this embodiment of the disclosure, the target image is an image whose rotation angle needs to be finely adjusted to align with the source image. The source image is an image used as a reference or benchmark, to which the target image needs to be aligned.

[0092] In this embodiment of the disclosure, the rotation angle is the degree by which the target image needs to be rotated relative to the source image, used to achieve complete alignment of the coordinate system.

[0093] In this embodiment of the disclosure, rotation transformation is an image processing operation that changes the direction and angle of an image by rotating the pixels in the image.

[0094] In some implementations, the rotation angle is determined based on the target point, including:

[0095] Select a set of corresponding target points in both the target image and the source image.

[0096] Using these corresponding target points, the rotation angle of the target image relative to the source image is calculated through image matching algorithms (such as feature point matching, template matching, etc.). This typically involves calculating the rotation matrix between the two sets of points.

[0097] Extract the rotation angle value from the rotation matrix. This angle value represents the degree by which the target image needs to be rotated around a center point (usually the image center or a specified point) in order to align with the coordinate system of the source image.

[0098] In some implementations, rotating the target image by a corresponding angle includes:

[0099] Use rotation transformation functions from image processing software or programming libraries to rotate the target image by a calculated rotation angle. The rotation transformation can be two-dimensional or three-dimensional, depending on the image's dimensions and the required alignment precision.

[0100] During rotation, it is necessary to ensure that the rotation center coincides with the center of the image or the specified alignment point to ensure that the rotated image is also correctly positioned.

[0101] After rotation, further fine-tuning of the image position or other image processing operations (such as scaling, cropping, etc.) may be required to optimize the alignment between the target image and the source image.

[0102] Thus, by accurately calculating the rotation angle and applying rotation transformations, the alignment accuracy between the target image and the source image can be significantly improved, reducing errors caused by image rotation. The aligned target image and source image are visually more consistent, facilitating subsequent image analysis, processing, and fusion operations. In applications requiring image alignment (such as virtual reality, augmented reality, and image stitching), accurately aligned images can significantly enhance application performance and user experience.

[0103] In some embodiments, fine-tuning the translation of the target image to match its position with that in the source image may include:

[0104] Determine at least one reference point based on the source image and the target image;

[0105] The translation vector is obtained based on the positional difference of at least one reference point in the source and target images;

[0106] The target image is moved based on the translation vector so that its position matches the position in the source image.

[0107] In this embodiment of the disclosure, the reference point is a prominent feature point used in image processing to determine the relative position or angular differences between images.

[0108] In this embodiment of the disclosure, the rotation angle is the rotation angle required to align the target image with the source image in the coordinate system.

[0109] In this embodiment of the disclosure, the image rotation algorithm is an algorithm that changes the pixel positions in an image through mathematical transformations to achieve image rotation.

[0110] In some implementations, at least one reference point is selected in both the source and target images. These reference points should have distinctive features, such as road intersections, building corners, road signs, etc., to ensure accurate identification in different images. In the context of autonomous driving, these reference points may be lane line intersections, traffic signs, or roadside facilities at known locations.

[0111] In some implementations, calculating the rotation angle includes: using image processing techniques (such as feature detection and matching algorithms) to determine the coordinates of each reference point in the source and target images; calculating the relative positional relationships between these reference points, particularly the angular differences. This typically involves vector operations, where the lines connecting the reference points are treated as vectors, and the rotation angle is determined by calculating the angle between these vectors.

[0112] In autonomous driving, it may also be necessary to consider the vehicle's direction of travel and the camera's orientation in order to accurately calculate the required rotation angle.

[0113] In some implementations, rotating a target image includes: after determining the rotation angle, rotating the target image using an image rotation algorithm. These algorithms typically involve transforming the image matrix, where the coordinates of each pixel are adjusted accordingly based on the rotation angle. During rotation, different interpolation methods can be selected to handle pixel gaps or overlaps that may occur after rotation, such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0114] In some implementations, after rotation, the positions of the reference points in the target and source images are checked again to verify the alignment effect. If a deviation is found, the rotation angle can be fine-tuned and the above steps repeated until a satisfactory alignment effect is achieved.

[0115] Suppose an autonomous vehicle captures two consecutive images of the road while driving: a source image (taken from the vehicle's current position) and a target image (taken from a later position). To ensure the vehicle can accurately perceive and locate obstacles and road signs, the target image needs to be rotated and fine-tuned to align with the coordinate system of the source image. In this example, a road intersection or a fixed roadside structure can be chosen as a reference point. By comparing the positional relationships of these reference points in the two images, the required rotation angle is calculated. Then, an image rotation algorithm is used to rotate the target image until the reference points are aligned in both images. This completes the alignment of the target image with the source image in the coordinate system.

[0116] In this way, by ensuring the accurate alignment of the coordinate system in the image, the autonomous driving system can more accurately identify and understand objects and scenes in the surrounding environment; the alignment of the coordinate system helps to reduce misjudgments and misoperations caused by image distortion or positional deviation, enhances the accuracy of decision-making, and thus improves the driving safety and stability of autonomous vehicles.

[0117] In some embodiments, fine-tuning the intrinsic parameters of the target image to achieve a preset image quality may include:

[0118] Obtain the camera's intrinsic parameters;

[0119] The camera's internal parameters should be recalibrated or adjusted according to the internal parameter requirements of different application scenarios.

[0120] In this embodiment of the disclosure, camera intrinsic parameters are a set of parameters describing the internal characteristics of the camera, including focal length, principal point coordinates, and lens distortion parameters. These parameters determine how the camera maps points in the three-dimensional world onto the two-dimensional image plane.

[0121] In this embodiment of the disclosure, calibration is the process of obtaining camera intrinsic parameters through specific methods and tools. The calibration process typically requires the use of a calibration board and image processing algorithms.

[0122] In this embodiment, distortion is an image deformation phenomenon caused by imperfections in the camera lens. Distortion includes types such as radial distortion and tangential distortion, which affect the accuracy and usability of the image.

[0123] In some implementations, obtaining the camera's intrinsic parameters includes acquiring these parameters through a camera calibration process. Calibration typically uses a specific calibration board (such as a checkerboard calibration board) and image processing algorithms to calculate the camera's intrinsic parameters by capturing images of the calibration board at different positions and angles.

[0124] Here, the camera's intrinsic parameters are parameters that describe the camera's internal characteristics, including focal length (fx, fy), principal point coordinates (cx, cy), and lens distortion parameters.

[0125] In some implementations, the requirements of the application scenario for intrinsic parameters are analyzed, including:

[0126] Autonomous driving systems face complex and ever-changing environments, and different application scenarios have different requirements for camera imaging quality. For example, when driving on a highway, the camera needs to be able to clearly capture distant vehicles and road signs; while in the complex traffic environment of the city, the camera needs to be able to accurately identify pedestrians and obstacles nearby.

[0127] Based on the requirements of the application scenario, analyze whether the current camera intrinsic parameters meet the requirements. If not, the intrinsic parameters need to be recalibrated or adjusted.

[0128] In some implementations, recalibrating or adjusting the camera's intrinsic parameters includes:

[0129] If the camera's intrinsic parameters do not meet the requirements of the application scenario, new intrinsic parameters can be obtained through recalibration. The recalibration process is similar to the initial calibration, but it may require the use of a calibration board and calibration method more suitable for the current application scenario.

[0130] Alternatively, existing intrinsic parameters can be fine-tuned using software algorithms. For example, based on imaging problems discovered in practical applications (such as image distortion, etc.), the focal length, principal point coordinates, or distortion parameters can be adjusted to optimize image quality.

[0131] After recalibrating or adjusting the internal parameters, the adjustment effect can be verified through actual testing. Testing can be conducted in simulated or real-world road environments, and the image quality before and after adjustment can be compared to evaluate whether the adjustment meets the requirements.

[0132] In the development of autonomous vehicles, intrinsic parameter calibration of the cameras is necessary to ensure they can accurately capture road information and identify obstacles. For example, a wide-angle camera used in a certain autonomous vehicle exhibited significant barrel distortion in its images after initial calibration. To optimize image quality and reduce the impact of distortion on the autonomous driving system, the camera was recalibrated using specialized calibration tools and algorithms, and intrinsic parameters such as focal length and distortion parameters were adjusted. After fine-tuning, the camera's image quality was significantly improved, and distortion in the images was noticeably reduced, providing the autonomous driving system with more accurate and clearer environmental information.

[0133] Thus, by fine-tuning the internal parameters, the camera's imaging quality can be optimized, reducing image distortion and other problems, resulting in clearer and more accurate images. Improved imaging quality helps autonomous driving systems more accurately identify and understand objects and scenes in their surroundings, thereby enhancing environmental perception capabilities. Optimized camera imaging quality can improve the decision-making accuracy and driving safety of autonomous driving systems, reducing the risk of traffic accidents caused by errors in environmental perception.

[0134] In some embodiments, determining a standard projection point based on the 3D anchor points of the target image after rotation angle fine-tuning, translation fine-tuning, and intrinsic parameter fine-tuning may include:

[0135] The projection matrix is ​​determined based on the camera's intrinsic and extrinsic parameters;

[0136] Standard projection points are obtained based on the coordinates and projection matrix of 3D anchor points.

[0137] In this embodiment of the disclosure, a 3D anchor point is a point with specific coordinates defined in three-dimensional space, which is often used as a reference point or target point for positioning, tracking or measurement.

[0138] In this embodiment of the disclosure, the projection matrix is ​​a mathematical transformation matrix used to map points in three-dimensional space onto a two-dimensional image plane. It contains intrinsic and extrinsic parameter information of the camera.

[0139] In this embodiment of the disclosure, distortion correction is a process of correcting image distortion caused by imperfections in the camera lens, in order to improve the accuracy and usability of the image.

[0140] In some implementations, determining the camera's intrinsic and extrinsic parameters includes:

[0141] Determine the intrinsic parameters: including focal length (fx, fy), principal point coordinates (cx, cy), and distortion coefficients, which describe the internal geometry and optical characteristics of the camera;

[0142] Determine the extrinsic parameters: these include the rotation matrix (R) and translation vector (T), which describe the camera's position and orientation in the world coordinate system.

[0143] In some implementations, calculating the projection matrix includes:

[0144] The projection matrix P is a combination of the camera's intrinsic and extrinsic parameters, which maps points in the 3D world coordinate system to the 2D image coordinate system. The formula for calculating the projection matrix is ​​usually: P = K[R|T], where K is the camera's intrinsic parameter matrix and [R|T] is the camera's extrinsic parameter matrix (including rotation and translation).

[0145] In some implementations, applying a projection matrix to 3D anchor points includes:

[0146] For a given 3D anchor point M(X,Y,Z), we multiply it by the projection matrix P to obtain its 2D coordinates (u,v) on the image plane. The calculation can be expressed as: m = PM, where m is the projected 2D point, P is the projection matrix, and M is the homogeneous coordinates of the 3D anchor point. (u,v) is the standard projection point of the 3D anchor point on the image. This projection point takes into account all the geometric and optical properties of the camera, as well as image rotation, translation, and intrinsic parameter adjustments.

[0147] In some implementations, if camera distortion exists (such as radial or tangential distortion), distortion correction of the 2D coordinates is required after projection. This typically involves additional mathematical transformations of the coordinates using distortion coefficients.

[0148] In the visual perception system of autonomous vehicles, cameras are used to capture images of the road and surrounding environment. To accurately identify and track obstacles such as pedestrians and vehicles on the road, the system needs to locate 3D anchor points of these obstacles on the image and project them onto the image plane for subsequent processing. For example, the system may first acquire 3D position information of obstacles using LiDAR or other sensors, and then use the camera's intrinsic and extrinsic parameters and projection matrix to convert this 3D position information into 2D projection points on the image. The system can then use these projection points to perform tasks such as obstacle detection, tracking, and obstacle avoidance. In this process, fine-tuning of the camera's rotation angle, translation, and intrinsic parameters is crucial to ensuring the accuracy of the projection points.

[0149] Thus, through precise camera calibration and adjustment of intrinsic and extrinsic parameters, the projection accuracy of 3D anchor points onto images can be significantly improved. This is crucial for tasks such as obstacle detection and path planning in autonomous driving. Accurate projection points help autonomous driving systems better understand their surroundings, improving the accuracy and robustness of environmental perception. It also reduces computational errors and decision-making mistakes caused by projection errors, optimizing the overall performance and safety of the autonomous driving system.

[0150] In some embodiments, the target detection method may further include:

[0151] With the virtual camera off, the initial image is input into a pre-trained second detection model to perform detection processing on the initial image, resulting in the second target detection result.

[0152] In this embodiment of the disclosure, the second detection model and the first detection model can be the same image processing model. Alternatively, the second detection model and the first detection model can be different image processing models; for example, the accuracy of the second detection model may be greater than that of the first detection model, or the generalization ability of the second detection model may be higher than that of the first detection model.

[0153] In some implementations, the initial image is input into a pre-trained second detection model to perform detection processing on the initial image, including: acquiring the original image (i.e., the initial image) from the vehicle camera, at which time the virtual camera function is not enabled and the image has not undergone any simulation or correction processing; and directly inputting the captured initial image into the pre-trained second detection model for detection processing to obtain the second target detection result.

[0154] In this embodiment of the disclosure, the second target detection result also includes the target's category, location, and confidence information, but may differ in accuracy and detection efficiency due to differences in viewpoint and image content.

[0155] In some implementations, if the virtual camera is not activated, the initial image is directly input into a pre-trained second detection model for object detection. The second detection model may differ from the first detection model and can be optimized for the original image without perspective transformation.

[0156] Suppose an autonomous vehicle is driving at night to a complex intersection. At this time, the system does not activate the virtual camera function. The initial image is input into a specially trained second detection model, such as a night target detection model. The model quickly identifies pedestrians, vehicles and traffic lights ahead, and accurately provides their location and category information, providing important information for the vehicle's decision-making.

[0157] Thus, by providing the option to enable or disable the virtual camera, the system can adapt to different usage scenarios and needs, enhancing its flexibility. Employing different detection models for different image types (raw images and virtual camera images) allows for more efficient use of computing resources and faster processing.

[0158] Figure 4 This is a flowchart of the virtual camera processing according to an embodiment of the present disclosure, such as... Figure 4 As shown, the processing flow consists of two main parts: the initialization process and the running process.

[0159] The initialization process includes calculating the homography matrix. This calculation involves: determining the source projection point of the source image based on camera parameters; determining the standard projection point of the target image based on virtual camera parameters; and performing Gaussian elimination on the source and standard projection points to obtain the homography matrix. Specifically, determining the standard projection point of the target image based on virtual camera parameters includes: fine-tuning the rotation angle of the target image to perfectly align the first coordinate system of the target image with the second coordinate system of the source image; fine-tuning the translation of the target image to match its position in the source image; fine-tuning the intrinsic parameters of the target image to achieve a preset imaging quality; and determining the standard projection point based on the 3D anchor points of the target image after rotation, translation, and intrinsic parameter adjustments. After calculating the standard projection point, it is stored.

[0160] The process includes: performing fine-tuning processing such as cropping and scaling on the acquired initial image; performing pixel mapping on the fine-tuned initial image based on the homography matrix to obtain the image from the virtual camera's perspective; and performing normalization on the image from the virtual camera's perspective to obtain the virtual camera image.

[0161] The pixel mapping of the fine-tuned initial image based on the homography matrix can be divided into online and offline modes, and the methods used can differ depending on the vehicle's operating mode. For example, in offline mode, a computer vision library is used to map the coordinates of each pixel in the initial image, transforming its coordinate position from the original camera's viewpoint to the virtual camera's viewpoint. In online mode, a parallel computing platform and programming model are used to map the coordinates of each pixel in the initial image, transforming its coordinate position from the original camera's viewpoint to the virtual camera's viewpoint.

[0162] Here, CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model. It allows developers to leverage powerful computing capabilities to perform computationally intensive tasks. CUDA's main applications include accelerating applications that require extensive parallel computing, such as image processing, scientific computing, machine learning, and artificial intelligence.

[0163] Here, OpenCV (Open Source Computer Vision Library) is an open-source computer vision library. It provides a wide range of tools and functions for image processing and computer vision tasks, and can be used in multiple fields such as face recognition, object detection, image enhancement, and video processing.

[0164] In some implementations, normalizing the image from the virtual camera's perspective to obtain a virtual camera image may include at least one of the following:

[0165] Image size standardization: Resizing images of different resolutions or sizes to a uniform size to ensure consistency in subsequent processing. This is typically achieved through scaling or cropping operations.

[0166] Color space normalization: Transforming or normalizing the color space of an image to conform to a specific color model or range. For example, normalizing the color values ​​of a primary color (Red, Green, Blue, RGB) image to the range of 0 to 1, or converting it to a grayscale image.

[0167] Pixel value normalization: Adjusting the pixel values ​​of an image to a standardized range, for example, scaling the range of pixel values ​​from 0 to 255 to a range of 0 to 1. This helps improve the consistency and comparability of images across different processing steps.

[0168] Geometric transformation normalization: Applying geometric transformations such as rotation, translation, and scaling to make the image consistent with the real camera view or the expected view from the perspective of the virtual camera.

[0169] Contrast and brightness normalization: Adjust the contrast and brightness of the image to ensure that different images have the same visual effect from the perspective of the virtual camera.

[0170] Thus, by using CUDA or OpenCV libraries, pixel mapping and image normalization operations based on the homography matrix are implemented, significantly improving the efficiency and accuracy of image processing. Through normalization, features such as brightness, contrast, color, and geometry of the image are standardized, resulting in higher consistency between images captured in different environments. This consistency not only reduces the interference of external condition changes on image processing and object detection but also provides more reliable input data for subsequent processing steps. Furthermore, the standardized input simplifies the model training and application process, improving the overall performance of the system.

[0171] Figure 5 This is a schematic diagram comparing an initial image and a virtual camera image according to an embodiment of the present disclosure, such as... Figure 5 As shown, the left image is the original image, and the right image is the virtual camera image obtained after processing the original image on the left. Clearly, the virtual camera image on the right is clearer and more accurate than the original image on the left. Even when the vehicle parameters differ significantly, the image quality is noticeably improved by using only a virtual camera to transform the left image into the right image.

[0172] It should be understood that Figure 4 , Figure 5 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figure 4 , Figure 5Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0173] This disclosure provides a target detection device, such as... Figure 6 As shown, the target detection device may include: an image acquisition module 601, used to acquire an image of the road where the vehicle is located at the current moment captured by a camera, which is an initial image; an image processing module 602, used to perform pixel mapping on the initial image based on the homography matrix when the virtual camera is turned on, to obtain a virtual camera image; and a first detection module 603, used to input the virtual camera image into a pre-trained first detection model, perform detection processing on the virtual camera image, and obtain a first target detection result.

[0174] In some embodiments, the image processing module 602 is further configured to: fine-tune the initial image before performing pixel mapping on the initial image based on the homography matrix to obtain an optimized initial image.

[0175] In some embodiments, the image processing module 602 may include:

[0176] The mapping submodule is used to perform pixel mapping on the initial image based on the homography matrix to obtain the image from the virtual camera's perspective.

[0177] The normalization submodule is used to normalize the image from the virtual camera's perspective to obtain the virtual camera image.

[0178] In some embodiments, the target detection device further includes:

[0179] First Determining Module ( Figure 6 (not shown in the image), used to determine the source projection point of the source image based on camera parameters;

[0180] Second determination module ( Figure 6 (not shown in the image), used to determine the standard projection points of the target image based on the virtual camera parameters;

[0181] The third determination module ( Figure 6 (not shown in the figure), used to perform Gaussian elimination based on the source projection point and the standard projection point to obtain the homography matrix.

[0182] In some implementations, the second determining module includes:

[0183] The first fine-tuning submodule is used to fine-tune the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0184] The second fine-tuning submodule is used to translate and fine-tune the target image so that the position of the target image matches the position in the source image.

[0185] The third fine-tuning submodule is used to fine-tune the intrinsic parameters of the target image so that the imaging quality of the target image reaches the preset value.

[0186] The determination submodule is used to determine the standard projection point based on the 3D anchor points of the target image after fine-tuning of rotation angle, translation, and intrinsic parameters.

[0187] In some implementations, the first fine-tuning submodule is used for:

[0188] Determine the angle by which the target image needs to be rotated relative to the source image based on the target points;

[0189] Rotate the target image by the corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0190] In some implementations, the second fine-tuning submodule is used for:

[0191] Determine at least one reference point based on the source image and the target image;

[0192] The translation vector is obtained based on the positional difference of at least one reference point in the source and target images;

[0193] The target image is moved based on the translation vector so that its position matches the position in the source image.

[0194] In some implementations, the third fine-tuning submodule is used for:

[0195] Obtain the camera's intrinsic parameters;

[0196] The camera's internal parameters should be recalibrated or adjusted according to the internal parameter requirements of different application scenarios.

[0197] In some implementations, the determining submodule is used for:

[0198] The projection matrix is ​​determined based on the camera's intrinsic and extrinsic parameters;

[0199] Standard projection points are obtained based on the coordinates and projection matrix of 3D anchor points.

[0200] In some embodiments, the target detection device further includes:

[0201] Second detection module ( Figure 6 (Not shown in the image) is used to input the initial image into a pre-trained second detection model when the virtual camera is not turned on, to perform detection processing on the initial image, and to obtain the second target detection result.

[0202] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0203] The target detection device of this disclosure can enable the detection model to uniformly process image data under different vehicle models and different camera configurations, reduce system errors caused by differences in camera parameters, thereby ensuring the consistency of image-to-3D position mapping under various configuration conditions, improving the accuracy of target detection, and making environmental perception in the autonomous driving system more accurate and stable.

[0204] This disclosure provides a scenario illustration of a target detection method, such as... Figure 7 As shown.

[0205] As previously described, the target detection method provided in this disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0206] Specifically, the electronic device may perform the following operations:

[0207] In response to the vehicle's start-up operation, the camera is controlled to capture an image of the road where the vehicle is located at the current moment; this image is the initial image.

[0208] With the virtual camera enabled, the initial image is pixel-mapped based on the homography matrix to obtain the virtual camera image;

[0209] The virtual camera image is input into the pre-trained first detection model to perform detection processing on the virtual camera image and obtain the first target detection result;

[0210] With the virtual camera off, the initial image is input into a pre-trained second detection model to perform detection processing on the initial image, resulting in the second target detection result.

[0211] The first and second detection models can be obtained from the data source.

[0212] Data sources can be various forms of data storage devices, such as laptops, desktop computers, workstations, personal digital assistants (PDAs), servers, blade servers, mainframes, and other suitable computers. Data sources can also represent various forms of mobile devices, such as PDAs, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0213] The vehicle adjusts its driving status and behavior in real time based on the target detection results, such as obstacle recognition and avoidance, traffic signal recognition and compliance, lane keeping and lane changing, speed control, emergency braking and obstacle avoidance, etc., to ensure driving safety and efficiency.

[0214] It should be understood that Figure 7 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 7 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0215] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0216] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0217] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0218] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0219] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0220] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as object detection methods. For example, in some embodiments, the object detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the object detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the target detection method by any other suitable means (e.g., by means of firmware).

[0221] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0223] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0226] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0227] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0228] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A target detection method applied to a vehicle, wherein the vehicle is equipped with a camera; wherein, The target detection method includes: In response to the vehicle's start-up operation, the camera is controlled to capture an image of the road where the vehicle is located at the current moment; the image is the initial image. With the virtual camera enabled, the initial image is pixel-mapped based on the homography matrix to obtain the virtual camera image. The virtual camera is used to convert initial images from different cameras into virtual camera images generated under a unified viewpoint and parameters. Specifically, the calculated homography matrix is ​​used to map the coordinates of each pixel in the initial image to the standard viewpoint coordinate system of the virtual camera to generate the virtual camera image. The virtual camera image is input into a pre-trained first detection model to perform detection processing on the virtual camera image, thereby obtaining a first target detection result; The homography matrix is ​​obtained through the following steps: The source projection point of the source image is determined based on the camera parameters; Determining the standard projection point of the target image based on the parameters of the virtual camera includes: fine-tuning the rotation angle of the target image to make the first coordinate system of the target image completely aligned with the second coordinate system of the source image; fine-tuning the translation of the target image to make the position of the target image match the position in the source image; recalibrating or adjusting the intrinsic parameters of the virtual camera according to the current application scenario to make the imaging quality of the target image reach a preset value; and determining the standard projection point based on the three-dimensional anchor points of the target image after the rotation angle fine-tuning, translation fine-tuning, and intrinsic parameter fine-tuning. The homography matrix is ​​obtained by performing Gaussian elimination based on the source projection points and the standard projection points. When the virtual camera is not turned on, the initial image is input into a pre-trained second detection model to perform detection processing on the initial image and obtain the second target detection result.

2. The method according to claim 1, wherein, Before performing pixel mapping on the initial image based on the homography matrix, the method further includes: The initial image is fine-tuned to obtain an optimized initial image.

3. The method according to claim 1, wherein, The step of mapping the initial image to pixels based on the homography matrix to obtain the virtual camera image includes: The initial image is pixel-mapped based on the homography matrix to obtain an image from the virtual camera's perspective. The image from the virtual camera's perspective is normalized to obtain the virtual camera image.

4. The method according to claim 1, wherein, The step of fine-tuning the rotation angle of the target image to make the first coordinate system of the target image completely aligned with the second coordinate system of the source image includes: Determine the angle by which the target image needs to be rotated relative to the source image based on the target point; The target image is rotated by a corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

5. The method according to claim 1, wherein, The step of translating and fine-tuning the target image to match its position with that in the source image includes: Determine at least one reference point based on the source image and the target image; Based on the positional difference of the at least one reference point in the source image and the target image, a translation vector is obtained; The target image is moved based on the translation vector so that the position of the target image matches the position in the source image.

6. The method according to claim 1, wherein, The determination of the standard projection point based on the three-dimensional anchor points of the target image after the rotation angle fine-tuning, translation fine-tuning, and intrinsic parameter fine-tuning includes: The projection matrix is ​​determined based on the camera's intrinsic and extrinsic parameters; The standard projection point is obtained based on the coordinates of the three-dimensional anchor point and the projection matrix.

7. A target detection device applied to a vehicle, wherein the vehicle is equipped with a camera; wherein, The target detection device includes: The image acquisition module is used to acquire an image of the road where the vehicle is located at the current moment, captured by the camera; the image is an initial image. The image processing module is used to perform pixel mapping on the initial image based on the homography matrix when the virtual camera is turned on, so as to obtain the virtual camera image. The virtual camera is used to convert the initial images from different cameras into virtual camera images generated under a unified viewpoint and parameters. Specifically, it uses the calculated homography matrix to map the coordinates of each pixel in the initial image to the standard viewpoint coordinate system of the virtual camera to generate the virtual camera image. The first detection module is used to input the virtual camera image into a pre-trained first detection model, perform detection processing on the virtual camera image, and obtain a first target detection result; The first determining module is used to determine the source projection point of the source image based on the camera parameters; The second determining module is used to determine the standard projection points of the target image based on the parameters of the virtual camera; The third determining module is used to perform Gaussian elimination based on the source projection points and the standard projection points to obtain the homography matrix. The second determining module includes: The first fine-tuning submodule is used to fine-tune the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image. The second fine-tuning submodule is used to perform translational fine-tuning on the target image so that the position of the target image matches the position in the source image; The third fine-tuning submodule is used to recalibrate or adjust the intrinsic parameters of the virtual camera according to the current application scenario, so that the imaging quality of the target image reaches the preset value. The determination submodule is used to determine the standard projection point based on the three-dimensional anchor points of the target image after the rotation angle fine-tuning, translation fine-tuning, and intrinsic parameter fine-tuning. The second detection module is used to input the initial image into a pre-trained second detection model when the virtual camera is not turned on, and to perform detection processing on the initial image to obtain the second target detection result.

8. The apparatus according to claim 7, wherein, The image processing module is further configured to: fine-tune the initial image before performing pixel mapping on the initial image based on the homography matrix, so as to obtain an optimized initial image.

9. The apparatus according to claim 7, wherein, The image processing module includes: The mapping submodule is used to perform pixel mapping on the initial image based on the homography matrix to obtain an image from the virtual camera's perspective. The normalization submodule is used to perform a normalization operation on the image from the perspective of the virtual camera to obtain the virtual camera image.

10. The apparatus according to claim 7, wherein, The first fine-tuning submodule is used for: Determine the angle by which the target image needs to be rotated relative to the source image based on the target point; The target image is rotated by a corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

11. The apparatus according to claim 7, wherein, The second fine-tuning submodule is used for: Determine at least one reference point based on the source image and the target image; Based on the positional difference of the at least one reference point in the source image and the target image, a translation vector is obtained; The target image is moved based on the translation vector so that the position of the target image matches the position in the source image.

12. The apparatus according to claim 7, wherein, The determining submodule is used for: The projection matrix is ​​determined based on the camera's intrinsic and extrinsic parameters; The standard projection point is obtained based on the coordinates of the three-dimensional anchor point and the projection matrix.

13. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program stored on a storage medium, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1-6.

Images