Target detection method and device, electronic equipment and storage medium

Abstract

The invention provides a target detection method and device, electronic equipment and a storage medium. The invention relates to the field of automatic driving, in particular to the field of image recognition of automatic driving. According to the specific implementation scheme, in response to starting operation of a vehicle, a camera is controlled to shoot an image of a road where the vehicle is located at the current moment, and the image is an initial image; under the condition that the virtual camera is started, pixel mapping is carried out on the initial image based on a homography matrix, and a virtual camera image is obtained; and inputting the virtual camera image into a pre-trained first detection model, and performing detection processing on the virtual camera image to obtain a first target detection result. According to the disclosure of the scheme, system errors caused by camera parameter differences can be reduced, and the accuracy of target detection is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, in particular to the field of image recognition for autonomous driving, and specifically to a target detection method, device, electronic device, and storage medium. Background Art

[0002] There are large differences in the camera installation positions and internal parameters of different models and within the same model. These differences will cause the same three-dimensional (3D) anchor point to be mapped to different positions on the image under different camera perspectives, thus causing system errors. This error may lead to inconsistent prediction results of computer vision models under different conditions, especially in road structure perception tasks, affecting the accuracy and stability of the model. Summary of the invention

[0003] The present disclosure provides a target detection method, device, electronic device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a target detection method, comprising:

[0005] In response to the vehicle start operation, the camera is controlled to capture an image of the road where the vehicle is currently located, where the image is an initial image;

[0006] When the virtual camera is turned on, the initial image is pixel mapped based on the homography matrix to obtain a virtual camera image;

[0007] The virtual camera image is input into a pre-trained first detection model, and detection processing is performed on the virtual camera image to obtain a first target detection result.

[0008] According to another aspect of the present disclosure, there is provided a target detection device, comprising:

[0009] An image acquisition module is used to acquire an image of the road where the vehicle is located at the current moment taken by the camera, and the image is an initial image;

[0010] An image processing module, used for performing pixel mapping on the initial image based on the homography matrix to obtain a virtual camera image when the virtual camera is turned on;

[0011] The first detection module is used to input the virtual camera image into a pre-trained first detection model, perform detection processing on the virtual camera image, and obtain a first target detection result.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] By adopting the solution disclosed in the present invention, the system error caused by the difference in camera parameters can be reduced and the accuracy of target detection can be improved.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a schematic diagram of camera positions according to an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of a target detection method according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of a process for determining a homography matrix according to an embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of virtual camera processing according to an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of comparison between an initial image and a virtual camera image according to an embodiment of the present disclosure;

[0026] Figure 6 is a schematic structural diagram of a target detection device according to an embodiment of the present disclosure;

[0027] Figure 7 is a schematic diagram of a scene of a target detection method according to an embodiment of the present disclosure;

[0028] Figure 8It is a schematic diagram of the structure of an electronic device used to implement the target detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0031] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0032] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0033] Homography Matrix: In image processing and computer vision, the homography matrix is ​​used to describe the projection mapping relationship between two planes and is often used in scenes such as image correction and perspective transformation.

[0034] Object Detection: It is a basic task in the field of computer vision, which aims to identify target objects in images or videos and determine their locations and categories.

[0035] In the related art, in the autonomous driving system, the camera is used to perceive the surrounding environment and generate a view of the road structure. However, there are large differences in the camera installation positions and internal parameters of different models and their internal stages. This difference will cause the same 3D anchor point in the same scene to be mapped to different positions on the image in different camera perspectives, thereby causing system errors. This error may cause inconsistent prediction results of the computer vision model under different conditions, especially in the road structure perception task, affecting the accuracy and stability of the model. Therefore, how to eliminate the system errors caused by differences in camera parameters under a variety of camera configuration conditions, ensure the consistency of the image to 3D position mapping relationship under different configuration conditions, and improve the generalization ability of the model is one of the keys to ensure the accuracy and stability of environmental perception in the autonomous driving system.

[0036] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a target detection method, which processes the initial image based on virtual camera parameters, and can uniformly process image data under different vehicle models and different camera configurations, eliminate system errors caused by differences in camera parameters, thereby ensuring the consistency of the image-to-3D position mapping relationship under various configuration conditions, and making the environmental perception in the autonomous driving system more accurate and stable.

[0037] According to an embodiment of the present disclosure, Figure 1 is a schematic diagram of camera positions according to an embodiment of the present disclosure, such as Figure 1 As shown, the vehicle 100 is equipped with multiple cameras (such as camera 101-camera 106), which are installed at different positions of the vehicle (such as the front, left front, right front, rear, left rear, and right rear of the vehicle). Cameras at different positions can capture images in different directions. It can be understood that Figure 1 The vehicle shown is only for illustration, and the number and installation positions of the cameras are not limited thereto. The present disclosure does not limit the vehicle model and type, nor does it limit the number and installation positions of the cameras.

[0038] The present disclosure provides a target detection method. Figure 2 : is a flow chart of a target detection method according to an embodiment of the present disclosure, and the target detection method can be applied to a target detection device. The target detection device is located in an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, a fixed device includes but is not limited to a server, and the server can be a cloud server or an ordinary server. For example, a mobile device includes but is not limited to: a mobile phone, a tablet computer, and a vehicle-mounted terminal. In some possible implementations, the target detection method can also be implemented by a processor calling a computer-readable instruction stored in a memory. For example Figure 2As shown, the target detection method includes:

[0039] S201, in response to a vehicle startup operation, controlling a camera to capture an image of a road where the vehicle is currently located, the image being an initial image;

[0040] S202, when the virtual camera is turned on, pixel mapping is performed on the initial image based on the homography matrix to obtain a virtual camera image;

[0041] S203: Input the virtual camera image into a pre-trained first detection model, perform detection processing on the virtual camera image, and obtain a first target detection result.

[0042] In the disclosed embodiments, roads may include but are not limited to urban roads: including standard lanes, complex intersections, pedestrian crossing areas, traffic signs, signal lights, etc.; expressways: covering multiple lanes, toll booths, ramps, diversion areas and viaducts, etc.; rural roads: including narrow roads, unpaved roads and roads in agricultural and forestry areas, etc.; tunnels and underpasses; ramps and mountain roads, etc.

[0043] In the disclosed embodiments, a virtual camera refers to a camera model implemented by software simulation, which is used to uniformly process images taken by different vehicles. The virtual camera can perform pixel mapping on initial images from different cameras based on a preset homography matrix, thereby converting these images into standardized images (i.e., virtual camera images) generated under a unified perspective and parameters.

[0044] In the disclosed embodiments, a homography matrix refers to a mathematical tool for describing the perspective transformation relationship between two planes. Specifically, a homography matrix can geometrically transform images from different perspectives so that image points originally at different positions can correspond to the same plane after the transformation.

[0045] In the embodiments of the present disclosure, the first detection model may include but is not limited to a perception model, a target detection model, etc. The above is only an exemplary description and is not intended to limit all possible types included in the first detection model, but it is not exhaustive here. It should be noted that the present disclosure does not limit the training method and model structure of the first detection model. The first detection model can be based on a deep learning algorithm and can automatically identify target objects in an image (such as pedestrians, vehicles, traffic signs, etc.).

[0046] In some embodiments, controlling the camera to capture images of the road on which the vehicle is located includes: in response to the driver or the automatic driving program starting the vehicle operation, the camera can be triggered to capture images around the vehicle. In this way, the vehicle can periodically capture images of the road on which the vehicle is located during driving.

[0047] In some embodiments, pixel mapping of the initial image based on the homography matrix includes: first, obtaining the intrinsic and extrinsic information of the camera from the sensor system of the vehicle. The intrinsic parameters usually include parameters such as focal length and optical center, while the extrinsic parameters include the rotation matrix and translation vector of the camera relative to the vehicle. Based on these intrinsic and extrinsic parameters, the ground hypothesis is applied to calculate the homography matrix. The homography matrix can map the ground points in the road image from the perspective of the original camera to the perspective of the virtual camera, and by projecting the three-dimensional anchor points under the ground hypothesis to the two-dimensional image plane, it is ensured that these points remain consistent under different camera parameters. Next, using the calculated homography matrix, the coordinates of each pixel point in the initial image are mapped to the standard perspective coordinate system of the virtual camera. Specifically, after the two-dimensional coordinates of each pixel point in the initial image are multiplied by the homography matrix, new coordinates will be obtained, which represent the position of the pixel under the perspective of the virtual camera. The transformed pixels are reconstructed into the virtual camera image to form a more standardized image compared with the original image, which can be used as the input of the subsequent detection model.

[0048] For example, suppose that after the vehicle starts, the system automatically activates the front camera and starts capturing images of the road in front of the vehicle. At this time, the image captured by the camera includes a pedestrian in front crossing the zebra crossing. The virtual camera obtains the intrinsic parameters of the current camera (for example, the focal length is 35mm, the optical center is located at the center of the image) and extrinsic parameters (for example, the rotation angle of the camera relative to the vehicle is 5 degrees, and the front and rear translation vector is 0.5 meters) from the sensor system. Based on these parameters, the system calculates a homography matrix using the ground assumption. Assuming that the position of a pedestrian in the image in the original image is (1000,500) pixels, through the transformation of the homography matrix, the point is mapped to the standard view of the virtual camera, and the new position may become (980,520). In this way, even if the camera parameters of the vehicle are different, the position of the pedestrian in the virtual camera's view is always consistent.

[0049] In some embodiments, when the vehicle is started and meets preset conditions (such as seat belts fastened, doors closed, etc.), the system automatically triggers the camera to start working. The camera captures the road environment in front of the vehicle at the current moment, generates a high-definition image as the initial image, and stores it in the system memory for subsequent processing.

[0050] In some embodiments, the system checks whether the virtual camera function is turned on. If it is turned on, the homography matrix is ​​calculated according to the preset virtual camera parameters (such as position, angle, focal length, etc.). Using the calculated homography matrix, the initial image is perspective transformed and pixel remapped to simulate the image seen from the perspective of the virtual camera, that is, the virtual camera image. This step is intended to optimize the effect of target detection by adjusting the perspective, especially for target recognition in specific scenes or road conditions.

[0051] In some embodiments, the virtual camera image is input into a pre-trained first detection model to obtain a first target detection result, which includes but is not limited to information such as the target category, location (bounding box), and confidence level.

[0052] The technical solution of the disclosed embodiment can unify the images taken by different vehicles and cameras to the same viewing angle through a virtual camera and a homography matrix, reduce the deviation caused by parameter differences, and ensure the consistency of the image. At the same time, since the virtual camera image reduces image distortion and redundant information relative to the initial image, inputting the virtual camera image generated by the virtual camera into the pre-trained model can improve the accuracy of target detection. The above-mentioned image processing method enables the same detection model to be applicable to target detection under different camera configuration conditions, which also helps to improve the versatility and stability of the model.

[0053] In some embodiments, before performing pixel mapping on the initial image based on the homography matrix, the object detection method may further include: performing fine-tuning processing on the initial image to obtain an optimized initial image.

[0054] Here, the fine-tuning process includes but is not limited to cropping, scaling, or other adjustment and optimization operations on the initial image to improve the image quality or correct certain problems in the image, so as to obtain an optimized image that is more suitable for subsequent processing. These adjustment and optimization operations may include but are not limited to translation, rotation, brightness and contrast adjustment, denoising, etc.

[0055] In this way, through fine-tuning, the quality of the initial image used to generate the virtual camera image is improved, problems such as noise, uneven lighting and geometric distortion are eliminated, and a clearer and more accurate optimized image is generated, thereby improving the image quality of the virtual camera, which helps to improve the accuracy of target detection.

[0056] In some embodiments, pixel mapping the initial image based on the homography matrix to obtain a virtual camera image may include:

[0057] Perform pixel mapping on the initial image based on the homography matrix to obtain the image from the perspective of the virtual camera;

[0058] The image under the perspective of the virtual camera is normalized to obtain a virtual camera image.

[0059] In this embodiment, pixel mapping refers to mapping pixels in the initial image to new positions under the perspective of the virtual camera using a homography matrix.

[0060] In some embodiments, after obtaining the homography matrix, the homography matrix can be used to map each pixel in the initial image and convert it to a new position under the virtual camera perspective. This step is usually implemented by matrix multiplication, that is, the homogeneous coordinates of each pixel are multiplied by the homography matrix to obtain its new coordinates under the virtual camera perspective. Since the mapped coordinates may not be integers, interpolation processing (such as bilinear interpolation or bicubic interpolation) is usually required to obtain the final pixel value.

[0061] In some embodiments, the mapped image may have uneven pixel values ​​or exceed a normal range due to perspective transformation and other reasons. Therefore, the mapped image needs to be normalized to ensure that its pixel values ​​are within a reasonable range. The normalization operation may include adjusting the brightness and contrast of the image, or scaling the pixel values ​​to a specific range.

[0062] Suppose there is an initial image of a street scene, and we want to simulate the image from the perspective of a virtual camera looking down from above. We can follow the following steps: select multiple sets of corresponding points (such as corners of buildings, intersections of roads, etc.) in the initial image and the desired virtual camera perspective respectively; use these corresponding points to calculate the homography matrix; apply the homography matrix to pixel map the initial image to obtain the image from the perspective of the virtual camera. Normalize the mapped image and adjust its brightness, contrast and other parameters to obtain the final virtual camera image. In this way, we can convert an ordinary street scene image into a virtual camera image with a bird's-eye view effect through the homography matrix and pixel mapping technology.

[0063] In this way, through the mapping of the homography matrix, the initial image can be easily converted into an image under any virtual camera perspective, realizing the free transformation of perspective. Combined with the normalization operation, the visual effect of the mapped image can be further improved to make it more in line with the needs of practical applications, thereby further improving the quality of the generated virtual camera image.

[0064] In some embodiments, Figure 3 As shown, the process of determining the homography matrix may include:

[0065] S301, determining a source projection point of a source image according to camera parameters;

[0066] S302, determining a standard projection point of a target image according to virtual camera parameters;

[0067] S303, performing Gaussian elimination according to the source projection point and the standard projection point to obtain a homography matrix.

[0068] In some embodiments, the source projection point of the source image is determined based on the camera parameters, including: when the vehicle is started, the intrinsic and extrinsic parameter information of the current camera is obtained by reading the camera configuration file or through online calibration; then, using these camera parameters, the three-dimensional point in the real world is mapped to a two-dimensional coordinate position on the source image plane; this two-dimensional coordinate point is the source projection point, which represents the position corresponding to the three-dimensional point in the source image.

[0069] In some embodiments, determining a standard projection point of a target image based on virtual camera parameters includes: obtaining internal and external parameter information of a current vehicle camera from a sensor system of the vehicle; performing standard virtual camera calibration based on the obtained internal and external parameters; firstly fine-tuning the rotation angle of the target image so that a first coordinate system of the target image is completely aligned with a second coordinate system of the source image; secondly fine-tuning the translation of the target image so that a position of the target image matches a position in the source image; again fine-tuning the internal parameters of the target image so that the imaging quality of the target image reaches a preset value; and finally determining a standard projection point based on the 3D anchor point of the target image after rotation angle fine-tuning, translation fine-tuning, and internal parameter fine-tuning.

[0070] In some embodiments, Gaussian elimination is performed based on the source projection points and the standard projection points to obtain a homography matrix, including: first, obtaining a number of known source projection points from the source image (these points can be two-dimensional coordinates in the source image); at the same time, obtaining the corresponding points of these points in the standard image, namely the standard projection points (these points can also be two-dimensional coordinates, usually corresponding to points in the virtual camera image); then, according to the correspondence between the source projection points and the standard projection points, a linear equation group about the homography matrix is ​​established; two equations are provided for each pair of source projection points and standard projection points, and multiple points form multiple equations; Gaussian elimination is used to solve the equation group to obtain various parameters of the homography matrix; Gaussian elimination method converts the equation group into an upper triangular form by performing row operations on the equation group, and then performs back substitution to solve it, and finally determines the homography matrix; after all unknowns are solved, a homography matrix is ​​constructed to map points in the source image to corresponding positions in the standard image.

[0071] In this way, by accurately calculating the source projection point and the standard projection point, and using the Gaussian elimination method to solve the homography matrix, the systematic error caused by different camera parameters is effectively eliminated, thereby improving the accuracy of image processing after virtual camera calibration. At the same time, the standard virtual camera calibration method is adopted, so that the camera parameters of different vehicles can be uniformly mapped to a standard model, reducing the errors caused by differences in vehicle camera configurations, and improving the adaptability and versatility of the target detection model.

[0072] In some embodiments, determining the standard projection point of the target image according to the virtual camera parameters may include:

[0073] The target image is rotated at a slightly different angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image;

[0074] The target image is translated and fine-tuned so that the position of the target image matches the position in the source image;

[0075] Performing internal parameter fine-tuning on the target image so that the imaging quality of the target image reaches a preset value;

[0076] Based on the 3D anchor point of the target image after rotation angle fine-tuning, translation fine-tuning and internal parameter fine-tuning, the standard projection point is determined.

[0077] In the disclosed embodiment, the virtual camera parameters are parameters used to simulate the camera shooting effect, including focal length, optical center position, camera position and direction, etc.

[0078] In the disclosed embodiment, the standard projection point is the precise position of a specific area or feature point in the target image under the perspective of the virtual camera.

[0079] In the disclosed embodiment, a 3D anchor point is a pixel position in an image that corresponds to a fixed point in the real world, and is used to establish a correspondence between the image and the real world.

[0080] In the disclosed embodiment, the preset value can be set or adjusted according to system requirements.

[0081] In some embodiments, fine-tuning the rotation angle of the target image may include: analyzing the angle deviation between the target image and the source image, and fine-tuning the target image using an image rotation algorithm (such as affine transformation, perspective transformation, etc.) until the coordinate systems of the two are aligned. In this way, it is possible to ensure that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0082] In some embodiments, the target image is fine-tuned by translation, including: fine-tuning the target image by a translation transformation algorithm according to the position deviation between the target image and the source image. This generally involves moving the image in the x and y directions. In this way, the position of the target image in the image space can be adjusted to match the corresponding position in the source image.

[0083] In some embodiments, the target image is fine-tuned using internal parameters, including: generally involving fine-tuning of camera internal parameters (such as parameters in the camera matrix K). Camera calibration technology or image quality assessment algorithms can be used to determine the optimal internal parameter settings. Imaging parameters of the target image, such as focal length, optical center position, etc., are adjusted so that the imaging quality of the target image reaches a preset value.

[0084] In some embodiments, based on the 3D anchor point of the target image after fine-tuning of the rotation angle, fine-tuning of the translation, and fine-tuning of the internal parameters, the standard projection point is determined, including: first, according to the correspondence between the fine-tuned target image and the source image, the corresponding position of each 3D anchor point in the target image under the perspective of the virtual camera is determined. Then, using these corresponding positions as a reference, the standard projection point is calculated. The standard projection point is the precise position of a specific area or feature point in the image under the perspective of the virtual camera. In this way, the standard projection point under the perspective of the virtual camera is determined using the fine-tuned target image in combination with its 3D anchor point (i.e., the pixel position in the image corresponding to the fixed point in the real world).

[0085] Assume that in an autonomous driving system, the road image (target image) taken by the on-board camera needs to be aligned with the high-precision map (source image). First, the initial image is rotated slightly to align it with the coordinate system of the map; then, the translation is fine-tuned to ensure the accuracy of the position of the road image in the map; then, the road image is fine-tuned internally to optimize its imaging quality; finally, the lane markings in the road image (as 3D anchor points) are used to determine the standard projection points from the perspective of the virtual camera, thereby providing accurate navigation information for the vehicle.

[0086] In this way, by fine-tuning the rotation angle and translation, the alignment accuracy between the target image and the source image can be significantly improved, and the error caused by image offset or rotation can be reduced. By fine-tuning the internal parameters, the imaging parameters of the target image can be adjusted to make it more in line with the actual application requirements and improve the clarity and accuracy of the image. Based on the fine-tuned target image and 3D anchor point, the standard projection point under the virtual camera's perspective can be determined to achieve precise projection and provide accurate position information for subsequent image processing and analysis.

[0087] In some embodiments, fine-tuning the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image may include:

[0088] Determine the angle at which the target image needs to be rotated relative to the source image based on the target point;

[0089] The target image is rotated by a corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0090] In the disclosed embodiment, the target point should be a feature point in the image that is easy to identify and has a relatively fixed position, such as a corner point, an edge intersection point, etc.

[0091] In the disclosed embodiment, the target image is an image that needs to be rotated slightly to be aligned with the source image. The source image is an image used as a reference or benchmark, and the target image needs to be aligned with it.

[0092] In the disclosed embodiment, the rotation angle is the degree by which the target image needs to be rotated relative to the source image, and is used to achieve complete alignment of the coordinate system.

[0093] In the disclosed embodiment, rotation transformation is an image processing operation that changes the direction and angle of an image by rotating the pixels in the image.

[0094] In some implementations, determining a rotation angle based on a target point includes:

[0095] A set of corresponding target points are selected in the target image and the source image respectively.

[0096] Using these corresponding target points, the rotation angle of the target image relative to the source image is calculated through image matching algorithms (such as feature point matching, template matching, etc.). This usually involves calculating the rotation matrix between the two sets of points.

[0097] Extract the rotation angle value from the rotation matrix. This angle value represents the degree by which the target image needs to be rotated around a central point (usually the center of the image or a specified point) in order to align with the coordinate system of the source image.

[0098] In some implementations, rotating the target image by a corresponding angle includes:

[0099] Use the rotation transformation function in the image processing software or programming library to rotate the target image by the calculated rotation angle. The rotation transformation can be two-dimensional or three-dimensional, depending on the dimensions of the image and the required alignment accuracy.

[0100] During the rotation process, you need to ensure that the center of rotation coincides with the center of the image or the specified alignment point to ensure that the rotated image is also correct in position.

[0101] After rotation, you may need to further fine-tune the image's position or perform other image processing operations (such as scaling, cropping, etc.) to optimize the alignment between the destination image and the source image.

[0102] In this way, by accurately calculating the rotation angle and applying the rotation transformation, the alignment accuracy between the target image and the source image can be significantly improved, and the error caused by image rotation can be reduced. The aligned target image is more visually consistent with the source image, which facilitates subsequent image analysis, processing, and fusion operations. In application scenarios that require image alignment (such as virtual reality, augmented reality, image stitching, etc.), accurately aligned images can significantly improve application effects and user experience.

[0103] In some embodiments, performing fine translation adjustment on the target image so that the position of the target image matches the position in the source image may include:

[0104] determining at least one reference point based on the source image and the target image;

[0105] Obtaining a translation vector based on a position difference of at least one reference point in the source image and the target image;

[0106] Move the destination image based on the translation vector so that the location of the destination image matches the location in the source image.

[0107] In the disclosed embodiment, the reference point is an obvious feature point used to determine the relative position or angle difference between images in image processing.

[0108] In the embodiment of the present disclosure, the rotation angle is the rotation angle required to align the target image with the source image in the coordinate system.

[0109] In the disclosed embodiment, the image rotation algorithm is an algorithm that changes the pixel positions in an image through mathematical transformation to achieve image rotation.

[0110] In some embodiments, at least one reference point is selected in each of the source image and the target image. These reference points should have obvious features, such as road intersections, corners of buildings, road signs, etc., so that they can be accurately identified in different images. In the context of autonomous driving, these reference points may be lane intersections, traffic signs, or roadside facilities with known locations.

[0111] In some embodiments, calculating the rotation angle includes: using image processing technology (such as feature detection and matching algorithm) to determine the coordinates of each reference point in the source image and the target image. Calculating the relative position relationship between these reference points, especially the angle difference. This usually involves vector operations, in which the lines between the reference points are regarded as vectors, and the rotation angle is determined by calculating the angle between these vectors.

[0112] In autonomous driving, it may also be necessary to consider the vehicle's direction of travel and the camera's attitude to accurately calculate the required rotation angle.

[0113] In some embodiments, rotating the target image includes: after determining the rotation angle, the target image can be rotated using an image rotation algorithm. These algorithms usually involve the transformation of the image matrix, wherein the coordinates of each pixel are adjusted accordingly according to the rotation angle. During the rotation, different interpolation methods can be selected to handle pixel blanks or overlaps that may occur after the rotation, such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation.

[0114] In some embodiments, after the rotation, the reference point is checked again to see if it is consistent in position in the target image and the source image to verify the alignment effect. If a deviation is found, the rotation angle can be fine-tuned and the above steps can be repeated until a satisfactory alignment effect is achieved.

[0115] Assume that an autonomous vehicle captures two consecutive road images while driving, one of which is the source image (taken at the vehicle's current position) and the other is the target image (taken at the vehicle's later position). In order to ensure that the vehicle can accurately perceive and locate obstacles and road signs on the road, the target image needs to be rotated and fine-tuned to align it with the coordinate system in the source image. In this example, a road intersection or a fixed facility on the roadside can be selected as a reference point. By comparing the positional relationship of these reference points in the two images, the required rotation angle is calculated. Then, the target image is rotated using an image rotation algorithm until the reference point is in the same position in the two images. In this way, the alignment of the target image and the source image in the coordinate system is completed.

[0116] In this way, by ensuring the accurate alignment of the coordinate system in the image, the autonomous driving system can more accurately identify and understand objects and scenes in the surrounding environment; the alignment of the coordinate system helps reduce misjudgments and misoperations caused by image distortion or position deviation, enhances the accuracy of decision-making, and thus improves the driving safety and stability of autonomous vehicles.

[0117] In some embodiments, fine-tuning the target image by internal parameters so that the imaging quality of the target image reaches a preset value may include:

[0118] Get the camera's internal parameters;

[0119] Recalibrate or adjust the camera's intrinsic parameters according to the intrinsic parameter requirements of different application scenarios.

[0120] In the disclosed embodiment, the camera intrinsic parameters are a set of parameters that describe the internal characteristics of the camera, including focal length, principal point coordinates, lens distortion parameters, etc. These parameters determine how the camera maps points in the three-dimensional world to the two-dimensional image plane.

[0121] In the embodiments of the present disclosure, calibration is a process of obtaining camera intrinsic parameters through specific methods and tools. The calibration process usually requires the use of a calibration plate and an image processing algorithm.

[0122] In the disclosed embodiment, distortion is an image deformation phenomenon caused by the imperfection of the camera lens. Distortion includes radial distortion and tangential distortion, which will affect the accuracy and usability of the image.

[0123] In some embodiments, obtaining the intrinsic parameters of the camera includes: obtaining these intrinsic parameters through a camera calibration process. Calibration usually uses a specific calibration plate (such as a checkerboard calibration plate) and an image processing algorithm, and calculates the intrinsic parameters of the camera by capturing images of the calibration plate at different positions and angles.

[0124] Here, the intrinsic parameters of the camera are parameters that describe the internal characteristics of the camera, including focal length (fx, fy), principal point coordinates (cx, cy), and lens distortion parameters.

[0125] In some implementations, analyzing the application scenario's requirements for internal parameters includes:

[0126] Autonomous driving systems face complex and ever-changing environments, and different application scenarios have different requirements for camera imaging quality. For example, when driving on a highway, the camera needs to be able to clearly capture distant vehicles and road signs; while in a complex urban traffic environment, the camera needs to be able to accurately identify nearby pedestrians and obstacles.

[0127] According to the needs of the application scenario, analyze whether the current camera internal parameters meet the requirements. If not, recalibrate or adjust the internal parameters.

[0128] In some implementations, recalibrating or adjusting the intrinsic parameters of the camera includes:

[0129] If the camera's intrinsic parameters do not meet the requirements of the application scenario, you can obtain new intrinsic parameters through recalibration. The recalibration process is similar to the initial calibration, but you may need to use a calibration board and calibration method that is more suitable for the current application scenario.

[0130] In addition, the existing internal parameters can also be fine-tuned through software algorithms. For example, according to the imaging problems found in actual applications (such as image distortion, etc.), the focal length, principal point coordinates or distortion parameters can be adjusted to optimize the imaging quality.

[0131] After completing the recalibration or adjustment of the internal parameters, the adjustment effect can be verified through actual testing. The test can be carried out in a simulated environment or in an actual road environment, and the image quality before and after the adjustment can be compared to evaluate whether the adjustment effect meets the requirements.

[0132] In the development of autonomous vehicles, in order to ensure that the on-board camera can accurately capture road information and identify obstacles, the camera needs to be calibrated with internal parameters. For example, after preliminary calibration, the wide-angle camera used in a certain autonomous vehicle was found to have obvious barrel distortion in the image. In order to optimize the image quality and reduce the impact of distortion on the autonomous driving system, professional calibration tools and algorithms are used to recalibrate the camera, and adjust internal parameters such as focal length and distortion parameters. After fine-tuning, the camera's imaging quality has been significantly improved, and the distortion in the image has been significantly reduced, providing the autonomous driving system with more accurate and clear environmental information.

[0133] In this way, by fine-tuning the internal parameters, the camera's imaging quality can be optimized, image distortion and distortion can be reduced, and the image can be made clearer and more accurate. The improvement in imaging quality helps the autonomous driving system to more accurately identify and understand objects and scenes in the surrounding environment, thereby improving environmental perception. The optimized camera imaging quality can improve the decision-making accuracy and driving safety of the autonomous driving system, and reduce the risk of traffic accidents caused by environmental perception errors.

[0134] In some embodiments, determining the standard projection point based on the 3D anchor point of the target image after fine-tuning the rotation angle, the translation, and the internal parameter may include:

[0135] Determine the projection matrix based on the camera's internal and external parameters;

[0136] Based on the coordinates of the 3D anchor point and the projection matrix, the standard projection point is obtained.

[0137] In the embodiments of the present disclosure, a 3D anchor point is a point with specific coordinates defined in a three-dimensional space, and is often used as a reference point or a target point for positioning, tracking or measurement.

[0138] In the disclosed embodiment, the projection matrix is ​​a mathematical transformation matrix used to map points in a three-dimensional space to a two-dimensional image plane. It contains the internal and external parameter information of the camera.

[0139] In the disclosed embodiment, distortion correction is a process of correcting image deformation caused by the imperfection of the camera lens to improve the accuracy and usability of the image.

[0140] In some implementations, determining intrinsic and extrinsic parameters of a camera includes:

[0141] Determine the internal parameters: including focal length (fx, fy), principal point coordinates (cx, cy) and distortion coefficients, which describe the internal geometric and optical characteristics of the camera;

[0142] Determine the external parameters: including the rotation matrix (R) and translation vector (T), which describe the position and orientation of the camera in the world coordinate system.

[0143] In some implementations, calculating a projection matrix includes:

[0144] The projection matrix P is a combination of the camera's internal and external parameters, which maps points in the 3D world coordinate system to the 2D image coordinate system. The calculation formula of the projection matrix is ​​usually: P = K[R|T], where K is the camera's internal parameter matrix and [R|T] is the camera's external parameter matrix (including rotation and translation).

[0145] In some embodiments, applying a projection matrix to a 3D anchor point includes:

[0146] For a given 3D anchor point M (X, Y, Z), we multiply it with the projection matrix P to obtain its 2D coordinates (u, v) on the image plane. The calculation process can be expressed as: m = PM, where m is the projected 2D point, P is the projection matrix, and M is the homogeneous coordinates of the 3D anchor point. (u, v) is the standard projection point of the 3D anchor point on the image. This projection point takes into account all the geometric and optical properties of the camera, as well as the rotation, translation, and intrinsic parameter adjustment of the image.

[0147] In some embodiments, if the camera has distortion (such as radial distortion or tangential distortion), the 2D coordinates need to be distorted after projection, which usually involves additional mathematical transformation of the coordinates using distortion coefficients.

[0148] In the visual perception system of an autonomous vehicle, cameras are used to capture images of the road and surrounding environment. In order to accurately identify and track obstacles such as pedestrians and vehicles on the road, the system needs to locate the 3D anchor points of these obstacles on the image and project them onto the image plane for subsequent processing. For example, the system may first obtain the 3D position information of the obstacle through a lidar or other sensor, and then use the camera's internal and external parameters and projection matrix to convert this 3D position information into 2D projection points on the image. The system can then use these projection points to perform tasks such as obstacle detection, tracking, and obstacle avoidance. In this process, fine-tuning the camera's rotation angle, translation, and internal parameter fine-tuning are crucial to ensuring the accuracy of the projection points.

[0149] In this way, through accurate camera calibration and adjustment of internal and external parameters, the projection accuracy of 3D anchor points on the image can be significantly improved, which is crucial for tasks such as obstacle detection and path planning in autonomous driving. Accurate projection points help the autonomous driving system better understand the surrounding environment and improve the accuracy and robustness of environmental perception. It reduces calculation errors and decision-making errors caused by projection errors and optimizes the overall performance and safety of the autonomous driving system.

[0150] In some embodiments, the target detection method may further include:

[0151] When the virtual camera is not turned on, the initial image is input into the pre-trained second detection model, and the initial image is detected and processed to obtain a second target detection result.

[0152] In the disclosed embodiment, the second detection model and the first detection model may be the same image processing model. The second detection model and the first detection model may also be different image processing models, for example, the accuracy of the second detection model is greater than the accuracy of the first detection model, or the generalization ability of the second detection model is higher than the generalization ability of the first detection model.

[0153] In some embodiments, the initial image is input into a pre-trained second detection model, and detection processing is performed on the initial image, including: obtaining the original image (i.e., the initial image) from the vehicle camera, at which time the virtual camera function is not turned on, and the image has not undergone any simulation or correction processing; directly inputting the captured initial image into the trained second detection model for detection processing to obtain a second target detection result.

[0154] In the disclosed embodiment, the second target detection result also includes the category, location and confidence information of the target, but may differ in accuracy and detection efficiency due to differences in viewing angle and image content.

[0155] In some embodiments, if the virtual camera is not turned on, the initial image is directly input into a pre-trained second detection model for object detection. The second detection model may be different from the first detection model and can be optimized for the original image without perspective transformation.

[0156] Suppose an autonomous vehicle drives to a complex intersection at night. At this time, the system does not turn on the virtual camera function. The initial image is input into a specially trained second detection model, such as a night target detection model. The model quickly identifies pedestrians, vehicles and traffic lights in front, and accurately gives their location and category information, providing important basis for vehicle decision-making.

[0157] In this way, by providing the option of whether to turn on the virtual camera, the system can adapt to different usage scenarios and needs, enhancing the flexibility of the system. Using different detection models for different image types (original images and virtual camera images) can more effectively utilize computing resources and speed up processing.

[0158] Figure 4 is a virtual camera processing flow chart according to an embodiment of the present disclosure, such as Figure 4 As shown, the processing flow includes two parts: initialization process and operation process.

[0159] Wherein, the initialization process includes calculating the homography matrix. Wherein, calculating the homography matrix includes: determining the source projection point of the source image according to the camera parameters; determining the standard projection point of the target image according to the virtual camera parameters; performing Gaussian elimination according to the source projection point and the standard projection point to obtain the homography matrix. Wherein, determining the standard projection point of the target image according to the virtual camera parameters includes: fine-tuning the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image; fine-tuning the translation of the target image so that the position of the target image matches the position in the source image; fine-tuning the internal parameters of the target image so that the imaging quality of the target image reaches a preset value; and determining the standard projection point based on the 3D anchor point of the target image after rotation angle fine-tuning, translation fine-tuning and internal parameter fine-tuning. After calculating the standard projection point, the standard projection point is stored.

[0160] The operation process includes: performing fine-tuning processing such as cropping and scaling on the acquired initial image, performing pixel mapping on the fine-tuned initial image based on the homography matrix to obtain the image under the perspective of the virtual camera; and performing normalization operation on the image under the perspective of the virtual camera to obtain the virtual camera image.

[0161] Among them, pixel mapping of the fine-tuned initial image based on the homography matrix can be divided into online mode and offline mode. When the vehicle is in different modes, the methods used can be different. For example, in offline mode, the computer vision library is used to map the coordinates of each pixel in the initial image, and convert it from the original camera perspective to the coordinate position under the virtual camera perspective. In online mode, a parallel computing platform and programming model are used to map the coordinates of each pixel in the initial image, and convert it from the original camera perspective to the coordinate position under the virtual camera perspective.

[0162] Here, CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model. It allows developers to use powerful computing power to perform computationally intensive tasks. The main uses of CUDA include accelerating image processing, scientific computing, machine learning, artificial intelligence and other applications that require a large amount of parallel computing.

[0163] Here, OpenCV (Open Source Computer Vision Library) is an open source computer vision library. It provides a wide range of tools and functions for image processing and computer vision tasks, and can be used in many fields such as face recognition, object detection, image enhancement, video processing, etc.

[0164] In some implementations, performing a normalization operation on the image under the virtual camera perspective to obtain a virtual camera image may include at least one of the following:

[0165] Image size normalization: Adjusting images of different resolutions or sizes to a uniform size to ensure consistency in subsequent processing. This is usually achieved through scaling or cropping operations.

[0166] Normalization of color space: Convert or normalize the color space of an image to conform to a specific color model or range. For example, standardize the color values ​​of a three-primary color (Red Green Blue, RGB) image to a range of 0 to 1, or convert it to a grayscale image.

[0167] Pixel value normalization: Adjust the pixel values ​​of an image to a standardized range, such as scaling the pixel values ​​from a range of 0 to 255 to a range of 0 to 1. This helps improve the consistency and comparability of images in different processing steps.

[0168] Normalization of geometric transformation: Apply geometric transformations such as rotation, translation, and scaling to make the image consistent with the real camera perspective or expected perspective under the virtual camera perspective.

[0169] Normalization of contrast and brightness: Adjust the contrast and brightness of the image to ensure that different images have the same visual effect under the virtual camera perspective.

[0170] In this way, by using CUDA or OpenCV libraries, pixel mapping and image normalization operations based on homography matrices are implemented, significantly improving the efficiency and accuracy of image processing. Through normalization operations, image features such as brightness, contrast, color, and geometric shape are standardized, making images taken in different environments more consistent. This consistency not only reduces the interference of changes in external conditions on image processing and target detection, but also provides more reliable input data for subsequent processing steps. At the same time, the normalized input simplifies the training and application process of the model and improves the overall performance of the system.

[0171] Figure 5 is a schematic diagram of comparison between an initial image and a virtual camera image according to an embodiment of the present disclosure, such as Figure 5 As shown in the figure, the left image is the original image, and the right image is the virtual camera image obtained after processing based on the original image on the left. Obviously, compared with the original image on the left, the virtual camera image on the right is clearer and more accurate. Even when the difference in vehicle parameters is too large, the effect can be significantly improved by only passing through the virtual camera from the left image to the right image.

[0172] It should be understood that Figure 4 , Figure 5 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 4 , Figure 5Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0173] The present disclosure provides a target detection device, such as Figure 6 As shown, the target detection device may include: an image acquisition module 601, which is used to acquire a road image of the vehicle at the current moment taken by a camera, and the image is an initial image; an image processing module 602, which is used to perform pixel mapping on the initial image based on a homography matrix when the virtual camera is turned on to obtain a virtual camera image; a first detection module 603, which is used to input the virtual camera image into a pre-trained first detection model, perform detection processing on the virtual camera image, and obtain a first target detection result.

[0174] In some implementations, the image processing module 602 is further configured to: before pixel mapping is performed on the initial image based on the homography matrix, perform fine-tuning on the initial image to obtain an optimized initial image.

[0175] In some implementations, the image processing module 602 may include:

[0176] A mapping submodule is used to perform pixel mapping on the initial image based on the homography matrix to obtain an image from the perspective of a virtual camera;

[0177] The normalization submodule is used to normalize the image under the perspective of the virtual camera to obtain a virtual camera image.

[0178] In some embodiments, the target detection device further includes:

[0179] The first determination module ( Figure 6 (not shown), for determining a source projection point of a source image according to camera parameters;

[0180] The second determination module ( Figure 6 (not shown) for determining a standard projection point of a target image according to virtual camera parameters;

[0181] The third determination module ( Figure 6 ), which is used to perform Gaussian elimination according to the source projection points and the standard projection points to obtain the homography matrix.

[0182] In some implementations, the second determination module includes:

[0183] A first fine-tuning submodule, used for fine-tuning the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image;

[0184] A second fine-tuning submodule is used to perform translation fine-tuning on the target image so that the position of the target image matches the position in the source image;

[0185] The third fine-tuning submodule is used to perform internal parameter fine-tuning on the target image so that the imaging quality of the target image reaches a preset value;

[0186] The determination submodule is used to determine the standard projection point based on the 3D anchor point of the target image after rotation angle fine-tuning, translation fine-tuning and internal parameter fine-tuning.

[0187] In some implementations, the first fine-tuning submodule is used to:

[0188] Determine the angle at which the target image needs to be rotated relative to the source image based on the target point;

[0189] The target image is rotated by a corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

[0190] In some implementations, the second fine-tuning submodule is used to:

[0191] determining at least one reference point based on the source image and the target image;

[0192] Obtaining a translation vector based on a position difference of at least one reference point in the source image and the target image;

[0193] Move the destination image based on the translation vector so that the location of the destination image matches the location in the source image.

[0194] In some implementations, the third fine-tuning submodule is used to:

[0195] Get the camera's internal parameters;

[0196] Recalibrate or adjust the camera's intrinsic parameters according to the intrinsic parameter requirements of different application scenarios.

[0197] In some implementations, the determining submodule is used to:

[0198] Determine the projection matrix based on the camera's internal and external parameters;

[0199] Based on the coordinates of the 3D anchor point and the projection matrix, the standard projection point is obtained.

[0200] In some embodiments, the target detection device further includes:

[0201] The second detection module ( Figure 6 ), which is used to input the initial image into a pre-trained second detection model when the virtual camera is not turned on, perform detection processing on the initial image, and obtain a second target detection result.

[0202] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0203] The target detection device of the disclosed embodiment can enable the detection model to uniformly process image data under different vehicle models and different camera configurations, reduce system errors caused by differences in camera parameters, thereby ensuring the consistency of the image-to-3D position mapping relationship under multiple configuration conditions, and improving the accuracy of target detection, thereby making the environmental perception in the autonomous driving system more accurate and stable.

[0204] The present disclosure provides a scene diagram of a target detection method, such as Figure 7 shown.

[0205] As mentioned above, the target detection method provided by the embodiment of the present disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0206] Specifically, the electronic device can perform the following operations:

[0207] In response to the vehicle start operation, the camera is controlled to capture an image of the road where the vehicle is currently located, the image being the initial image;

[0208] When the virtual camera is turned on, the initial image is pixel mapped based on the homography matrix to obtain a virtual camera image;

[0209] Inputting the virtual camera image into a pre-trained first detection model, performing detection processing on the virtual camera image, and obtaining a first target detection result;

[0210] When the virtual camera is not turned on, the initial image is input into the pre-trained second detection model, and the initial image is detected and processed to obtain a second target detection result.

[0211] The first detection model and the second detection model can be obtained from a data source.

[0212] The data source may be a data storage device in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The data source may also represent a mobile device in various forms, such as a personal digital assistant, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0213] The vehicle adjusts its driving status and behavior in real time based on the target detection results, such as obstacle recognition and avoidance, traffic signal recognition and compliance, lane keeping and lane changing, speed control, emergency braking and obstacle avoidance, etc., to ensure driving safety and efficiency.

[0214] It should be understood that Figure 7 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 7 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0215] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0216] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0217] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0218] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0219] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0220] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the target detection method. For example, in some embodiments, the target detection method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the target detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the target detection method in any other appropriate manner (eg, by means of firmware).

[0221] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0223] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0225] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0226] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0227] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0228] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection method, applied to a vehicle, wherein the vehicle is equipped with a camera; wherein: The target detection method comprises: In response to a start-up operation of the vehicle, controlling the camera to capture an image of a road where the vehicle is currently located, wherein the image is an initial image; When the virtual camera is turned on, pixel mapping is performed on the initial image based on a homography matrix to obtain a virtual camera image; The virtual camera image is input into a pre-trained first detection model, and detection processing is performed on the virtual camera image to obtain a first target detection result.

2. The method according to claim 1, wherein: Before performing pixel mapping on the initial image based on the homography matrix, the method further includes: The initial image is fine-tuned to obtain an optimized initial image.

3. The method according to claim 1, wherein: The performing pixel mapping on the initial image based on the homography matrix to obtain a virtual camera image includes: Perform pixel mapping on the initial image based on the homography matrix to obtain an image under the perspective of a virtual camera; A normalization operation is performed on the image under the perspective of the virtual camera to obtain the virtual camera image.

4. The method according to claim 1, further comprising: Determine a source projection point of a source image according to camera parameters; Determine the standard projection point of the target image according to the virtual camera parameters; Gaussian elimination is performed according to the source projection point and the standard projection point to obtain the homography matrix.

5. The method according to claim 4, wherein: The step of determining the standard projection point of the target image according to the virtual camera parameters comprises: Fine-adjusting the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image; Performing a translation and fine-tuning on the target image so that the position of the target image matches the position in the source image; Performing internal parameter fine-tuning on the target image so that the imaging quality of the target image reaches a preset value; The standard projection point is determined based on the three-dimensional anchor point of the target image after the rotation angle fine-tuning, the translation fine-tuning and the internal parameter fine-tuning.

6. The method according to claim 5, wherein: The step of fine-tuning the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image includes: Determine the angle at which the target image needs to be rotated relative to the source image based on the target point; The target image is rotated by a corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

7. The method according to claim 5, wherein: The step of performing a translation fine adjustment on the target image so that the position of the target image matches the position in the source image includes: determining at least one reference point based on the source image and the target image; Obtaining a translation vector based on a position difference of the at least one reference point in the source image and the target image; The target image is moved based on the translation vector so that the position of the target image matches the position in the source image.

8. The method according to claim 5, wherein: The step of fine-tuning the target image by internal parameters so that the imaging quality of the target image reaches a preset value includes: Obtaining the intrinsic parameters of the camera; Recalibrate or adjust the camera's internal parameters according to the internal parameter requirements of different application scenarios.

9. The method according to claim 5, wherein: The determining of the standard projection point based on the three-dimensional anchor point of the target image after the rotation angle fine-tuning, the translation fine-tuning and the internal parameter fine-tuning comprises: Determine a projection matrix according to the intrinsic and extrinsic parameters of the camera; The standard projection point is obtained based on the coordinates of the three-dimensional anchor point and the projection matrix.

10. The method according to claim 1, further comprising: When the virtual camera is not turned on, the initial image is input into a pre-trained second detection model, and detection processing is performed on the initial image to obtain a second target detection result.

11. A target detection device, applied to a vehicle, wherein the vehicle is equipped with a camera; wherein: The target detection device comprises: An image acquisition module, used to acquire an image of the road where the vehicle is located at the current moment taken by the camera, wherein the image is an initial image; An image processing module, used for performing pixel mapping on the initial image based on a homography matrix to obtain a virtual camera image when the virtual camera is turned on; The first detection module is used to input the virtual camera image into a pre-trained first detection model, perform detection processing on the virtual camera image, and obtain a first target detection result.

12. The device according to claim 11, wherein The image processing module is further used to: before performing pixel mapping on the initial image based on the homography matrix, perform fine-tuning processing on the initial image to obtain an optimized initial image.

13. The device according to claim 11, wherein: The image processing module comprises: A mapping submodule, used for performing pixel mapping on the initial image based on the homography matrix to obtain an image under the perspective of a virtual camera; The normalization submodule is used to perform a normalization operation on the image under the perspective of the virtual camera to obtain the virtual camera image.

14. The apparatus according to claim 11, further comprising: A first determination module, used to determine a source projection point of a source image according to camera parameters; A second determination module is used to determine a standard projection point of a target image according to virtual camera parameters; The third determination module is used to perform Gaussian elimination according to the source projection point and the standard projection point to obtain the homography matrix.

15. The device according to claim 14, wherein: The second determining module includes: A first fine-tuning submodule, used for fine-tuning the rotation angle of the target image so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image; A second fine-tuning submodule, used for performing translation fine-tuning on the target image so that the position of the target image matches the position in the source image; A third fine-tuning submodule is used to perform internal parameter fine-tuning on the target image so that the imaging quality of the target image reaches a preset value; A determination submodule is used to determine the standard projection point based on the three-dimensional anchor point of the target image after the rotation angle fine-tuning, the translation fine-tuning and the internal parameter fine-tuning.

16. The device according to claim 15, wherein: The first fine-tuning submodule is used to: Determine the angle at which the target image needs to be rotated relative to the source image based on the target point; The target image is rotated by a corresponding angle so that the first coordinate system of the target image is completely aligned with the second coordinate system of the source image.

17. The device according to claim 15, wherein: The second fine-tuning submodule is used to: determining at least one reference point based on the source image and the target image; Obtaining a translation vector based on a position difference of the at least one reference point in the source image and the target image; The target image is moved based on the translation vector so that the position of the target image matches the position in the source image.

18. The device according to claim 15, wherein: The third fine-tuning submodule is used to: Obtaining the intrinsic parameters of the camera; Recalibrate or adjust the camera's internal parameters according to the internal parameter requirements of different application scenarios.

19. The device according to claim 15, wherein: The determining submodule is used to: Determine a projection matrix according to the intrinsic and extrinsic parameters of the camera; The standard projection point is obtained based on the coordinates of the three-dimensional anchor point and the projection matrix.

20. The apparatus according to claim 11, further comprising: The second detection module is used to input the initial image into a pre-trained second detection model when the virtual camera is not turned on, perform detection processing on the initial image, and obtain a second target detection result.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to perform a method according to any one of claims 1-10.

23. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 10 when executed by a processor.

Images