Image processing method, vehicle and computer readable storage medium

Abstract

The application discloses an image processing method, which comprises the following steps: acquiring an environment image of an environment where a vehicle is located; determining a target position of the vehicle in a bird's-eye view image according to a first position of a license plate in the environment image when the environment image comprises the license plate; and performing desensitization processing on the license plate in the bird's-eye view image based on the target position of the vehicle in the bird's-eye view image. Thus, the vehicle can determine the target position of the license plate in the bird's-eye view image according to the first position of the license plate in a target image captured by a target camera, so that the position detection of the license plate in the bird's-eye view image can be obtained through position transformation, the situation that the position detection of the license plate in the bird's-eye view image needs to be completely based on a computer vision scheme can be avoided, the accurate detection of the position of the license plate in the bird's-eye view image is ensured to a certain extent, and thus the desensitization processing of the license plate in the bird's-eye view image can be reliably completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method, a vehicle and a computer readable storage medium. BACKGROUND

[0002] In the related art, a vehicle can generate an aerial view after parking, so as to push the aerial view to a mobile phone of a vehicle owner, thereby providing a reference for the vehicle owner to find the vehicle. According to relevant laws and regulations, sensitive information such as a license plate in the aerial view needs to be anonymized before the aerial view can be transmitted externally. However, since the aerial view is distorted, there is a situation of low detection accuracy when detecting the license plate from the aerial view by a computer vision scheme. SUMMARY

[0003] The present application provides an image processing method, a vehicle and a computer readable storage medium.

[0004] The present application provides an image processing method, a vehicle and a computer readable storage medium.

[0005] An environment image of an environment where the vehicle is located is acquired.

[0006] In a case where the environment image includes a license plate, a target position of the license plate in an aerial view image is determined according to a first position of the license plate in the environment image.

[0007] The license plate in the aerial view image is desensitized based on the target position.

[0008] The present application provides an image processing method, a vehicle and a computer readable storage medium.

[0009] In this way, in the present application, the vehicle can determine the target position of the license plate in the aerial view image according to the first position of the license plate in the target image captured by the target camera, so that the position detection of the license plate in the aerial view image can be obtained by position transformation, which can avoid the situation that the position detection of the license plate in the aerial view image needs to be completely based on the computer vision scheme, thereby guaranteeing the accurate detection of the position of the license plate in the aerial view image to a certain extent, so as to reliably complete the desensitization of the license plate in the aerial view image.

[0010] In some embodiments of the present application, in a case where the environment image includes a license plate, a target position of the license plate in an aerial view image is determined according to a first position of the license plate in the environment image, including:

[0011] In a case where the environment image comprises a license plate, a coordinate system conversion is performed on a first position of the license plate in the environment image to determine a second position of the license plate in a camera coordinate system corresponding to the environment image.

[0012] According to a mapping of the second position in a predetermined bird's-eye coordinate system, the target position of the license plate in the bird's-eye image is determined.

[0013] In this way, the embodiments of the present application can convert the first position of the license plate in the environment image to the second position in the camera coordinate system through coordinate system conversion, and determine the target position of the license plate in the bird's-eye image based on the second position of the license plate in the camera coordinate system and the predetermined bird's-eye coordinate system, so that the determination of the target position is reliably performed.

[0014] In some embodiments of the present application, in a case where the environment image comprises a license plate, the coordinate system conversion is performed on a first position of the license plate in the environment image to determine a second position of the license plate in a camera coordinate system corresponding to the environment image, comprising:

[0015] In a case where the environment image comprises a license plate, a license plate coordinate system is constructed according to predetermined license plate size information;

[0016] A camera projection equation is constructed according to the license plate coordinate system and a predetermined camera intrinsic parameter matrix;

[0017] The camera projection equation is solved according to the first position and the license plate size information to obtain a first translation matrix and a first rotation matrix of the license plate coordinate system relative to the camera coordinate system;

[0018] The second position is determined based on the first position, the first translation matrix and the first rotation matrix.

[0019] In this way, the embodiments of the present application can determine the first conversion relationship between the coordinate system corresponding to the license plate and the camera coordinate system according to the size of the license plate, and then determine the second position of the license plate in the camera coordinate system based on the first position, the coordinate system corresponding to the license plate, the first conversion relationship and the camera coordinate system, so that the second position can be reliably determined.

[0020] In some embodiments of the present application, the target position of the license plate in the bird's-eye image is determined according to a mapping of the second position in a predetermined bird's-eye coordinate system, comprising:

[0021] A second translation matrix and a second rotation matrix of the camera coordinate system relative to the bird's-eye coordinate system are obtained;

[0022] determine a third position of the license plate in the bird's eye view coordinate system based on the second position, the second translation matrix and the second rotation matrix;

[0023] determine the target position of the license plate in the bird's eye view image according to the third position.

[0024] Thus, the embodiments of the present application can determine the second position of the license plate in the camera coordinate system to the third position in the bird's eye view coordinate system according to the second conversion relationship between the camera coordinate system and the bird's eye view coordinate system, and then determine the target position of the license plate in the bird's eye view image according to the third position of the license plate in the bird's eye view coordinate system, so that the target position can be reliably determined.

[0025] In some embodiments of the present application, the determining the target position of the license plate in the bird's eye view image according to the third position comprises:

[0026] obtain a fourth position of the camera optical center in the bird's eye view coordinate system;

[0027] determine a ground point position corresponding to a line formed by the license plate and the camera optical center in the bird's eye view coordinate system according to the third position and the fourth position;

[0028] determine the target position of the license plate in the bird's eye view image according to the ground point position.

[0029] Thus, the vehicle of the embodiments of the present application can determine the target position of the license plate in the bird's eye view image based on the ground point position corresponding to the line formed by the camera optical center and the vehicle in the bird's eye view coordinate system, so that the target position can be reliably determined.

[0030] In some embodiments of the present application, the determining the target position of the license plate in the bird's eye view image according to the ground point position comprises:

[0031] construct a conversion equation of the bird's eye view coordinate system relative to the bird's eye view image according to the ground point position, an image size of the bird's eye view image and a field of view range corresponding to the bird's eye view image;

[0032] substitute the third position into the conversion equation to obtain the target position of the license plate in the bird's eye view image.

[0033] Thus, the embodiments of the present application can determine the target position of the license plate in the bird's eye view image based on the ground point position, the image size of the bird's eye view image and the field of view range corresponding to the bird's eye view image, so that the target position can be reliably determined.

[0034] In some embodiments of the present application, the third position comprises a corner point position of the license plate in the bird's eye coordinate system, and the determining of the ground point position corresponding to the line formed by the license plate and the camera optical center in the bird's eye coordinate system according to the third position and the fourth position comprises:

[0035] solving the ground point position corresponding to the line formed by the camera optical center and the line according to the fourth position and the corner point position.

[0036] In this way, the vehicle of the embodiments of the present application can determine the position of the corner point of the license plate in the bird's eye image and the target position in the environment image based on the ground point position corresponding to the line formed by the corner point of the license plate and the camera optical center in the bird's eye coordinate system, so that the target position of the license plate can be reliably determined, and the desensitization processing of the license plate can be reliably performed.

[0037] In some embodiments of the present application, the method further comprises:

[0038] identifying the license plate and the first position of the license plate in the environment image based on the pre-trained identification model.

[0039] In this way, the vehicle of the embodiments of the present application can identify the license plate and the first position of the license plate in the environment image based on the pre-trained identification model, so that the license plate and the first position of the license plate can be accurately determined, and reliable data support can be provided for subsequent target position determination.

[0040] The embodiments of the present application provide a vehicle comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the image processing method described above.

[0041] The embodiments of the present application provide a computer readable storage medium storing a computer program, and when the computer program is executed by one or more processors, the image processing method described above is implemented.

[0042] The vehicle and the computer readable storage medium provided by the embodiments of the present application can determine the target position of the license plate in the bird's eye image according to the first position of the license plate in the target image captured by the target camera, so that the position detection of the license plate in the bird's eye image can be obtained through position transformation, and the situation that the position detection of the license plate in the bird's eye image needs to be completely based on the computer vision scheme can be avoided, which to some extent guarantees the accurate detection of the position of the license plate in the bird's eye image, and thus the desensitization processing of the license plate in the bird's eye image can be reliably completed.

[0043] Additional aspects and advantages of the embodiments of the present application will be set forth in part in the description which follows, and in part will be obvious from the description or can be learned by practice of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the references to the figures, in which:

[0045] Figure 1 Flowchart of image processing method in some embodiments of the present application;

[0046] Figure 2 Flowchart of image processing method in some embodiments of the present application;

[0047] Figure 3 Flowchart of image processing method in some embodiments of the present application;

[0048] Figure 4 Flowchart of image processing method in some embodiments of the present application;

[0049] Figure 5 Flowchart of image processing method in some embodiments of the present application;

[0050] Figure 6 Flowchart of image processing method in some embodiments of the present application;

[0051] Figure 7 Flowchart of image processing method in some embodiments of the present application;

[0052] Figure 8 Flowchart of image processing method in some embodiments of the present application;

[0053] Figure 9 Flowchart of image processing method in some embodiments of the present application;

[0054] Figure 10 Flowchart of image processing method in some embodiments of the present application;

[0055] Figure 11 Flowchart of image processing method in some embodiments of the present application. DETAILED DESCRIPTION

[0056] Embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein the same or like reference numerals are used throughout the drawing figures to refer to the same or like elements or components. The embodiments described below are merely examples for explaining the embodiments of the present application, and should not be interpreted in a limiting manner.

[0057] After the vehicle is parked, the vehicle can capture multiple surround view images, stitch the multiple surround view images to form an overhead view image, and push the surround view images and the overhead view image to a user's mobile phone, thereby providing a convenient reference for the user to find the vehicle.

[0058] Further, according to laws and regulations, after the vehicle end acquires the photographed image information, the vehicle end must anonymize the personal sensitive information (such as license plates) in the image before transmitting the image to the outside of the vehicle. Anonymization refers to a process in which personal information cannot be identified as a specific natural person and cannot be restored.

[0059] Further, for the surround view image, the shape and contour of the license plate in the surround view image are relatively clear, so the position of the license plate in the surround view image can be determined relatively simply, and the anonymization processing of the license plate in the surround view image can be reliably completed, or in other words, the desensitization processing of the license plate in the surround view image can be reliably completed.

[0060] In the overhead view image, the license plate in the overhead view image presents a perspective distortion shape because the overhead view image is stitched from multiple surround view images. Further, if the license plate detection in the overhead view image is implemented based on a computer vision scheme, or in other words, if the license plate detection in the overhead view image is implemented by a trained neural network model or a specific algorithm, the shape and contour of the license plate in the overhead view image are significantly different from the shape and contour of a real license plate, which makes it difficult to achieve the expected accuracy of license plate detection.

[0061] In addition, if a top view sample including a license plate is made to train a neural network model, so that the trained neural network model can specifically identify the license plate in the overhead view image, there are problems of high difficulty in making the top view sample and low efficiency in making the top view sample. Moreover, when the neural network model is deployed and called locally by the vehicle to identify the license plate in the overhead view image, the neural network model requires high computing power support, and it takes a certain amount of time for the neural network model to infer the position of the license plate from the overhead view image.

[0062] Based on the above possible problems, please refer to Figure 1 The image processing method provided by the embodiments of the present application comprises:

[0063] 01: acquiring an environment image of an environment in which a vehicle is located;

[0064] 02: in a case where the environment image includes a license plate, determining a target position of the license plate in an overhead view image according to a first position of the license plate in the environment image;

[0065] 03: performing desensitization processing on the license plate in the overhead view image based on the target position.

[0066] The image processing apparatus provided by the embodiments of the present application can implement the image processing method provided by the embodiments of the present application. Specifically, the image processing apparatus comprises an acquisition module, a determination module and a processing module. The acquisition module is configured to acquire an environment image of an environment in which a vehicle is located. The determination module is configured to, when the environment image comprises a license plate, determine a target position of the license plate in a bird's-eye view image according to a first position of the license plate in the environment image. The processing module is configured to perform desensitization processing on the license plate in the bird's-eye view image based on the target position.

[0067] The embodiments of the present application also provide a vehicle comprising a memory and a processor. The image processing method provided by the embodiments of the present application can be implemented by the vehicle provided by the embodiments of the present application. Specifically, the memory stores a computer program, and the processor is configured to acquire an environment image of an environment in which a vehicle is located, and configured to, when the environment image comprises a license plate, determine a target position of the license plate in a bird's-eye view image according to a first position of the license plate in the environment image, and configured to perform desensitization processing on the license plate in the bird's-eye view image based on the target position.

[0068] Specifically, in the embodiments of the present application, the vehicle can acquire an environment image of an environment in which the vehicle is located by capturing the environment around the vehicle through a surround-view camera such as a fisheye camera. When it is confirmed that the environment image comprises a license plate, the vehicle can determine a target position of the license plate in a bird's-eye view image based on a first position of the license plate in the environment image, and perform desensitization processing on the license plate in the bird's-eye view image based on the target position, thereby completing the anonymization of the license plate in the bird's-eye view image. Then, the vehicle can send the bird's-eye view image to a user terminal, or in other words, to a car owner's mobile phone, thereby providing a reference for the car owner to find the vehicle.

[0069] It can be understood that the environment image in the embodiments of the present application can be understood as an image obtained by capturing a specific direction outside the vehicle cabin space based on a surround-view camera such as a fisheye camera.

[0070] Optionally, in some embodiments of the present application, the vehicle is provided with four surround-view cameras, i.e., a front surround-view camera, a left surround-view camera, a rear surround-view camera and a right surround-view camera. Then, the vehicle can capture environment images of four directions, i.e., a front direction, a left direction, a rear direction and a right direction outside the vehicle cabin space through the front surround-view camera, the left surround-view camera, the rear surround-view camera and the right surround-view camera, respectively. It can be understood that the above-mentioned front direction, left direction, rear direction and right direction can be understood as front direction, left direction, rear direction and right direction parallel to the horizontal plane and centered on the vehicle (or origin).

[0071] It is also understood that, in the case of the vehicle in the embodiments of this application, when the environmental image includes a license plate, or when the vehicle detects the presence of a license plate in the environmental image, the vehicle can perform corresponding processing or conversion based on the position of the license plate in the detected environmental image, thereby obtaining the target position of the license plate in the bird's-eye view image.

[0072] As an example, to more clearly illustrate the implementation of this application, please refer to the parameters. Figure 2 and Figure 3 , Figure 2 and Figure 3 These are all schematic diagrams illustrating application scenarios in certain embodiments of this application. Figure 2 This can be understood as an environmental image in the embodiments of this application. Figure 2 The four coordinates (u1,v1), (u2,v2), (u3,v3), and (u4,v4) can be understood as the coordinates of the four corner points of the license plate in the environmental image, which is also the first position of the license plate in the environmental image in the embodiment of this application.

[0073] as well as, Figure 3 This can be understood as a bird's-eye view image in the embodiments of this application. Figure 3 The four coordinates and This can be understood as the coordinates of the four corner points of the license plate in the bird's-eye view image, which is also the target position of the license plate in the bird's-eye view image in the embodiments of this application.

[0074] Therefore, the vehicle in the embodiment of this application can perform desensitization processing on the license plate in the environmental image based on the first position of the license plate in the environmental image to achieve anonymization (e.g., Figure 2 (As shown). Furthermore, in this embodiment, the vehicle can also undergo desensitization processing on the license plate within the bird's-eye view image based on the target location of the license plate within the image to achieve anonymization.

[0075] It is understood that the vehicle in the embodiment of this application determines the target position of the license plate in the bird's-eye view image based on the first position of the license plate in the environmental image, rather than directly detecting the license plate in the bird's-eye view image through a trained neural network model to determine the position of the license plate in the bird's-eye view image. Therefore, it can avoid the situation where the accuracy of the neural network model in detecting the license plate is difficult to achieve as expected due to the large difference between the shape, outline and other features of the license plate in the bird's-eye view image and the shape, outline and other features of the real license plate.

[0076] Meanwhile, the need to make a top-down sample including a license plate to train a neural network model is avoided, and the need for a vehicle to locally deploy and call a neural network model is also avoided, so that the vehicle does not need to call and run a neural network model with a high calling and running cost to determine the position of the license plate in the bird's eye view, and the vehicle can efficiently determine the position of the license plate in the bird's eye view.

[0077] In summary, in the embodiments of the present application, the vehicle can determine the target position of the license plate in the bird's eye view image with distortion according to the first position of the license plate in the target image captured by the target camera, so that the position detection of the license plate in the bird's eye view image can be obtained through position transformation, and the position detection of the license plate in the bird's eye view image can be completely based on a computer vision scheme, which to some extent guarantees the accurate detection of the position of the license plate in the bird's eye view image, thereby reliably completing the desensitization processing of the license plate in the bird's eye view image.

[0078] Please refer to Figure 4 In some embodiments of the present application, step 02 includes:

[0079] 020: In the case where the environment image includes a license plate, performing coordinate system conversion on the first position of the license plate in the environment image to determine the second position of the license plate in the camera coordinate system corresponding to the environment image;

[0080] 021: According to the mapping of the second position in the predetermined bird's eye coordinate system, determining the target position of the license plate in the bird's eye view image.

[0081] The determination module of the embodiments of the present application is also used to, in the case where the environment image includes a license plate, perform coordinate system conversion on the first position of the license plate in the environment image to determine the second position of the license plate in the camera coordinate system corresponding to the environment image, and is used to determine the target position of the license plate in the bird's eye view image according to the mapping of the second position in the predetermined bird's eye coordinate system.

[0082] The processor of the embodiments of the present application is also used to, in the case where the environment image includes a license plate, perform coordinate system conversion on the first position of the license plate in the environment image to determine the second position of the license plate in the camera coordinate system corresponding to the environment image, and is used to determine the target position of the license plate in the bird's eye view image according to the mapping of the second position in the predetermined bird's eye coordinate system.

[0083] Specifically, in the embodiments of the present application, the vehicle can convert the first position of the license plate in the environment image to the target position of the license plate in the bird's eye view image based on coordinate system conversion when it is confirmed that the environment image includes a license plate.

[0084] It can be understood that in the embodiments of the present application, the first position of the license plate in the environment image can be understood as the pixel coordinates of the license plate in the pixel coordinate system corresponding to the environment image.

[0085] And, the camera coordinate system corresponding to the environment image can be understood as the camera coordinate system corresponding to the camera which takes the "environment image containing the license plate". Therefore, the second position of the license plate can be understood as the coordinate of the vehicle in the camera coordinate system.

[0086] For example, in some embodiments of the present application, the vehicle is equipped with four surround view cameras, including a front surround view camera, a left surround view camera, a rear surround view camera, and a right surround view camera. The front surround view camera corresponds to the first camera coordinate system, the left surround view camera corresponds to the second camera coordinate system, the rear surround view camera corresponds to the third camera coordinate system, and the right surround view camera corresponds to the fourth camera coordinate system. When the vehicle confirms that the environment image taken by the front surround view camera contains a license plate, the vehicle can complete the coordinate transformation from the first position to the target position based on the first camera coordinate system.

[0087] It can also be understood that the vehicle can convert multiple environment images to the world coordinate system (or vehicle coordinate system), or convert multiple environment images to the bird's eye view coordinate system in the embodiments of the present application, and then stitch the converted multiple environment images to obtain a bird's eye view.

[0088] For example, in some embodiments of the present application, the vehicle is equipped with four surround view cameras, including a front surround view camera, a left surround view camera, a rear surround view camera, and a right surround view camera. The front surround view camera is used to take a front view image, the left surround view camera is used to take a left view image, the rear surround view camera is used to take a rear view image, and the right surround view camera is used to take a right view image. The vehicle can convert the front view image, the left view image, the rear view image, and the right view image to the bird's eye view coordinate system, and then stitch the front view image, the left view image, the rear view image, and the right view image in the bird's eye view coordinate system to obtain a bird's eye view image.

[0089] Therefore, in the embodiments of the present application, the target position of the license plate in the bird's eye view image can be understood as the position of the license plate projected from the environment image to the bird's eye view, or the position of the license plate in the pixel coordinate system corresponding to the bird's eye view image.

[0090] In this way, the embodiments of the present application can convert the first position of the license plate in the environment image to the second position in the camera coordinate system, and determine the target position of the license plate in the bird's eye view image based on the second position of the license plate in the camera coordinate system and the predetermined bird's eye view coordinate system, so that the determination of the target position can be reliably performed.

[0091] Please refer to Figure 5 In some embodiments of the present application, step 020 includes:

[0092] 0200: In the case that the environment image contains a license plate, constructing a license plate coordinate system according to predetermined license plate size information;

[0093] 0201: constructing a camera projection equation according to the license plate coordinate system and a predetermined camera intrinsic matrix;

[0094] 0202: solving the camera projection equation according to the first position and the license plate size information to obtain a first translation matrix and a first rotation matrix of the license plate coordinate system relative to the camera coordinate system;

[0095] 0203: determining the second position based on the first position, the first translation matrix and the first rotation matrix.

[0096] The determining module of the embodiment of the present application is further configured to, in the case that the environment image comprises a license plate, construct a license plate coordinate system according to predetermined license plate size information, and configured to construct a camera projection equation according to the license plate coordinate system and a predetermined camera intrinsic matrix, and configured to solve the camera projection equation according to the first position and the license plate size information to determine a first translation matrix and a first rotation matrix of the license plate coordinate system relative to the camera coordinate system, and configured to determine the second position based on the first position, the first translation matrix and the first rotation matrix.

[0097] The processor of the embodiment of the present application is further configured to, in the case that the environment image comprises a license plate, construct a license plate coordinate system according to predetermined license plate size information, and configured to construct a camera projection equation according to the license plate coordinate system and a predetermined camera intrinsic matrix, and configured to solve the camera projection equation according to the first position and the license plate size information to obtain a first translation matrix and a first rotation matrix of the license plate coordinate system relative to the camera coordinate system, and configured to determine the second position based on the first position, the first translation matrix and the first rotation matrix.

[0098] It can be understood that the size of the license plate is fixed, and the center of the license plate is taken as the origin O P A planar rectangular coordinate system O P -X P Y P Z P The positions of the four corner points of the license plate in the world coordinate system are fixed, or in other words, the three-dimensional coordinates of the four corner points of the license plate in the planar rectangular coordinate system O P -X P Y P Z P are fixed. It can be understood that the planar rectangular coordinate system O P -X P Y P Z P may be the license plate coordinate system in the embodiment of the present application.

[0099] For a clearer description of the embodiments of the present application, please refer to Figure 6 , Figure 6A schematic diagram of a license plate coordinate system in some embodiments of the present application. Figure 6 The above-mentioned planar rectangular coordinate system O P -X P Y P Z P Further, since the plane in which the license plate is located coincides with the plane of the O P -X P Y P plane, and assuming that the length of the license plate is W and the width is L, it can be known that the three-dimensional coordinates (x k , y k , z k ) of the four corner points of the license plate are (-W / 2, L / 2, 0), (W / 2, L / 2, 0), (W / 2, -L / 2, 0), and (-W / 2, -L / 2, 0), respectively. It can be understood that k takes the values 0, 1, 2, and 4.

[0100] Further, since the coordinates (u k , v k ) of the license plate in the pixel coordinate system are known, or in other words, the first position of the license plate in the environment image is known, and the first position of the license plate in the environment image is obtained by projecting the position of the license plate in the license plate coordinate system, (u k , v k ) and the above-mentioned (x k , y k , z k ) are a set of 3D-2D (3Dimensions-2Dimensions) point pairs, and therefore, the following camera projection equation can be constructed, that is:

[0101]

[0102] In the formula, u k and v k are the coordinates of the license plate on the two axes of the pixel coordinate system, that is, the first position of the license plate; x k , y k , z k are the coordinates of the license plate on the three axes in the license plate coordinate system; K i is the intrinsic matrix of the i-th camera, that is, the camera coordinate system corresponding to the camera that “captures the environment image containing the license plate”; s is a scale factor; R represents the first rotation matrix to be solved; and T represents the first translation matrix to be solved. It can be understood that the first rotation matrix R and the first translation matrix T can be understood as the relative pose of the license plate and the camera, or as the extrinsic parameters of the i-th camera, and can represent the conversion relationship of the camera coordinate system relative to the license plate coordinate system.

[0103] Further, since the license plate coincides with the O P -X PY P are coplanar, thus z k = 0, and thus the camera projection equation can be converted to the following equation:

[0104]

[0105] where r0represents the first column vector in R, r1represents the second column vector in R, K i · [r0r1T] represents O P -X P Y P Z P a point in space to the pixel plane. Let H = K i · [r0r1T], then we have:

[0106]

[0107] Expanding, we have:

[0108]

[0109] Further, by eliminating the scale factor s, we have:

[0110]

[0111] Further, by solving the equations with the four 3D-2D point pairs, we can solve H = K i · [r0r1T], and thus we have:

[0112]

[0113] It can be understood that K i can be determined in advance or offline calibrated, and thus by constraining ||r0|| = 1 and ||r1|| = 1, we can solve the scale factor s and the components of the first translation matrix T x , T y , and T z in x, y, and z. In addition, the third column vector r2corresponding to the first rotation matrix R can be expressed by the following equation:

[0114] r2= r0x r1

[0115] Based on this, the first rotation matrix R can be solved.

[0116] Further, for the license plate in the environment image captured by the ith camera, the first position (u k , v k ) of the license plate in the environment image can be used to determine the plane rectangular coordinate system O P -X PY P Z P under the three-dimensional coordinates (x k ,y k ,z k ) solved by the foregoing several formulas, i.e. the first rotation matrix R and the first translation matrix T of the i-th camera, the three-dimensional coordinates (x P -X P Y P Z P under the plane rectangular coordinate system O k ,y k ,z k ) can be mapped to the three-dimensional coordinates (x k C ,y k C ,z k C ) under the camera coordinate system of the i-th camera, i.e. the second position in the embodiment of the application. For more clearly illustrating the embodiment of the application, please refer to Figure 7 , Figure 7 for the schematic diagram of the camera coordinate system in some embodiments of the application.

[0117] and the three-dimensional coordinates (x P -X P Y P Z P under the plane rectangular coordinate system O k ,y k ,z k ) are converted to the three-dimensional coordinates (x k C ,y k C ,z k C ) of the camera coordinate system, i.e. the process of converting the position of the license plate under the license plate coordinate system to the second position (x k C ,y k C ,z k C ) under the camera coordinate system can refer to the following formula, i.e.:

[0118]

[0119] Thus, the embodiment of the application can determine the first rotation matrix and the first translation matrix of the license plate coordinate system and the camera coordinate system according to the size of the license plate, and further can determine the second position of the license plate under the camera coordinate system based on the first position, the license plate coordinate system, the first rotation matrix, the first translation matrix and the camera coordinate system.

[0120] Please refer to Figure 8In some embodiments of the present application, step 021 comprises:

[0121] 0210: obtaining a second translation matrix and a second rotation matrix of the camera coordinate system relative to the bird's eye view coordinate system;

[0122] 0211: determining a third position of the license plate in the bird's eye view coordinate system based on the second position, the second translation matrix and the second rotation matrix;

[0123] 0212: determining a target position of the license plate in the bird's eye view image according to the third position.

[0124] The processor of the embodiments of the present application is also used to obtain a second translation matrix and a second rotation matrix of the camera coordinate system relative to the bird's eye view coordinate system, and to determine a third position of the license plate in the bird's eye view coordinate system based on the second position, the second translation matrix and the second rotation matrix, and to determine a target position of the license plate in the bird's eye view image according to the third position.

[0125] Specifically, the vehicle of the embodiments of the present application is also used to obtain a second translation matrix and a second rotation matrix of the camera coordinate system relative to the bird's eye view coordinate system, and to determine a third position of the license plate in the bird's eye view coordinate system based on the second position, the second translation matrix and the second rotation matrix, and to determine a target position of the license plate in the bird's eye view image according to the third position.

[0126] For example, in some embodiments of the present application, the vehicle is installed with four surround view cameras, i.e. a front surround view camera, a left surround view camera, a rear surround view camera and a right surround view camera, the transformation relationship information between the camera coordinate system corresponding to the front surround view camera and the bird's eye view coordinate system comprises T1 and R1, the transformation relationship information between the camera coordinate system corresponding to the left surround view camera and the bird's eye view coordinate system comprises T2 and R2, the transformation relationship information between the camera coordinate system corresponding to the rear surround view camera and the bird's eye view coordinate system comprises T3 and R3, and the transformation relationship information between the camera coordinate system corresponding to the right surround view camera and the bird's eye view coordinate system comprises T4 and R4, then when the vehicle confirms that the environment image captured by the front surround view camera contains a license plate, the vehicle can complete the conversion from the second position to the target position according to T1 and R1.

[0127] It can be understood that the vehicle of the embodiments of the present application can map the environment images of each camera to the bird's eye view coordinate system based on the conversion relationship between each camera coordinate system and the bird's eye view coordinate system, and stitch each environment image in the bird's eye view coordinate system to obtain a bird's eye view image.

[0128] It can also be understood that the bird's eye view coordinate system in the embodiments of the present application can be understood as a coordinate system describing the environment where the vehicle is located from a virtual overhead perspective, and taking the vehicle as the origin, taking the ground plane (or horizontal plane) where the vehicle is located as the XY plane, and taking the direction perpendicular to the ground plane and vertically upward as the Z axis direction. For a clearer description of the embodiments of the present application, please refer toFigure 9 , Figure 9 This is a schematic diagram of a bird's-eye view coordinate system in some embodiments of this application. It is understood that... Figure 9 The O shown W -X W Y W Z W This can be understood as the bird's-eye view coordinate system of the embodiments of this application.

[0129] Furthermore, if the environmental image captured by the i-th camera includes a license plate, the process of transforming the second position of the license plate in the camera coordinate system corresponding to the i-th camera to the third position in the bird's-eye view coordinate system can be referred to in the following formula:

[0130]

[0131] In the formula, R i T represents the second rotation matrix of the camera coordinate system corresponding to the i-th camera relative to the bird's-eye view coordinate system. i Let represent the second translation matrix of the camera coordinate system corresponding to the i-th camera relative to the bird's-eye view coordinate system. The second rotation matrix and the second translation matrix can be pre-calibrated.

[0132] Furthermore, in the license plate coordinate system O P -X P Y P Z P The first rotation matrix for transforming to the camera coordinate system corresponding to the i-th camera is R, and the license plate coordinate system is O. P -X P Y P Z P When the first translation matrix for transforming to the camera coordinate system corresponding to the i-th camera is T, the vehicle moves from the license plate coordinate system O P -X P Y P Z P Position transformation to the third position in the bird's-eye view coordinate system (x k C ,y k C ,z k C The process can be referred to in the following formula:

[0133]

[0134] Thus, the embodiments of this application can determine the second position of the license plate in the camera coordinate system and transform it to the third position in the bird's-eye view coordinate system based on the second translation matrix and the second rotation matrix of the camera coordinate system and the bird's-eye view coordinate system in advance. Then, the target position of the license plate in the bird's-eye view image can be determined based on the third position of the license plate in the bird's-eye view coordinate system.

[0135] Referring to Figure 10 In some embodiments of the present application, step 0212 comprises:

[0136] 02120: obtaining a fourth position of the camera optical center in the bird's eye coordinate system;

[0137] 02121: determining, according to the third position and the fourth position, a ground point position corresponding to the line formed by the license plate and the camera optical center in the bird's eye coordinate system;

[0138] 02122: determining the target position of the license plate in the bird's eye image according to the ground point position.

[0139] The determination module of the embodiments of the present application is also used to obtain a fourth position of the camera optical center in the bird's eye coordinate system, and to determine, according to the third position and the fourth position, a ground point position corresponding to the line formed by the license plate and the camera optical center in the bird's eye coordinate system, and to determine the target position of the license plate in the bird's eye image according to the ground point position.

[0140] The processor of the embodiments of the present application is also used to obtain a fourth position of the camera optical center in the bird's eye coordinate system, and to determine, according to the third position and the fourth position, a ground point position corresponding to the line formed by the license plate and the camera optical center in the bird's eye coordinate system, and to determine the target position of the license plate in the bird's eye image according to the ground point position.

[0141] Specifically, in order to determine the two-dimensional coordinate position of the license plate in the two-dimensional bird's eye image, i.e. the target position, the vehicle of the embodiments of the present application can determine the line formed by the license plate and the camera optical center in the bird's eye coordinate system based on the third position of the license plate in the bird's eye coordinate system and the mapping of the optical center of the camera that captures the license plate in the bird's eye coordinate system, i.e. the fourth position, so as to determine the ground point coordinate of the line formed by the license plate and the camera optical center in the bird's eye coordinate system, and thus determine the position of the license plate in the bird's eye image, i.e. the target position.

[0142] It can be understood that, based on the ground point position corresponding to the line formed by the license plate and the camera optical center in the bird's eye coordinate system, the position of the license plate in the two-dimensional bird's eye image based on the optical center can be estimated.

[0143] It can also be understood that the fourth position of the camera optical center in the bird's eye coordinate system can be obtained through pre-calibration.

[0144] In this way, the vehicle of the embodiments of the present application can determine the target position of the license plate in the bird's eye image based on the ground point coordinate corresponding to the line formed by the camera optical center and the vehicle in the bird's eye coordinate system, so that the target position can be reliably determined.

[0145] Referring to Figure 11 In some embodiments of the present application, step 02122 comprises:

[0146] 021220: Based on the ground point coordinates, the image size of the bird's-eye view, and the field of view corresponding to the bird's-eye view, construct the transformation equation of the bird's-eye view coordinate system relative to the bird's-eye view image;

[0147] 021221: Substitute the third position into the transformation equation to obtain the target position of the license plate in the bird's-eye view image.

[0148] The determination module in this application is further used to construct a transformation equation of the bird's-eye view coordinate system relative to the bird's-eye view image based on the grounding point coordinates, the image size of the bird's-eye view image, and the field of view corresponding to the bird's-eye view image, and to substitute the third position into the transformation equation to obtain the target position of the license plate in the bird's-eye view image.

[0149] The processor in this embodiment is further configured to construct a transformation equation of the bird's-eye view coordinate system relative to the bird's-eye view image based on the ground point coordinates, the image size of the bird's-eye view image, and the field of view corresponding to the bird's-eye view image, and to substitute the third position into the transformation equation to obtain the target position of the license plate within the bird's-eye view image.

[0150] Specifically, let the camera's optical center be (u0, v0), and the ground point coordinates be... The image size of the bird's-eye view is I X and I Y The field of view corresponding to the bird's-eye view image is D. X and D Y I X I represents the pixel width of the bird's-eye view image. Y D represents the pixel height of a bird's-eye view image. X I represents the width of the field of view corresponding to the bird's-eye view image. X The height represents the field of view corresponding to the bird's-eye view image. Based on the ground point coordinates, the target location from the third location to the license plate within the bird's-eye view image can be constructed. The transformation equation is:

[0151]

[0152] Understandably, a bird's-eye view can be understood as an image obtained by taking a picture of the vehicle's environment from a top-down angle using a virtual camera. Therefore, the aforementioned I... X I Y D X and D Y This can be understood as the internal parameters of this virtual camera.

[0153] What is also understandable is that when placing the third position (x) k C ,y k C ,z kC ) into the above formula, the pixel coordinates of the license plate in the bird's eye view image can be obtained.

[0154] Thus, the target position of the license plate in the bird's eye view image can be determined based on the ground point coordinates, the image size of the bird's eye view image, and the field of view range corresponding to the bird's eye view image, so that the target position can be reliably determined.

[0155] In some embodiments of the present application, the third position includes the corner point positions of the corner points of the license plate in the bird's eye coordinate system, and the step 02121 includes:

[0156] According to the fourth position and the corner point positions, the ground point positions corresponding to the lines formed by the camera optical center and the lines are solved.

[0157] The determination module of the embodiments of the present application is also used to solve the ground point positions corresponding to the lines formed by the camera optical center and the lines according to the fourth position and the corner point positions.

[0158] The processor of the embodiments of the present application is also used to solve the ground point positions corresponding to the lines formed by the camera optical center and the lines according to the fourth position and the corner point positions.

[0159] Specifically, in the case that the first position of the license plate includes the positions of each corner point of the license plate in the environment image, the second position of the license plate includes the positions of each corner point of the license plate in the camera coordinate system, and the third position of the license plate includes the positions of each corner point of the license plate in the bird's eye coordinate system, the vehicle of the embodiments of the present application can determine the lines formed by the camera optical center and each corner point of the license plate in the bird's eye coordinate system by using the corner point positions of each corner point of the license plate in the bird's eye coordinate system and the fourth position of the camera optical center, and determine the ground point positions corresponding to each corner point of the license plate in the bird's eye coordinate system.

[0160] Further, the vehicle can determine the positions of each corner point of the license plate in the bird's eye view image according to the ground point positions corresponding to each corner point, so as to obtain the target position of the license plate in the embodiments of the present application.

[0161] Further, the vehicle can complete the desensitization processing of the license plate in the bird's eye view image based on the positions of each corner point of the license plate in the bird's eye view image, as shown in Figure 3 and so as to realize the anonymization of the license plate or the license plate number.

[0162] ​Therefore, the vehicle of the embodiment of the present application can determine the position of the corner point of the license plate in the bird's eye view image and the target position based on the position of the grounding point corresponding to the line formed by the corner point of the license plate and the optical center of the camera in the bird's eye view coordinate system, so that the target position of the license plate can be reliably determined, and the desensitization processing of the license plate can be reliably performed.

[0163] In some embodiments of the present application, the image processing method further comprises:

[0164] The license plate and the first position of the license plate in the environment image are identified based on the pre-trained recognition model.

[0165] The image processing device of the embodiment of the present application further comprises an identification module. The identification module is configured to identify the license plate and the first position of the license plate in the environment image based on the pre-trained recognition model.

[0166] The processor of the embodiment of the present application is further configured to identify the license plate and the first position of the license plate in the environment image based on the pre-trained recognition model.

[0167] Specifically, the vehicle in the embodiment of the present application can be equipped with a pre-trained target detection model, i.e., a recognition model, to perform target detection on the environment image captured by the camera of the vehicle. In the case that the detection result output by the recognition model is the license plate and the first position of the license plate, the vehicle can determine the target position of the license plate in the bird's eye view image based on the first position of the license plate.

[0168] It can be understood that in the embodiment of the present application, the shapes or contours of the objects in the environment image captured by the camera of the vehicle are relatively normal, or in other words, the distortion degree of the objects in the environment image is relatively low, so that the vehicle can efficiently and accurately detect the license plate through the recognition model.

[0169] Therefore, the vehicle of the embodiment of the present application can identify the license plate and the first position of the license plate in the environment image based on the pre-trained recognition model, so that the license plate and the first position of the license plate can be accurately determined, and reliable data support can be provided for subsequent target position determination.

[0170] Optionally, in some embodiments of the present application, in the case that the license plates included in the environment images captured by two or more cameras of the vehicle are actually the same, for example, the environment image captured by the front-view camera includes a license plate, the environment image captured by the left-view camera also includes a license plate, and the vehicles in the two environment images are actually the same, the vehicle of the embodiment of the present application can determine the target position of the license plate in the bird's eye view image based on the first position of the license plate in the environment image captured by each camera, and determine the distance relationship of the target positions of the license plates captured by different cameras.

[0171] For example, in the case that the license plate is included in the environment image captured by the front-view camera and the license plate is also included in the environment image captured by the left-view camera, the environment image captured by the front-view camera is P1, the license plate included in the environment image P1 is T1, the environment image captured by the left-view camera is P2, and the license plate included in the environment image P2 is T2, the vehicle can calculate or determine the distance between the target position of T1 and the target position of T2 after determining the target positions of T1 and T2.

[0172] Further, if the distance is less than the preset value, it indicates that T1 in P1 and T2 in P2 are the same license plate, and further, the vehicle can complete the desensitization processing of the license plate in the bird's eye view according to any one of the target position of T1 and the target position of T2.

[0173] Conversely, if the distance between the target position of T1 and the target position of T2 is greater than or equal to the preset value, the vehicle can perform desensitization processing on T1 in the bird's eye view based on the target position of T1, and perform desensitization processing on T2 in the bird's eye view based on the target position of T2.

[0174] The application also provides a computer readable storage medium storing a computer program, which, when executed by one or more processors, implements the image processing method described above.

[0175] In the description of the present specification, the description referring to the terms "specifically", "further", "particularly", "can be understood", and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not mean to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0176] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing one or more steps in a set of steps performed by one or more machines or devices that operate in accordance with the instructions or codes of the computer-readable medium, and the preferred embodiments of the present application include additional implementations in which the set of steps are performed in an order different from the order shown or discussed, including substantially concurrently or in reverse order, and including additional or fewer steps performed in one or more of the methodologies. The set of steps performed in the preferred embodiments of the present application can be understood from the functions performed in these steps by one skilled in the art, and the functions described and claimed herein define the scope of the preferred embodiments of the present application.

[0177] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring an environment image of an environment in which a vehicle is located; in a case where the environment image comprises a license plate, determining a target position of the license plate in an aerial view image according to a first position of the license plate in the environment image; based on the target position, performing desensitization processing on the license plate in the aerial view image; the case where the environment image comprises a license plate, the first position of the license plate in the environment image comprises: in a case where the environment image comprises a license plate, determining a position of the license plate in a license plate coordinate system according to a first position of the license plate in the environment image, wherein the license plate coordinate system is determined according to pre-determined license plate size information; determining the position of the license plate in the camera coordinate system according to the position of the license plate in the license plate coordinate system and the conversion relationship between the license plate coordinate system and the camera coordinate system; determining the target position of the license plate in the aerial view image according to the position of the license plate in the camera coordinate system; the case where the environment image comprises a license plate, the first position of the license plate in the environment image comprises: in a case where the distance between the first target position and the second target position is greater than or equal to a preset value, performing desensitization processing on the license plate in the aerial view image according to the first target position and the second target position, wherein the first target position is determined according to a first position of the license plate in one of the environment images, and the second target position is determined according to a first position of the license plate in another of the environment images; in a case where the distance is less than the preset value, performing desensitization processing on the license plate in the aerial view image according to the first target position or the second target position.

2. The method of claim 1, wherein, the case where the environment image comprises a license plate, the first position of the license plate in the environment image comprises: in a case where the environment image comprises a license plate, performing coordinate system conversion on the first position of the license plate in the environment image to determine a second position of the license plate in a camera coordinate system corresponding to the environment image; determining the target position of the license plate in the aerial view image according to the mapping of the second position in a pre-determined aerial coordinate system.

3. The method of claim 2, wherein, the case where the environment image comprises a license plate, the first position of the license plate in the environment image comprises: in a case where the environment image comprises a license plate, constructing the license plate coordinate system according to pre-determined license plate size information; constructing a camera projection equation according to the license plate coordinate system and pre-determined camera intrinsic parameter matrix; solving the camera projection equation according to the first position and the license plate size information to obtain a first translation matrix and a first rotation matrix of the license plate coordinate system relative to the camera coordinate system; determining the second position based on the first position, the first translation matrix and the first rotation matrix.

4. The method of claim 2, wherein, The method further comprises: obtaining a second translation matrix and a second rotation matrix of the camera coordinate system relative to the bird's-eye coordinate system; determining a third position of the license plate in the bird's-eye coordinate system based on the second position, the second translation matrix and the second rotation matrix; determining the target position of the license plate in the bird's-eye image based on the third position.

5. The method of claim 4, wherein, The method further comprises: obtaining a fourth position of a camera optical center in the bird's-eye coordinate system; determining a ground point position corresponding to a line formed by the license plate and the camera optical center in the bird's-eye coordinate system based on the third position and the fourth position; determining the target position of the license plate in the bird's-eye image based on the ground point position.

6. The method of claim 5, wherein, The method further comprises: constructing a conversion equation of the bird's-eye coordinate system relative to the bird's-eye image based on the ground point position, an image size of the bird's-eye image and a field of view range corresponding to the bird's-eye image; substituting the third position into the conversion equation to obtain the target position of the license plate in the bird's-eye image.

7. The method of claim 5, wherein, The third position comprises a corner point position of a corner point of the license plate in the bird's-eye coordinate system, and the method further comprises: solving the ground point position corresponding to the line formed by the camera optical center and the line based on the fourth position and the corner point position.

8. The method of claim 1, wherein, The method further comprises: recognizing the license plate and the first position of the license plate in the environment image based on a pre-trained recognition model.

9. A vehicle characterized by comprising: The device comprises a memory and a processor, and the memory stores a computer program which is executed by the processor to implement the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program which is executed by one or more processors to implement the method of any one of claims 1-8.

Images