Blind spot detection method, device, electronic device and storage medium

Abstract

The present disclosure provides a blind spot detection method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology such as intelligent transportation. The specific implementation scheme is: dedistorting the original image collected by the fisheye camera in the roadside perception system to obtain a dedistorted image; based on the original image, inverse mapping the dedistorted image to obtain a mapped image; based on the mapped image, the original image and the imaging image collected by the gun camera, detecting whether the roadside perception system has a perception blind spot; wherein the gun camera is installed on the same monitoring pole as the fisheye camera in the roadside perception system. The technology disclosed in the present disclosure can effectively improve the accuracy and efficiency of blind spot detection.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as intelligent transportation, and in particular to a blind spot detection method, device, electronic device and storage medium. Background Art

[0002] The construction of the roadside infrastructure of the Internet of Vehicles is mainly based on the construction of the roadside perception system. The roadside perception system mainly uses sensors such as cameras and relies on roadside edge computing to form perception data, providing beyond-line-of-sight perception information for unmanned vehicles in vehicle-road collaboration.

[0003] As one of the most important sensors of the roadside perception system, the accurate installation and angle adjustment of the camera is the key to the deployment of the roadside perception system and the key to full coverage of roadside perception without blind spots. Summary of the invention

[0004] The present disclosure provides a blind spot detection method, device, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, a method for detecting a blind spot is provided, comprising:

[0006] Dedistorting the original image captured by the fisheye camera in the roadside perception system to obtain a dedistorted image;

[0007] Based on the original image, performing inverse mapping processing on the dedistorted image to obtain a mapped image;

[0008] Based on the mapping image, the original image and the imaging image collected by the gun camera, it is detected whether the roadside perception system has a perception blind spot; wherein the gun camera is installed on the same monitoring pole as the fisheye camera in the roadside perception system.

[0009] According to another aspect of the present disclosure, a blind spot detection device is provided, comprising:

[0010] A dedistortion processing module is used to dedistort the original image collected by the fisheye camera in the roadside perception system to obtain a dedistorted image;

[0011] An inverse mapping processing module, configured to perform inverse mapping processing on the dedistorted image based on the original image to obtain a mapped image;

[0012] A detection module is used to detect whether the roadside perception system has a perception blind spot based on the mapping image, the original image and the imaging image collected by the gun; wherein the gun is installed on the same monitoring pole as the fisheye camera in the roadside perception system.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and any possible implementation method.

[0019] According to the technology disclosed in the present invention, the accuracy and efficiency of blind spot detection can be effectively improved.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0024] Figure 3 is an original image captured by a fisheye camera provided by an embodiment of the present disclosure;

[0025] Figure 4 Yes Figure 3 A dedistorted image obtained by dedistorting the original image shown;

[0026] Figure 5 is a schematic diagram of a first overlay graph provided by an embodiment of the present disclosure;

[0027] Figure 6is a schematic diagram of a spherical model of a fisheye camera in an embodiment of the present disclosure;

[0028] Figure 7 It is an equivalent bolt diagram from a front bolt perspective provided by an embodiment of the present disclosure;

[0029] Figure 8 It is an equivalent bolt diagram from a rear bolt perspective provided by an embodiment of the present disclosure;

[0030] Fig. 9 is a schematic diagram according to a third embodiment of the present disclosure;

[0031] Fig.10 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0032] Fig.11 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0035] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0036] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0037] For different cameras on a monitoring pole, such as a gun camera and a fisheye camera, the traditional method is to dedistort the fisheye camera image after installation; then combine it with the gun camera image and manually check the road signs or ground signs in the image to see if there is a static blind spot. However, this method requires manual inspection, which is inefficient and not very accurate.

[0038] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1 As shown, this embodiment provides a blind spot detection method, which can be applied in a roadside perception system, and specifically may include the following steps:

[0039] S101, dedistorting an original image captured by a fisheye camera in a roadside perception system to obtain a dedistorted image;

[0040] S102, based on the original image, performing inverse mapping processing on the dedistorted image to obtain a mapped image;

[0041] S103. Based on the mapping image, the original image, and the imaging image captured by the gun camera, detect whether there is a perception blind spot in the roadside perception system; wherein the gun camera is installed on the same monitoring pole as the fisheye camera in the roadside perception system.

[0042] The camera is the main unit in the roadside perception system, which realizes the perception of the environment by collecting images. For example, the roadside perception system can include two types of cameras, such as fisheye cameras and gun cameras. The gun camera refers to a gun-type camera. In actual road scenes, two types of cameras can be installed on the same monitoring pole. In order to achieve full coverage of the detection area, the fisheye camera can be installed with the optical center facing downward and parallel to the monitoring pole. The fisheye camera has a large field of view, which can monitor the road surface in the area where the monitoring pole is located. The gun camera can be set on both sides of the fisheye camera on the monitoring pole to detect the road surface in two directions, forward and backward, and achieve comprehensive monitoring together with the fisheye camera.

[0043] The actual application scenarios are different, and the types of gun cameras are also different. For example, in ordinary urban scenes, gun cameras include short-focus gun cameras. In high-speed scenes, based on the needs of the scene, there is also a long-focus gun camera, which is set outside the short-focus camera to achieve detection and perception of farther areas.

[0044] In actual application scenarios, in addition to all the corresponding cameras on the monitoring poles, the roadside perception system may also include a roadside computing unit set up on each road to perform roadside edge calculation based on the images collected by all the cameras on each monitoring pole on the current road to obtain corresponding perception data. The blind spot detection method of this embodiment can be specifically applied in the roadside computing unit in the roadside perception system to detect the area monitored by all the cameras on each monitoring pole to determine whether there is a blind spot.

[0045] Among them, the roadside computing unit can also share and broadcast the calculated perception data in real time, and can provide autonomous driving vehicles with perception information such as beyond-visual-range perception, blind spot warning, and driving intention, making up for the limitations of single-vehicle perception. On the other hand, it can also report the perception data to the cloud in real time, realize the full collection of traffic environment data, and support the construction of a refined and real-time traffic monitoring system.

[0046] Since the field of view of the fisheye camera is relatively large, the pixels in the collected original image that are closer to the coordinate point corresponding to the optical center are within the effective perception area of ​​the fisheye camera. However, the coordinate points in the original image that are closer to the optical center are severely distorted and cannot be effectively perceived, and can be considered to be outside the effective perception area of ​​the fisheye camera. Therefore, in this embodiment, after the original image is dedistorted, all the pixels in the dedistorted image obtained are within the effective perception area of ​​the fisheye camera. Usually, the coordinate point corresponding to the optical center is near the center point of the image. Based on this, it can be considered that the dedistorted image has removed the edge area of ​​the original image relative to the original image. The specific dedistortion processing can refer to the relevant existing technology, which will not be described in detail here.

[0047] In this embodiment, the dedistorted image is also de-mapped based on the original image, so that the obtained mapped image is returned to the space of the original image. That is to say, each pixel in the mapped image has the same information as the corresponding pixel in the original image. However, it should be noted that since the mapped image is the image de-mapped from the dedistorted image of the original image, the dedistorted image itself has removed the edge area in the original image. Therefore, the mapped image obtained based on the de-mapped image is smaller in size than the original image and lacks the surrounding edge area.

[0048] Finally, multi-angle detection can be performed based on the mapping image, the original image, and the imaging image collected by the gun camera installed on the same monitoring pole as the fisheye camera to determine whether there is a perception blind spot in the roadside perception system.

[0049] The blind spot detection method of this embodiment performs inverse mapping processing on the dedistorted image corresponding to the original image based on the original image to obtain a mapping image; and further detects whether there is a perception blind spot in the roadside perception system based on the mapping image, the original image and the imaging image collected by the gun, and can accurately and effectively detect the blind spot. Moreover, the technical solution of this embodiment can automatically detect the blind spot without manual detection, which can effectively improve the accuracy and efficiency of blind spot detection.

[0050] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; the blind spot detection method provided in this embodiment, in the above Figure 1Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 2 As shown, the blind spot detection method of this embodiment may specifically include the following steps:

[0051] S201, dedistorting an original image collected by a fisheye camera in a roadside perception system to obtain a dedistorted image;

[0052] For example Figure 3 It is an original image captured by a fisheye camera provided by an embodiment of the present disclosure. Figure 4 Yes Figure 3 The dedistorted image is obtained by dedistorting the original image shown in FIG. Figure 3 and Figure 4 By comparison, we can see that the original image is collected in the spherical coordinate system used by the fisheye camera and projected onto the plane. After the original image is dedistorted, not only the edge area with severe distortion is removed, but also the coordinate system of the effective perception area is transformed, so that the coordinate system of the dedistorted image is the coordinate system corresponding to the gun-type camera, such as Figure 4 As shown, the dedistorted image may be a square image.

[0053] In this embodiment, the Ocam model or the OpenCV model can be used to dedistort the original image captured by the fisheye camera.

[0054] S202, based on the original image, performing inverse mapping processing on the dedistorted image to obtain a mapped image;

[0055] The inverse mapping process of this embodiment can be implemented based on the internal parameters of the fisheye camera.

[0056] Through calibration, the internal parameter matrix of the fisheye camera can be obtained, for example, it can include the focal length fx of the x-axis, the focal length fy of the y-axis and the principal point coordinates (cx, cy), distortion parameters k1, k2 and the equivalent focal lengths fx', fy' of the dedistorted image; the principal point coordinates (cx, cy) represent the coordinates of the optical center of the fisheye camera projected on the pixel point in the image. The equivalent gun focal lengths fx', fy' of the dedistorted image can also be called the equivalent gun pixel focal lengths of the dedistorted image. For example Figure 4 The corresponding dedistorted image can be regarded as being shot by an equivalent gun camera. The pixel focal length of the equivalent gun camera of the dedistorted image here can be understood as the pixel focal length of the equivalent gun camera that can directly capture the dedistorted image. For example, fx' can be equal to (the height of the monitoring pole * half of the pixel length of the side length of the dedistorted image) / half of the distance of the side length corresponding to the dedistorted image. fy' can be calculated using the same principle. Considering that the dedistorted image can be a square image, fy'=fx' can be directly taken.

[0057] During the inverse mapping process, xc1 and yc1 are used to represent the normalized coordinates in the camera coordinate system, which can be specifically expressed as:

[0058] xc1=(u'-cx) / fx'

[0059] yc1=(v'-cy) / fy'

[0060] Where (u', v') represents the pixel coordinates of the dedistorted image of the fisheye camera.

[0061] Then, based on xc1 and yc1, the coordinates of the pixel point in the original image are obtained.

[0062] The radius r in the normalized coordinate system of the fisheye camera can be expressed by the following formula:

[0063]

[0064] Distortion of fisheye camera θ d It can be expressed by the following formula:

[0065] θ d =θ*(1+k1*θ 2 +k2*θ 4 )

[0066] Wherein, θ represents the magnitude of the incident angle in the fisheye camera model.

[0067]

[0068]

[0069] Among them, (u, v) represents the 2D coordinates of the corner point in the original image.

[0070] By adopting the above method, in the inverse mapping process, the pixel point corresponding to each pixel point in the dedistorted image in the original image can be obtained, thereby achieving inverse mapping.

[0071] S203, performing a superposition process based on the mapped image and the original image to obtain a first superposition image, so that the superposition effect is presented as a superimposed area being a bright area, and a non-superimposed area being a dark area;

[0072] After the above-mentioned inverse mapping process, the correspondence between each pixel in the mapped image and a pixel in the original image can be obtained. When superimposing, based on the correspondence between each pixel in the mapped image and the corresponding pixel in the original image, the pixels with the corresponding relationship can be superimposed together to obtain a first superimposed image, so that the superimposed effect is presented as a bright area in the superimposed area and a dark area in the non-superimposed area.

[0073] For example, step S203 may include the following steps during specific implementation:

[0074] (1) configuring a first weight for each pixel in the mapped image;

[0075] (2) configuring a second weight for each pixel in the original image, wherein the first weight is greater than the second weight;

[0076] (3) Based on the first weight and the second weight, each pixel point in the mapped image is superimposed on a corresponding pixel point on the original image to obtain a first superimposed image, so that the superimposed effect is presented as a bright area and a dark area.

[0077] Since each pixel in the mapped image has a corresponding pixel in the original image, that is, each pixel in the mapped image can be superimposed in the original image. The mapped image is the dedistorted image of the original image, so the pixels in the edge area of ​​the original image with more serious distortion have been removed in the dedistorted image, so there are no pixels in the edge area of ​​the original image in the mapped image.

[0078] In this embodiment, based on the first weight and the second weight, when the corresponding pixels in the two images are superimposed, the grayscale value of the superimposed pixel is equal to the product of the first weight and the grayscale value of the corresponding pixel in the mapped image, and the product of the second weight and the grayscale value of the corresponding pixel in the mapped image. Since the first weight is greater than the second weight, the grayscale value in the mapped image has a greater influence on the grayscale value of the superimposed pixel. For the pixel in the edge area of ​​the original image, since there is no corresponding pixel in the mapped image, the grayscale value is equal to the second weight multiplied by the grayscale value of the pixel. After the above-mentioned pixel superposition processing, the grayscale value of the superimposed pixel can be increased, while the grayscale value of the non-superimposed pixel becomes smaller. For example, in actual applications, by setting the first weight and the second weight, the grayscale value of each pixel in the superimposed area can be greater than 200, while the grayscale value of each pixel in the non-superimposed area can be less than 100, thereby presenting the superimposed area as a bright area and the non-superimposed area as a dark area. By configuring the weights in this way, it can be clearly shown that the superimposed area in the first superimposed image is a bright area, while the non-superimposed area is a dark area. The result is very accurate and reliable, and the implementation method is very simple.

[0079] For example, Figure 5 is a schematic diagram of a first superposition diagram provided by an embodiment of the present disclosure. After the superposition processing of the above steps (1)-(3), the following can be obtained: Figure 5 The superposition effect diagram shown. Figure 5As shown in the figure, the bright area in the middle is the superimposed area, which is the effective perception area of ​​the fisheye camera, while the dark area at the edge is the area that exists in the original image, but does not exist in the mapped image mapped back from the dedistorted image. The dark area is the area that the fisheye camera cannot effectively perceive.

[0080] Through the above processing, the first overlay image can be divided into bright areas and dark areas, and the effective perception area and the area that cannot be effectively perceived by the language camera can be accurately and effectively identified.

[0081] S204, based on the first superposition image, obtaining an equivalent gun image corresponding to the gun viewing angle;

[0082] Based on the above, it can be known that the first superimposed image is an image from the fisheye camera perspective. In this embodiment, the original image captured by the fisheye camera can be transformed into a bolt image at a certain angle by using a spherical model projection transformation.

[0083] For example, Figure 6 Schematic diagram of a spherical model of a fisheye camera in an embodiment of the present disclosure. Figure 6 As shown, O is the projection center, point D is the geometric center of the gun-type plane projection surface, and the gun-type plane projection surface is tangent to the spherical projection surface of the fisheye camera at point D, θ 1 express Angle with the Z axis, θ 2 Indicates the X-axis direction to The angle that the projection on the XOY plane needs to rotate counterclockwise. The radius of the original image of the fisheye camera and the radius of the spherical projection surface are both r. It can be expressed as the following formula:

[0084]

[0085] Let P be any point on the projection surface of the gun-type plane, and the projection point of P on the XOY plane is Q. α is the distance from the Z axis to The angle between the X axis and The angle of counterclockwise rotation. The coordinates of point P and point D in the gun-type plane coordinate system are (u P , v P ) and (u D , v D ). In order to find the coordinates of point P in the space coordinate system, Decompose into and These two vectors are parallel to the u-axis and the v-axis respectively. According to the geometric relationship, we can conclude that:

[0086]

[0087]

[0088] Then we can get the following relationship:

[0089]

[0090]

[0091] The straight line OP and the spherical projection surface intersect at one point, and the projection of the intersection point on the XOY plane is point M.

[0092] Assume that the unit vectors of the three axes XYZ are The following relationship can be obtained:

[0093]

[0094]

[0095]

[0096]

[0097] The radius r of the circular image captured by the fisheye camera can be detected by opencv contour detection. For example, in this embodiment, the radius r of the circular image can be used = 593 pixels, and the size of the incident angle at this time is θ = PI / 2. From the equidistant projection model, it can be obtained:

[0098] r=focal×θ

[0099] Then, the value of focal can be obtained, which represents the pixel focal length of the fisheye camera obtained based on the intrinsic calibration of the fisheye camera.

[0100] Then, according to the formula r = focal × θ, we can get The corresponding r p :

[0101] r P = focal × α

[0102] Then the pixel coordinates of the fisheye camera image corresponding to point P can be expressed as:

[0103] u P =r P ×sinβ+center x

[0104] v P =r P ×cosβ+center y

[0105] Center x,center y Represents the pixel coordinates of the center point of the fisheye camera image.

[0106] At this point, the correspondence between the pixel coordinates of the gun-type plane projection surface and the pixel coordinates of the image of the fisheye camera can be obtained, so that the image of the fisheye camera can be converted into the image of the gun-type plane projection surface.

[0107] Since the first overlay image in the above embodiment is an image corresponding to the fisheye camera, based on the above principle, it is possible to obtain an equivalent gun camera image corresponding to the gun camera viewing angle based on the first overlay image.

[0108] For example, based on the above implementation principle, θ can be obtained based on the first overlay graph. 1 =70°,θ 2 = 90°, which is the equivalent gun image corresponding to the front gun angle. 1 =70°,θ 2 = 270°, that is, the equivalent bolt image corresponding to the rear bolt angle of view.

[0109] For example, Figure 7 This is an equivalent bolt diagram from the front bolt perspective provided by the embodiment of the present disclosure. Figure 7 As shown, it is based on Figure 5 The equivalent bolt diagram from the front bolt perspective is obtained by the first superposition diagram shown. Figure 8 is an equivalent bolt diagram from a rear bolt perspective provided by an embodiment of the present disclosure. Figure 8 is based on Figure 5 The equivalent bolt diagram of the rear bolt perspective obtained by the first superposition diagram shown. Figure 7 and Figure 8 As shown, the bright area and the dark area in the equivalent gun image of the equivalent gun perspective obtained do not change.

[0110] S205. Based on the imaging image of the gun and the equivalent gun image, detect whether there is a perception blind spot in the roadside perception system.

[0111] For example, in this embodiment, in order to facilitate detection, the imaging image of the gun and the equivalent gun image can be superimposed based on image registration to obtain a second superimposed image, so that the superimposed area is a bright area; then based on the second superimposed image, it is detected whether there is a perception blind spot in the roadside perception system.

[0112] In a specific implementation, a third weight can be configured for each pixel in the imaging image of the gun; a fourth weight can be configured for each pixel in the equivalent gun image, and the third weight is greater than the fourth weight; and in an image registration manner, based on the third weight and the fourth weight, each pixel in the imaging image of the gun is superimposed on the corresponding pixel on the equivalent gun image to obtain a second superimposed image, so that the superimposed area is a bright area, and the non-superimposed area is a dark area.

[0113] Specifically, by setting the size of the third weight and the fourth weight, when the pixels with corresponding relationships in the two images are superimposed, the grayscale value of the superimposed pixel is equal to the sum of the product of the third weight and the grayscale value of the corresponding pixel in the imaging image of the gun, and the product of the fourth weight and the grayscale value of the corresponding pixel in the equivalent gun image. Since the third weight is greater than the fourth weight, the grayscale value in the imaging image of the gun has a greater impact on the grayscale value of the superimposed pixel. In the equivalent gun image, for the pixel that has no corresponding relationship with the imaging image of the gun, the grayscale value is equal to the fourth weight multiplied by the grayscale value of its pixel. After the above-mentioned pixel superposition processing, the grayscale value of the superimposed pixel can be increased, while the grayscale value of the non-superimposed pixel becomes smaller. Similarly, by setting the third weight and the fourth weight, the grayscale value of each pixel in the superimposed area is greater than 200, while the grayscale value of each pixel in the non-superimposed area is less than 100, thereby presenting the superimposed area as a bright area and the non-superimposed area as a dark area.

[0114] It should be noted that the implementation process of the second overlay map is exactly the same as the implementation process of the first overlay map in the above embodiment. For details, reference can also be made to the implementation process of the first overlay map. Similarly, through the weight configuration method, it is possible to clearly present that the superimposed area in the second overlay map is a bright area, while the non-superimposed area is a dark area, and the result is very accurate and reliable, and the implementation method is very simple.

[0115] Before the superposition process, it can be considered that the imaging images of the gun are all bright areas. However, there are bright areas and dark areas in the equivalent gun image. After superposition, it is necessary to adjust the areas that originally belonged to the dark area but belong to the superposition after superposition, that is, the areas where the imaging image of the gun and the equivalent gun image coexist, to bright areas. At this time, the entire bright area in the second superposition image is the effective perception area of ​​the fisheye camera and the effective perception area of ​​the gun. The dark area is the area that the fisheye camera and the gun cannot perceive. The blind area detected in this embodiment is the dark area. However, not all dark areas in the second superposition image are blind areas. For example, when the opposite lane or other areas outside the road surface are dark areas, they do not belong to the perception blind area of ​​the current perception system. The blind area of ​​this embodiment mainly refers to the perception blind area of ​​the road surface by the camera of the roadside perception system.

[0116] In this embodiment, based on the second overlay image, detecting whether the roadside perception system has a perception blind spot may include the following implementation methods:

[0117] The first method uses a pre-trained blind spot detection model to detect the second overlay image to determine whether there is a perception blind spot in the roadside perception system.

[0118] When in use, the second overlay image is directly input into the blind spot detection model, and the blind spot detection model can detect and output whether there is a perception blind spot in the second overlay image.

[0119] The second method may include the following steps:

[0120] (a) Detecting road boundaries in the equivalent gun image;

[0121] The road boundary here includes the boundaries on both sides of the road. The road refers to the road monitored by the camera on the current monitoring pole. Specifically, this step can be performed before the imaging image of the gun camera and the equivalent gun camera image are superimposed to obtain the second superimposed image.

[0122] For example, the detection of road boundaries in this embodiment can use a pre-trained road boundary detection model to detect the road boundaries in the equivalent gun image. Alternatively, certain detection rules can be set based on the boundary characteristics of the road to detect the road boundaries in the equivalent gun image.

[0123] (b) Detect whether there is a dark area on the road surface surrounded by the road boundary in the second overlay image; if so, determine that there is a perception blind area between the fisheye camera and the gun camera; otherwise, determine that there is no perception blind area between the fisheye camera and the gun camera.

[0124] In this embodiment, the second overlay image can also be output to the detection device and displayed. Based on the second overlay image displayed on the detection device, the staff detects whether there is a perception blind spot between the fisheye camera and the gun camera, and returns the detection result. Alternatively, in this embodiment, the imaging image of the gun camera and the equivalent gun camera image can also be directly output to the detection device and displayed. Based on the imaging image of the gun camera and the equivalent gun camera image displayed on the detection device, the staff detects whether there is a perception blind spot between the fisheye camera and the gun camera, and returns the detection result.

[0125] Regardless of which of the above methods is used, it is possible to effectively detect whether there is a perception blind area between the fisheye camera and the gun camera based on the second overlay image, and the detection process is fast and accurate.

[0126] The gun in the above technical solution of this embodiment can be a front gun or a rear gun. In practical applications, for each fisheye camera and any gun on a monitoring pole in the roadside monitoring system, the above technical solution needs to be used to detect whether there is a perception blind spot between the fisheye camera and the front gun, and whether there is a perception blind spot between the fisheye camera and the rear gun, so as to achieve all-round monitoring of the blind spot.

[0127] In practical applications, based on the blind spots monitored in this embodiment, the installation angle of the gun-type camera can be adjusted to remove the blind spots and improve the perception ability of the driverless car, so as to effectively improve the driving safety of the driverless car.

[0128] Steps S203-S205 are as above Figure 1 One implementation of step S103 of the embodiment shown, in actual application, other methods can be used to detect whether there is a perception blind spot in the roadside perception system based on the mapping image, the original image and the imaging image collected by the gun camera, for example, it can be output to the detection device and displayed, and the detection device can be used to directly identify whether there is a detection blind spot between the fisheye camera and the gun camera. Or other methods can be used for detection, which will not be repeated here.

[0129] The blind spot detection method of this embodiment is to superimpose the reverse mapping image obtained by reverse mapping the dedistorted image of the original image onto the original image, so that the superimposed area of ​​the obtained first superimposed image is a bright area, and the non-superimposed area is a dark area, which can effectively identify the effective perception area of ​​the fisheye camera and the area that cannot be effectively perceived. Then, the equivalent gun camera image of the gun camera angle of view of the first superimposed image and the imaging image of the gun camera are superimposed to obtain a second superimposed image, which can accurately and clearly identify the perception areas of the fisheye camera and the gun camera. Finally, based on the second superimposed image, it is quickly, accurately and effectively detected whether there is a perception blind spot between the fisheye camera and the gun camera. Compared with the prior art, no manual participation is required throughout the process, which can not only quickly and accurately detect whether there is a perception blind spot in the roadside perception system, but also effectively improve the accuracy and efficiency of blind spot detection.

[0130] Fig. 9 is a schematic diagram according to the third embodiment of the present disclosure; Fig. 9 As shown, this embodiment provides a blind spot detection device, which can be applied in a roadside perception system, such as a roadside computing unit of a roadside perception system. The blind spot detection device 900 of this embodiment can specifically include:

[0131] The dedistortion processing module 901 is used to dedistort the original image collected by the fisheye camera in the roadside perception system to obtain a dedistorted image;

[0132] The inverse mapping processing module 902 is used to perform inverse mapping processing on the dedistorted image based on the original image to obtain a mapped image;

[0133] The detection module 903 is used to detect whether there is a perception blind spot in the roadside perception system based on the mapping image, the original image and the imaging image collected by the gun camera; wherein the gun camera is installed on the same monitoring pole as the fisheye camera in the roadside perception system.

[0134] The blind spot detection device of this embodiment realizes the implementation principle and technical effect of blind spot detection by adopting the above-mentioned module, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0135] Fig.10 is a schematic diagram according to a fourth embodiment of the present disclosure; Fig.10 As shown, this embodiment provides a blind spot detection device 1000, relative to Fig. 9 The embodiments shown in the figure further introduce the technical solution of the present disclosure in more detail. Fig.10 As shown, the blind spot detection device 1000 of this embodiment includes Fig. 9 The modules with the same name and function are a de-distortion processing module 1001 , an inverse mapping processing module 1002 and a detection module 1003 .

[0136] like Fig.10 As shown, in this embodiment, the detection module 1003 includes:

[0137] The overlay processing unit 10031 is used to perform overlay processing based on the mapped image and the original image to obtain a first overlay image, so that the overlay effect is presented as the overlay area being a bright area and the non-overlay area being a dark area;

[0138] An acquisition unit 10032 is used to acquire an equivalent gun camera image corresponding to the gun camera angle of view based on the first overlay image;

[0139] The blind spot detection unit 10033 is used to detect whether there is a perception blind spot in the roadside perception system based on the imaging image of the gun camera and the equivalent gun camera image.

[0140] Further optionally, in one embodiment of the present disclosure, the overlay processing unit 10031 is configured to:

[0141] Assigning a first weight to each pixel in the mapped image;

[0142] Assign a second weight to each pixel in the original image, wherein the first weight is greater than the second weight;

[0143] Based on the first weight and the second weight, each pixel in the mapped image is superimposed on the corresponding pixel on the original image to obtain a first superimposed image, so that the superimposed area is a bright area and the non-superimposed area is a dark area.

[0144] Further optionally, in one embodiment of the present disclosure, the blind spot detection unit 10033 is configured to:

[0145] Based on the image registration method, the imaging image of the gun and the equivalent gun image are superimposed to obtain a second superimposed image, so that the superimposed area is a bright area, and the non-superimposed area is a dark area;

[0146] Based on the second overlay image, detect whether the roadside perception system has a perception blind spot.

[0147] Further optionally, in one embodiment of the present disclosure, the blind spot detection unit 10033 is configured to:

[0148] Assign a third weight to each pixel in the imaging image of the gun;

[0149] Assigning a fourth weight to each pixel in the equivalent gun image, wherein the third weight is greater than the fourth weight;

[0150] In an image registration manner, based on the third weight and the fourth weight, each pixel in the imaging image of the gun is superimposed on the corresponding pixel on the equivalent gun image to obtain a second superimposed image, so that the superimposed area is a bright area, and the non-superimposed area is a dark area.

[0151] Further optionally, in one embodiment of the present disclosure, the blind spot detection unit 10033 is configured to:

[0152] The second overlay image is detected using a pre-trained blind spot detection model to determine whether the roadside perception system has a perception blind spot.

[0153] Further optionally, in one embodiment of the present disclosure, the detection module 1003 further includes:

[0154] The road boundary detection unit 10034 is used to detect the road boundary in the equivalent gun image.

[0155] Further optionally, in one embodiment of the present disclosure, the road boundary detection unit 10034 is used to:

[0156] A pre-trained road boundary detection model is used to detect road boundaries in equivalent gun camera images.

[0157] Further optionally, in one embodiment of the present disclosure, the blind spot detection unit 10033 is configured to:

[0158] Detecting whether there is a dark area on the road surface surrounded by the road boundary in the second overlay image;

[0159] If so, make sure there is a blind spot between the fisheye camera and the bolt camera.

[0160] The blind spot detection device of this embodiment realizes the implementation principle and technical effect of blind spot detection by adopting the above-mentioned module, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0161] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0162] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0163] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0164] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0165] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0166] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the above-mentioned methods of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the above method of the present disclosure in any other appropriate manner (for example, by means of firmware).

[0167] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0169] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0171] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0172] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0173] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0174] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting a blind spot, comprising: Dedistorting the original image captured by the fisheye camera in the roadside perception system to obtain a dedistorted image; Based on the original image, performing inverse mapping processing on the dedistorted image to obtain a mapped image; Based on the mapping image, the original image and the imaging image collected by the gun, detecting whether the roadside perception system has a perception blind spot; wherein the gun camera and the fisheye camera are installed on the same monitoring pole in the roadside perception system; Wherein, based on the mapping image, the original image and the imaging image collected by the gun, detecting whether the roadside perception system has a perception blind spot includes: Performing a superposition process based on the mapped image and the original image to obtain a first superposition image, so that the superposition effect is presented as a superimposed area being a bright area, and a non-superimposed area being a dark area; Based on the first overlay image, acquiring an equivalent gun image corresponding to the gun viewing angle; Based on the imaging image of the gun and the equivalent gun image, detecting whether the roadside perception system has a perception blind spot; The step of performing superposition processing on the mapped image and the original image to obtain a first superposition image includes: Based on the first weight pre-configured for each pixel in the mapped image and the second weight pre-configured for each pixel in the original image, each pixel in the mapped image is superimposed on the corresponding pixel on the original image to obtain the first superimposed image; the first weight is greater than the second weight.

2. The method according to claim 1, wherein: Based on a first weight pre-configured for each pixel in the mapped image and a second weight pre-configured for each pixel in the original image, before superimposing each pixel in the mapped image onto a pixel having a corresponding relationship on the original image to obtain the first superimposed image, the method further includes: configuring the first weight for each pixel in the mapped image; The second weight is configured for each pixel in the original image.

3. The method according to claim 1 or 2, wherein: Based on the imaging image of the gun and the equivalent gun image, detecting whether the roadside perception system has a perception blind spot includes: Based on the image registration method, the imaging image of the gun and the equivalent gun image are superimposed to obtain a second superimposed image, so that the superimposed area is a bright area, and the non-superimposed area is a dark area; Based on the second overlay image, it is detected whether the roadside perception system has a perception blind spot.

4. The method according to claim 3, wherein: Based on the image registration method, the imaging image of the gun and the equivalent gun image are superimposed to obtain a second superimposed image, so that the superimposed area is a bright area and the non-superimposed area is a dark area, including: Assigning a third weight to each of the pixel points in the imaging image of the gun camera; Assigning a fourth weight to each pixel in the equivalent gun image, wherein the third weight is greater than the fourth weight; In an image registration manner, based on the third weight and the fourth weight, each pixel in the imaging image of the gun is superimposed on a corresponding pixel on the equivalent gun image to obtain the second superimposed image, so that the superimposed area is a bright area and the non-superimposed area is a dark area.

5. The method according to claim 3, wherein: Detecting whether the roadside perception system has a perception blind spot based on the second overlay image includes: The second overlay image is detected using a pre-trained blind spot detection model to determine whether the roadside perception system has a perception blind spot.

6. The method according to claim 3, wherein: Based on the image registration method, the imaging image of the gun and the equivalent gun image are superimposed to obtain a second superimposed image so that the superimposed area is a bright area, and the method further includes: Detecting road boundaries in the equivalent gun image.

7. The method according to claim 6, wherein: Detecting a road boundary in the equivalent gun image includes: A pre-trained road boundary detection model is used to detect the road boundary in the equivalent gun image.

8. The method according to claim 6 or 7, wherein: Detecting whether the roadside perception system has a perception blind spot based on the second overlay image includes: Detecting whether there is a dark area on the road surface surrounded by the road boundary in the second overlay image; If so, it is determined that there is a perception blind area between the fisheye camera and the gun camera.

9. A blind spot detection device, comprising: A dedistortion processing module is used to dedistort the original image collected by the fisheye camera in the roadside perception system to obtain a dedistorted image; An inverse mapping processing module, configured to perform inverse mapping processing on the dedistorted image based on the original image to obtain a mapped image; A detection module, used to detect whether the roadside perception system has a perception blind spot based on the mapping image, the original image and the imaging image collected by the gun; wherein the gun camera and the fisheye camera are installed on the same monitoring pole in the roadside perception system; Wherein, the detection module includes: A superposition processing unit, configured to perform superposition processing based on the mapped image and the original image to obtain a first superposition image, so that the superposition effect is presented as a superimposed area being a bright area and a non-superimposed area being a dark area; an acquisition unit, configured to acquire an equivalent gun image corresponding to the gun viewing angle based on the first overlay image; A blind spot detection unit, configured to detect whether the roadside perception system has a perception blind spot based on the imaging image of the gun and the equivalent gun image; The overlay processing unit is used to overlay each pixel in the mapped image onto corresponding pixel points on the original image based on a first weight pre-configured for each pixel in the mapped image and a second weight pre-configured for each pixel in the original image, so as to obtain the first overlay image; the first weight is greater than the second weight.

10. The device according to claim 9, wherein: The superposition processing unit is further used for: configuring the first weight for each pixel in the mapped image; The second weight is configured for each pixel in the original image.

11. The device according to claim 9 or 10, wherein: The blind spot detection unit is used to: Based on the image registration method, the imaging image of the gun and the equivalent gun image are superimposed to obtain a second superimposed image, so that the superimposed area is a bright area, and the non-superimposed area is a dark area; Based on the second overlay image, it is detected whether the roadside perception system has a perception blind spot.

12. The device according to claim 11, wherein The blind spot detection unit is used to: Assigning a third weight to each of the pixel points in the imaging image of the gun camera; Assigning a fourth weight to each pixel in the equivalent gun image, wherein the third weight is greater than the fourth weight; In an image registration manner, based on the third weight and the fourth weight, each pixel in the imaging image of the gun is superimposed on a corresponding pixel on the equivalent gun image to obtain the second superimposed image, so that the superimposed area is a bright area and the non-superimposed area is a dark area.

13. The device according to claim 11, wherein: The blind spot detection unit is used to: The second overlay image is detected using a pre-trained blind spot detection model to determine whether the roadside perception system has a perception blind spot.

14. The device according to claim 11, wherein: The detection module further includes: A road boundary detection unit is used to detect the road boundary in the equivalent gun image.

15. The device according to claim 14, wherein: The road boundary detection unit is used to: A pre-trained road boundary detection model is used to detect the road boundary in the equivalent gun image.

16. The device according to claim 14 or 15, wherein: The blind spot detection unit is used to: Detecting whether there is a dark area on the road surface surrounded by the road boundary in the second overlay image; If so, it is determined that there is a perception blind area between the fisheye camera and the gun camera.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images