A camera anomaly detection method, apparatus and electronic device

Abstract

The application discloses a camera anomaly detection method, device and electronic equipment. The method comprises the following steps: acquiring camera data and corresponding laser radar data, wherein the camera data comprises a plurality of auxiliary grids for partitioning an image, a plurality of image frames and corresponding 2D target detection boxes, and the laser radar data comprises a plurality of laser point clouds and corresponding 3D target bounding boxes; acquiring a 2D target projection box of the 3D target bounding box of each laser point cloud on a corresponding image frame; performing frame correspondence detection on the 2D target detection box on each image frame and the 2D target projection box of the corresponding frame; acquiring an auxiliary grid corresponding to each 2D target projection box on each image frame, setting a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection box on each image frame, and determining whether the camera is abnormal according to the final state value of the auxiliary grid. The technical scheme of the application can quickly and accurately detect whether the camera is abnormal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, and in particular to a camera anomaly detection method and device and electronic equipment. BACKGROUND

[0002] The intelligent transportation system (ITS) is becoming more and more mature, and the most important part of the system application is to use the roadside camera to detect vehicles and pedestrians on the road, so as to send the detection and early warning of traffic accidents and the basis of path planning to the vehicle end.

[0003] In the prior art, maintenance personnel generally maintain the roadside camera regularly, for example, the maintenance personnel need to regularly check whether the lens of the camera is blocked and whether the lens is blurred, etc., and the maintenance of a single roadside camera on a roadside pole takes 1 person + 1 car per day, the maintenance period is long, and the roadside camera on the roadside pole is not maintained again at least one month after being maintained once, which leads to that the operation and testing personnel cannot know in time when the roadside camera has a problem, and a lot of manpower is consumed for the next maintenance. It can be seen that the prior art lacks an efficient and automated scheme for detecting whether the roadside camera is abnormal. SUMMARY

[0004] Based on the above problems existing in the prior art, the present application provides a camera anomaly detection method, device and electronic equipment to quickly detect whether the camera can effectively identify the image content in a certain lens area.

[0005] The embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a camera anomaly detection method, which comprises:

[0007] Obtaining camera data and laser radar data corresponding to the camera data, the camera data comprising a plurality of auxiliary grids for partitioning an image, a plurality of image frames and a 2D target detection frame corresponding to each image frame, and the laser radar data comprising a plurality of laser point clouds and a 3D target bounding box corresponding to each laser point cloud;

[0008] Projecting the 3D target bounding box of each laser point cloud onto the corresponding image frame to obtain a 2D target projection frame corresponding to the 3D target bounding box;

[0009]

[0009] Detecting the 2D target detection frame on each image frame and the 2D target projection frame of the corresponding frame for frame correspondence to obtain a frame correspondence detection result corresponding to the 2D target projection frame;

[0010] An auxiliary grid corresponding to each 2D target projection frame on each frame of image is obtained, a state value of the auxiliary grid is set according to a frame corresponding detection result of each 2D target projection frame on each frame of image, and whether the camera is abnormal is determined according to a final state value of the auxiliary grid.

[0011] Optionally, the frame corresponding detection of the 2D target detection frame on each frame of image and the 2D target projection frame of the corresponding frame is performed to obtain the frame corresponding detection result of the 2D target projection frame, including:

[0012] The frame corresponding detection of the 2D target detection frame on each frame of image and the 2D target projection frame of the corresponding frame is performed to obtain the frame corresponding detection result of the 2D target projection frame, including:

[0013] The frame corresponding detection result of the 2D target projection frame is obtained according to the overlap calculation result.

[0014] Optionally, the frame corresponding detection result of the 2D target projection frame is obtained according to the overlap calculation result, including:

[0015] If the overlap calculation result is greater than an overlap threshold, it is determined that the frame corresponding detection result of the 2D target projection frame is that there is a 2D target detection frame corresponding to the 2D target projection frame.

[0016] Otherwise, it is determined that the frame corresponding detection result of the 2D target projection frame is that there is no 2D target detection frame corresponding to the 2D target projection frame.

[0017] Optionally, the auxiliary grid corresponding to each 2D target projection frame on each frame of image is obtained, including:

[0018] The center point coordinates of each 2D target projection frame on each frame of image are determined, and the region coordinate information corresponding to each auxiliary grid is obtained;

[0019] The auxiliary grid corresponding to the 2D target projection frame is obtained according to the center point coordinates of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid.

[0020] Optionally, the region coordinate information includes region center point coordinates, and the auxiliary grid corresponding to the 2D target projection frame is obtained according to the center point coordinates of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid, including:

[0021] The relative distance between each 2D target projection frame and each auxiliary grid is obtained according to the center point coordinates of each 2D target projection frame and the region center point coordinates of each auxiliary grid;

[0022] The auxiliary grid corresponding to the 2D target projection frame is obtained according to the relative distance between each 2D target projection frame and each auxiliary grid.

[0023] Optionally, the frame corresponding detection result includes two detection results that a 2D target detection frame corresponding to the 2D target projection frame exists or does not exist, and the setting of the state value of the auxiliary grid according to the frame corresponding detection result of each 2D target projection frame on each image frame includes:

[0024] If the frame corresponding detection result of each 2D target projection frame on each image frame is that a 2D target detection frame corresponding to the 2D target projection frame exists, the state value of the auxiliary grid corresponding to the 2D target projection frame is set to 0;

[0025] If the frame corresponding detection result of each 2D target projection frame on each image frame is that a 2D target detection frame corresponding to the 2D target projection frame does not exist, the state value of the auxiliary grid corresponding to the 2D target projection frame is accumulated by 1.

[0026] Optionally, the determining whether the camera is abnormal according to the final state value of the auxiliary grid includes:

[0027] acquiring whether there is a target auxiliary grid with a final state value greater than a state threshold value in the plurality of auxiliary grids;

[0028] If the target auxiliary grid exists, it is determined that the camera is abnormal;

[0029] Otherwise, it is determined that the camera is not abnormal.

[0030] Optionally, when it is determined that the camera is abnormal, the method further includes:

[0031] generating a camera abnormality alarm, and sending the camera abnormality alarm to a camera operation and maintenance management platform.

[0032] In a second aspect, the embodiments of the present application further provide a camera abnormality detection device, and the device includes:

[0033] a data acquisition unit, configured to acquire camera data and laser radar data corresponding to the camera data, the camera data including a plurality of auxiliary grids for partitioning an image, a plurality of image frames, and a 2D target detection frame corresponding to each image frame, and the laser radar data including a plurality of laser point clouds and a 3D target bounding box corresponding to each laser point cloud;

[0034] a point cloud projection unit, configured to project the 3D target bounding box of each laser point cloud onto a corresponding image frame to acquire a 2D target projection frame corresponding to the 3D target bounding box;

[0035] The first detection unit is configured to perform frame correspondence detection on the 2D target detection frame on each image and the 2D target projection frame of the corresponding frame, and obtain a frame correspondence detection result corresponding to the 2D target projection frame.

[0036] The second detection unit is configured to obtain an auxiliary grid corresponding to each 2D target projection frame on each image, set a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection frame on each image, and determine whether the camera is abnormal according to a final state value of the auxiliary grid.

[0037] In a third aspect, an electronic device is provided, including:

[0038] a processor; and

[0039] a memory arranged to store computer-executable instructions that, when executed, cause the processor to perform the camera abnormality detection method.

[0040] In a fourth aspect, a computer-readable storage medium is provided, which stores one or more programs, and the one or more programs, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the camera abnormality detection method.

[0041] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects: The embodiments of the present application first obtain camera data and laser radar data corresponding to the camera data, the camera data including a plurality of auxiliary grids for dividing image regions, and further including a plurality of image frames and corresponding 2D target detection frames, and the laser radar data including a plurality of laser point clouds and corresponding 3D target bounding boxes; then the 2D target projection frame of the 3D target bounding box in the laser radar data on the image frame is detected in frame correspondence with the 2D target detection frame on the image frame; finally, the state value of the corresponding auxiliary grid can be set according to the frame correspondence detection result, and after all image frames are processed, the final state value of the auxiliary grid can be obtained, so that whether the camera is abnormal can be determined according to the final state value of each auxiliary grid. The embodiments of the present application use laser radar assistance to detect camera abnormalities, and combine image partitioning technology to improve the accuracy of camera abnormality detection. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application, and the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0043] Figure 1 a flowchart of a camera abnormality detection method shown in the embodiments of the present application;

[0044] Figure 2 FIG. 1 is a schematic structural diagram of a camera anomaly detection device according to an embodiment of the present application;

[0045] Figure 3 FIG. 2 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0047] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0048] The execution subject of the camera anomaly detection method provided by the embodiments of the present application can be a roadside device (such as a roadside camera or a roadside computing device), a server or a cloud platform, wherein the execution subject of the camera anomaly detection method of the embodiments of the present application can be software or hardware.

[0049] Reference is made to Figure 1 , Figure 1 For example, taking the roadside device as the execution subject, a camera anomaly detection method provided by an embodiment of the present application is introduced. As shown in Figure 1 The camera anomaly detection method provided by an embodiment of the present application can include the following steps S110 to S140:

[0050] Step S110, camera data and corresponding laser radar data are acquired, the camera data includes a plurality of auxiliary grids for partitioning images, a plurality of image frames and a 2D target detection frame corresponding to each image frame, and the laser radar data includes a plurality of laser point clouds and a 3D target bounding box corresponding to each laser point cloud.

[0051] When the roadside device of the embodiments of the present application performs camera anomaly detection, camera data and corresponding laser radar data are acquired, each auxiliary grid corresponding image area is detected based on the camera data and the laser radar data, it is determined whether there is a continuous image area that cannot be recognized, and the automatic detection of camera anomaly is completed accordingly, where the camera anomaly mainly refers to that the camera cannot effectively recognize the image content of a certain image area for a period of time.

[0052] In actual application, the camera can be temporarily blocked by air floating objects, raindrops and the like, or cause temporary blurring of part of the image area. In order to avoid temporary factors from causing false detection of camera abnormalities, the embodiment of the application obtains multiple frames of images and corresponding multiple frames of laser point clouds in a period of time in the data acquisition stage, and performs target detection on each frame of image and each frame of laser point cloud to obtain target detection results, so as to assist in continuous detection of camera abnormalities by using laser point clouds.

[0053] The camera and the laser radar have completed joint calibration in advance, and the image collected by the camera and the laser point cloud collected by the laser radar have a space-time correspondence relationship. The space-time correspondence relationship refers to the characteristics of time alignment and space alignment of the data collected by the camera and the laser radar. The time alignment refers to that the camera and the laser radar can collect the roadside traffic scene at the same time, and the space alignment refers to that the camera and the laser radar can collect the same roadside traffic scene.

[0054] In addition, it should be noted that the camera data in the embodiment of the application includes multiple auxiliary grids for partitioning the image. The auxiliary grids can be rectangular grids, regular hexagonal grids, etc. In some scenarios, the image area of the camera can be divided into multiple auxiliary grids in advance, for example, the image area of the camera is uniformly divided into N*N (N is a positive integer, N≥2) auxiliary grids, and the auxiliary grids are initialized and saved. The initial state value of the auxiliary grid is 0, for example, indicating that the image content at the auxiliary grid can be effectively recognized. In this way, the roadside device can obtain the auxiliary grid from the saved position of the auxiliary grid in the data acquisition stage. Of course, in other scenarios, the roadside device can also dynamically divide the auxiliary grid of the camera image area in real time. Those skilled in the art can flexibly obtain the auxiliary grid of the camera, and the embodiment of the application is not limited in this regard.

[0055] In step S120, the 3D target bounding box of each frame of laser point cloud is projected onto the corresponding image frame to obtain a 2D target projection box corresponding to the 3D target bounding box.

[0056] According to the joint calibration between the laser radar and the camera and the space-time correspondence relationship between the image and the laser point cloud, the embodiment of the application can project each 3D target bounding box of each frame of laser point cloud onto the corresponding image frame to obtain a 2D target projection box corresponding to each 3D target bounding box.

[0057] In step S130, the 2D target detection box on each frame of image is detected in frame correspondence with the 2D target projection box of the corresponding frame to obtain a frame correspondence detection result corresponding to the 2D target projection box.

[0058] By using bounding box detection, it can be determined whether there is a corresponding 2D target detection box in the image region where the 2D target projection box is located on each frame of the image. If it exists, it means that the image region where the 2D target projection box is located on the frame of the image is normal and the image content of the image region can be effectively identified, that is, the camera lens corresponding to the image region is not obstructed. If it does not exist, it means that the image region where the 2D target projection box is located on the frame of the image is abnormal and the image content of the image region cannot be identified at present, that is, the camera lens corresponding to the image region may be obstructed at this time.

[0059] Step S140: Obtain the auxiliary mesh corresponding to each 2D target projection box on each frame image, set the state value of the auxiliary mesh according to the box-to-box detection result of each 2D target projection box on each frame image, and determine whether the camera has any abnormalities based on the final state value of the auxiliary mesh.

[0060] This embodiment sets the state values ​​of the auxiliary mesh corresponding to each 2D target projection frame according to the frame order of multiple images, until the last frame of the multi-frame images has been processed. This yields the final state values ​​of all auxiliary meshes. The magnitude of the final state value indicates the probability of camera anomaly; a larger final state value indicates a higher probability of camera anomaly, and vice versa. When the final state value equals 0, the camera is functioning normally. Therefore, the presence or absence of camera anomalies can be determined based on the final state values ​​of the auxiliary meshes.

[0061] like Figure 1 As can be seen from the camera anomaly detection method shown, this embodiment uses the camera's image partitioning technology to divide the entire image area into multiple small auxiliary grids for partition detection, thereby improving the detection accuracy. Furthermore, during the detection process, LiDAR data is used as auxiliary data to determine whether there is a detection target in the corresponding grid area of ​​each frame image. In this way, different settings can be made for the state values ​​of the corresponding auxiliary grids, so that the camera can be judged to have an anomaly based on the final state value of the auxiliary grids.

[0062] In some embodiments of this application, a box-to-box mapping is performed between the 2D target detection box on each frame of the image and the 2D target projection box of the corresponding frame to obtain the box-to-box mapping detection result corresponding to the 2D target projection box, including:

[0063] The overlap between the 2D target detection bounding box on each frame and the 2D target projection bounding box of the corresponding frame is calculated to obtain the overlap calculation result; the box-corresponding detection result corresponding to the 2D target projection bounding box is obtained based on the overlap calculation result.

[0064] The overlap degree can be calculated according to a pre-designed calculation method, for example, one or more of an intersection over union calculation method, an overlap area calculation method, and a common pixel point quantity calculation method, and the calculation result obtained according to one or more calculation methods is used to obtain the overlap degree calculation result. For example, when the overlap degree is calculated according to two or more calculation methods, the calculation results obtained by each calculation method are weighted, and the weighted calculation result is used as the overlap degree calculation result.

[0065] The overlap degree calculation result can indicate the overlap between the 2D target projection frame and each 2D target detection frame. The greater the numerical value of the overlap degree calculation result, the higher the degree of coincidence between the two frames. When the numerical value of the overlap degree calculation result is equal to 1, the two frames are completely coincident. Conversely, the smaller the numerical value of the overlap degree calculation result, the lower the degree of coincidence between the two frames. When the numerical value of the overlap degree calculation result is equal to 0, the two frames are completely non-coincident.

[0066] Therefore, in some possible implementation schemes of the embodiment, the frame corresponding detection result corresponding to the 2D target projection frame is obtained according to the overlap degree calculation result, including:

[0067] If the overlap degree calculation result is greater than an overlap degree threshold, it is determined that the frame corresponding detection result corresponding to the 2D target projection frame is that there is a 2D target detection frame corresponding to the 2D target projection frame.

[0068] Otherwise, it is determined that the frame corresponding detection result corresponding to the 2D target projection frame is that there is no 2D target detection frame corresponding to the 2D target projection frame.

[0069] The overlap degree threshold indicates a critical value of the degree of overlap between two corresponding frames. When the overlap degree between the 2D target projection frame and the 2D target detection frame is less than the threshold, it indicates that there is no 2D target detection frame corresponding to the 2D target projection frame on the image frame. The overlap degree threshold can be set according to statistical data in the embodiment.

[0070] In some embodiments of the present application, the auxiliary grid corresponding to each 2D target projection frame on each image frame is obtained, including:

[0071] The center point coordinates of each 2D target projection frame on each image frame are determined, and the region coordinate information corresponding to each auxiliary grid is obtained, the region coordinate information including region center point coordinates.

[0072] The auxiliary grid corresponding to each 2D target projection frame is obtained according to the center point coordinates of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid.

[0073] In some possible implementation scenarios of the embodiment, the auxiliary grid corresponding to each 2D target projection frame is obtained according to the center point coordinates of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid, including:

[0074] The relative distance between each 2D target projection frame and each auxiliary grid is obtained according to the center point coordinates of each 2D target projection frame and the region center point coordinates of each auxiliary grid.

[0075] The auxiliary grid corresponding to each 2D target projection frame is obtained according to the relative distance between each 2D target projection frame and each auxiliary grid.

[0076] The auxiliary grids can be sorted according to the order of the relative distances from small to large, the auxiliary grid with the smallest relative distance can be taken as the auxiliary grid corresponding to the 2D target projection frame, or the difference of the relative distances corresponding to the first M (M is a positive integer, M>2) auxiliary grids can be judged, if the difference is less than a difference threshold, the first M auxiliary grids are determined as the corresponding auxiliary grids. For example, the sorting obtained is the first auxiliary grid (the relative distance is 1 cm), the second auxiliary grid (the relative distance is 1.6 cm), the third auxiliary grid (the relative distance is 1.8 cm), the fourth auxiliary grid (the relative distance is 6 cm)…, wherein the relative distances of the first auxiliary grid, the second auxiliary grid and the third auxiliary grid to the 2D target projection frame are relatively close, and the first auxiliary grid, the second auxiliary grid and the third auxiliary grid can be taken as the auxiliary grids corresponding to the 2D target projection frame.

[0077] In some embodiments of the application, the state value of the auxiliary grid is set according to the frame corresponding detection result corresponding to each 2D target projection frame on each frame of image, including:

[0078] If the frame corresponding detection result corresponding to each 2D target projection frame on each frame of image is that there is a 2D target detection frame corresponding to the 2D target projection frame, the state value of the auxiliary grid corresponding to the 2D target projection frame is set to 0;

[0079] If the frame corresponding detection result corresponding to each 2D target projection frame on each frame of image is that there is no 2D target detection frame corresponding to the 2D target projection frame, the state value of the auxiliary grid corresponding to the 2D target projection frame is accumulated by 1.

[0080] In the specific implementation process, the setting of the auxiliary grid of multiple frames of image can be performed through a loop algorithm, and for example, the steps of the loop algorithm are as follows:

[0081] Step 1, obtaining the target auxiliary grid of the current frame of image;

[0082] The target auxiliary grid refers to the auxiliary grid corresponding to each 2D target projection frame on the current frame of image.

[0083] Step 2, obtaining a frame corresponding detection result corresponding to each target auxiliary grid;

[0084] As described above, the frame corresponding detection result includes: a 2D target detection frame corresponding to the 2D target projection frame exists and a 2D target detection frame corresponding to the 2D target projection frame does not exist.

[0085] Step 3, setting a state value of each target auxiliary grid;

[0086] Specifically, if the frame corresponding detection result is that the 2D target detection frame corresponding to the 2D target projection frame exists, the state value of the auxiliary grid corresponding to the 2D target projection frame is set to 0; otherwise, the state value of the auxiliary grid corresponding to the 2D target projection frame is accumulated by 1.

[0087] Step 4, determining whether the current frame image is the last frame image, if the current frame image is the last frame image, the state value of each auxiliary grid in the current frame image is the final state value, if the current frame image is not the last frame image, the next frame image is obtained and the step 1 is returned.

[0088] In the embodiment, whether the current frame image is the last frame image can be determined based on the number of image frames, for example, if the current frame image is the 500th frame image, it is determined that the current frame image is the last frame image. Alternatively, whether the current frame image is the last frame image can be determined according to a preset time length, for example, if the current time reaches the preset time, the current frame image is regarded as the last frame image. Of course, in actual application, a person skilled in the art can flexibly select an exit strategy of the loop body.

[0089] After obtaining the final state value of each auxiliary grid according to the above embodiment, some embodiments of the present application determine whether the camera is abnormal by the following steps:

[0090] Obtaining whether there is a target auxiliary grid with a final state value greater than a state threshold value in the plurality of auxiliary grids;

[0091] If the target auxiliary grid exists, it is determined that the camera is abnormal;

[0092] Otherwise, it is determined that the camera is not abnormal.

[0093] For example, when the final state value of some auxiliary grids is greater than 100, it indicates that the image content corresponding to these auxiliary grids cannot be recognized continuously, and it can be determined that the camera is abnormal.

[0094] In some embodiments of the present application, when it is determined that the camera is abnormal, the method further includes:

[0095] A camera anomaly alarm is generated and sent to a camera operation and maintenance platform to remind operation and test personnel to check and handle.

[0096] In addition, in some embodiments of the present application, after determining whether the camera is abnormal according to the final state value of the auxiliary grid, the state values of all auxiliary grids are also initialized.

[0097] The present application also provides a camera anomaly detection device 200, as shown in Figure 2 The structure diagram of the camera anomaly detection device in the embodiments of the present application is provided, and the camera anomaly detection device 200 comprises a data acquisition unit 210, a point cloud projection unit 220, a first detection unit 230 and a second detection unit 240, wherein:

[0098] The data acquisition unit 210 is configured to acquire camera data and laser radar data corresponding to the camera data, wherein the camera data comprises a plurality of auxiliary grids for partitioning images, a plurality of image frames and a 2D target detection frame corresponding to each image frame, and the laser radar data comprises a plurality of laser point clouds and a 3D target bounding box corresponding to each laser point cloud;

[0099] The point cloud projection unit 220 is configured to project the 3D target bounding box of each laser point cloud onto the corresponding image frame to obtain a 2D target projection frame corresponding to the 3D target bounding box;

[0100] The first detection unit 230 is configured to perform frame correspondence detection on the 2D target detection frame on each image frame and the 2D target projection frame of the corresponding frame to obtain a frame correspondence detection result corresponding to the 2D target projection frame.

[0101] The second detection unit 240 is configured to acquire an auxiliary grid corresponding to each 2D target projection frame on each image frame, set a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection frame on each image frame, and determine whether the camera is abnormal according to a final state value of the auxiliary grid.

[0102] In some embodiments of the present application, the first detection unit 230 is specifically configured to perform overlap calculation on the 2D target detection frame on each image frame and the 2D target projection frame of the corresponding frame to obtain an overlap calculation result, and obtain a frame correspondence detection result corresponding to the 2D target projection frame according to the overlap calculation result.

[0103] In some embodiments of the present application, the first detection unit 230 is specifically configured to: if the overlap degree calculation result is greater than an overlap degree threshold, determine that a frame corresponding detection result of the 2D target projection frame is that there is a 2D target detection frame corresponding to the 2D target projection frame; or otherwise, determine that the frame corresponding detection result of the 2D target projection frame is that there is no 2D target detection frame corresponding to the 2D target projection frame.

[0104] In some embodiments of the present application, the second detection unit 240 comprises an auxiliary grid acquisition module.

[0105] The auxiliary grid acquisition module is configured to: determine a center point coordinate of each 2D target projection frame on each frame of image, and acquire region coordinate information corresponding to each auxiliary grid; and acquire an auxiliary grid corresponding to each 2D target projection frame according to the center point coordinate of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid.

[0106] In some embodiments of the present application, the region coordinate information comprises a region center point coordinate, and the auxiliary grid acquisition module is specifically configured to: acquire a relative distance between each 2D target projection frame and each auxiliary grid according to the center point coordinate of each 2D target projection frame and the region center point coordinate of each auxiliary grid; and acquire an auxiliary grid corresponding to each 2D target projection frame according to the relative distance between each 2D target projection frame and each auxiliary grid.

[0107] In some embodiments of the present application, the frame corresponding detection result comprises two detection results of that there is a 2D target detection frame corresponding to the 2D target projection frame and that there is no 2D target detection frame corresponding to the 2D target projection frame, and the second detection unit 240 further comprises an auxiliary grid setting module.

[0108] The auxiliary grid setting module is configured to: if the frame corresponding detection result of each 2D target projection frame on each frame of image is that there is a 2D target detection frame corresponding to the 2D target projection frame, set a state value of an auxiliary grid corresponding to the 2D target projection frame to 0; or if the frame corresponding detection result of each 2D target projection frame on each frame of image is that there is no 2D target detection frame corresponding to the 2D target projection frame, accumulate the state value of the auxiliary grid corresponding to the 2D target projection frame by 1.

[0109] In some embodiments of the present application, the second detection unit 240 further comprises a judgment module.

[0110] The judgment module is configured to: acquire whether there is a target auxiliary grid with a final state value greater than a state threshold in the plurality of auxiliary grids; if there is the target auxiliary grid, determine that the camera is abnormal; or otherwise, determine that the camera is normal.

[0111] In some embodiments of the present application, the camera anomaly detection apparatus 200 further comprises an alarm unit;

[0112] The alarm unit is configured to generate a camera anomaly alarm when it is determined that the camera has an anomaly, and send the camera anomaly alarm to a camera operation and maintenance management platform.

[0113] It can be understood that the camera anomaly detection apparatus described above can realize each step of the camera anomaly detection method provided in the foregoing embodiments, and the related explanations about the camera anomaly detection method are all applicable to the camera anomaly detection apparatus, which will not be described here.

[0114] Figure 3 is a structural schematic diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 3 At the hardware level, the electronic device comprises a processor and a memory, and optionally further comprises an internal bus and a network interface. The memory can include a memory such as a high-speed random access memory (RAM), and can also include a non-volatile memory such as at least one disk memory. Of course, the electronic device can also include other hardware required by the business.

[0115] The processor, network interface and memory can be connected to each other through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus and a control bus. For ease of representation, Figure 3 In the figure, only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0116] The memory is used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0117] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, and forms a camera anomaly detection apparatus at the logical level. The processor executes the program stored in the memory, and is specifically configured to perform the following operations:

[0118] acquire camera data and laser radar data corresponding to the camera data, the camera data including a plurality of auxiliary grids for partitioning an image, a plurality of image frames, and a 2D target detection box corresponding to each image frame, and the laser radar data including a plurality of laser point clouds and a 3D target bounding box corresponding to each laser point cloud;

[0119] project the 3D target bounding box of each laser point cloud onto the corresponding image frame to obtain a 2D target projection box corresponding to the 3D target bounding box;

[0120] perform frame correspondence detection on the 2D target detection box in each image frame and the 2D target projection box of the corresponding frame to obtain a frame correspondence detection result corresponding to the 2D target projection box;

[0121] acquire an auxiliary grid corresponding to each 2D target projection box in each image frame, set a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection box in each image frame, and determine whether the camera is abnormal according to a final state value of the auxiliary grid.

[0122] The above as described in the present application Figure 1The method performed by the camera anomaly detection apparatus disclosed in the embodiments shown can be applied in a processor or implemented by the processor. The processor can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits in the hardware of the processor or instructions in the form of software. The processor mentioned above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the storage, and the processor reads the information in the storage and combines the hardware to complete the steps of the above camera anomaly detection method.

[0123] The electronic device can also perform the method performed by the camera anomaly detection apparatus in the related art Figure 1 , and realize the functions of the camera anomaly detection apparatus in the related art Figure 1 . The embodiments of the present application will not be repeated here.

[0124] The embodiments of the present application also propose a computer readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an electronic device including a plurality of applications, can cause the electronic device to perform the method performed by the camera anomaly detection apparatus in the embodiments shown Figure 1 , and specifically perform the following operations:

[0125] Obtaining camera data and laser radar data corresponding to the camera data, the camera data including a plurality of auxiliary grids for partitioning an image, a plurality of image frames and a 2D target detection frame corresponding to each image frame, and the laser radar data including a plurality of laser point clouds and a 3D target bounding box corresponding to each laser point cloud;

[0126] projecting the 3D target bounding box of each frame of laser point cloud onto a corresponding image frame to obtain a 2D target projection box corresponding to the 3D target bounding box;

[0127] detecting the 2D target detection box on each frame of image and the 2D target projection box of the corresponding frame to obtain a frame correspondence detection result corresponding to the 2D target projection box;

[0128] obtaining an auxiliary grid corresponding to each 2D target projection box on each frame of image, setting a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection box on each frame of image, and determining whether the camera is abnormal according to a final state value of the auxiliary grid.

[0129] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0130] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce an apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0131] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0132] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0133] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0134] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), flash memory, or a combination of non-volatile memories in different types. The memory is an example of computer readable storage media.

[0135] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article or apparatus that includes the recited element.

[0137] ​​Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0138] The foregoing is merely illustrative of the principles of the application and various modifications can be made by persons skilled in the art. The present application is not intended to be limited to the embodiments shown, but is to be accorded the full scope that resides in the art thereof. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A camera abnormality detection method characterized by comprising: The method comprises: acquiring camera data and laser radar data corresponding to the camera data, the camera data comprising a plurality of auxiliary grids for partitioning an image, a plurality of image frames, and a 2D target detection frame corresponding to each image frame, the laser radar data comprising a plurality of laser point cloud frames and a 3D target bounding box corresponding to each laser point cloud frame; projecting the 3D target bounding box of each laser point cloud frame onto a corresponding image frame to obtain a 2D target projection frame corresponding to the 3D target bounding box; performing frame correspondence detection on the 2D target detection frame of each image frame and the 2D target projection frame of the corresponding frame to obtain a frame correspondence detection result corresponding to the 2D target projection frame; acquiring an auxiliary grid corresponding to each 2D target projection frame on each image frame, setting a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection frame on each image frame, and determining whether the camera is abnormal according to a final state value of the auxiliary grid.

2. The camera abnormality detection method according to claim 1, wherein The frame correspondence detection on the 2D target detection frame of each image frame and the 2D target projection frame of the corresponding frame to obtain a frame correspondence detection result corresponding to the 2D target projection frame comprises: performing overlap calculation on the 2D target detection frame of each image frame and the 2D target projection frame of the corresponding frame to obtain an overlap calculation result; obtaining a frame correspondence detection result corresponding to the 2D target projection frame according to the overlap calculation result.

3. The camera abnormality detection method according to claim 2, wherein The obtaining of a frame correspondence detection result corresponding to the 2D target projection frame according to the overlap calculation result comprises: if the overlap calculation result is greater than an overlap threshold, determining that the frame correspondence detection result corresponding to the 2D target projection frame is that there is a 2D target detection frame corresponding to the 2D target projection frame; otherwise, determining that the frame correspondence detection result corresponding to the 2D target projection frame is that there is no 2D target detection frame corresponding to the 2D target projection frame.

4. The camera abnormality detection method according to claim 1, wherein The acquiring of an auxiliary grid corresponding to each 2D target projection frame on each image frame comprises: determining a center point coordinate of each 2D target projection frame on each image frame, and acquiring region coordinate information corresponding to each auxiliary grid; acquiring an auxiliary grid corresponding to the 2D target projection frame according to the center point coordinate of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid.

5. The camera abnormality detection method according to claim 4, wherein The region coordinate information comprises a region center point coordinate, and the acquiring of an auxiliary grid corresponding to the 2D target projection frame according to the center point coordinate of each 2D target projection frame and the region coordinate information corresponding to each auxiliary grid comprises: acquiring a relative distance between each 2D target projection frame and each auxiliary grid according to the center point coordinate of each 2D target projection frame and the region center point coordinate of each auxiliary grid; acquiring an auxiliary grid corresponding to the 2D target projection frame according to the relative distance between each 2D target projection frame and each auxiliary grid.

6. The camera abnormality detection method according to claim 1, wherein The frame correspondence detection result comprises two detection results of existing a 2D target detection frame corresponding to the 2D target projection frame and not existing a 2D target detection frame corresponding to the 2D target projection frame, and the setting of a state value of the auxiliary grid according to the frame correspondence detection result corresponding to each 2D target projection frame on each image frame comprises: if the frame-by-frame box correspondence detection result of each 2D target projection box on each frame of image is that there is a 2D target detection box corresponding to the 2D target projection box, the state value of the auxiliary grid corresponding to the 2D target projection box is set to 0; if the frame-by-frame box correspondence detection result of each 2D target projection box on each frame of image is that there is no 2D target detection box corresponding to the 2D target projection box, the state value of the auxiliary grid corresponding to the 2D target projection box is accumulated by 1.

7. The camera abnormality detection method according to claim 6, wherein The determining whether the camera is abnormal according to the final state value of the auxiliary grid comprises: acquiring whether there is a target auxiliary grid with a final state value greater than a state threshold value in the plurality of auxiliary grids; if the target auxiliary grid exists, it is determined that the camera is abnormal; otherwise, it is determined that the camera is not abnormal.

8. The camera abnormality detection method according to claim 7, wherein When it is determined that the camera is abnormal, the method further comprises: generating a camera abnormality alarm and sending the camera abnormality alarm to a camera operation and maintenance management platform.

9. A camera abnormality detection apparatus characterized by comprising: The apparatus comprises: a data acquisition unit configured to acquire camera data and lidar data corresponding to the camera data, the camera data comprising a plurality of auxiliary grids for partitioning an image, a plurality of frames of image, and a 2D target detection box corresponding to each frame of image, and the lidar data comprising a plurality of frames of lidar point cloud and a 3D target bounding box corresponding to each frame of lidar point cloud; a point cloud projection unit configured to project the 3D target bounding box of each frame of lidar point cloud onto a corresponding frame of image to obtain a 2D target projection box corresponding to the 3D target bounding box; a first detection unit configured to perform frame-by-frame box correspondence detection on the 2D target detection box on each frame of image and the 2D target projection box of the corresponding frame to obtain a frame-by-frame box correspondence detection result of the 2D target projection box; a second detection unit configured to acquire an auxiliary grid corresponding to each 2D target projection box on each frame of image, set a state value of the auxiliary grid according to the frame-by-frame box correspondence detection result of each 2D target projection box on each frame of image, and determine whether the camera is abnormal according to a final state value of the auxiliary grid. 10.An electronic device comprising: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the camera abnormality detection method of any one of claims 1-8.

Images