Street lamp under-lighting detection method and system based on dynamic patrol data

Abstract

The present invention discloses a street lamp under-lighting detection method and system based on dynamic patrol data. The method comprises: obtaining a street lamp video collected by a device installed on a moving vehicle within a period of time; adjusting the illumination of each frame of the street lamp video to obtain an adjustment result; inputting all the frame images in the adjustment result into a street lamp under-lighting detection model to identify the street lamp position and under-lighting state to obtain the recognition results of all the frame images; assigning an ID to the detection frame of each frame image and performing ID matching, calculating the IoU value between the detection frames with the same ID, and using the recognition result of the next frame image to update or confirm the current recognition result of the detection frame; outputting each detection frame and the corresponding final recognition result. By implementing the method of the present invention, automatic detection of the lighting state of the street lamp can be realized, which effectively improves the patrol efficiency, overcomes the shortcomings of the traditional patrol method, and solves the problem of low accuracy caused by single image detection.

Description

Technical Field

[0001] The present invention relates to a street lamp under-lighting detection method, and more specifically to a street lamp under-lighting detection method and system based on dynamic patrol data. Background Art

[0002] With the continuous advancement of urban construction, the importance of road lighting systems has become increasingly prominent. However, street lamps often fail in daily use due to various reasons, such as lamp aging, line short circuit, bad weather, etc., resulting in insufficient lighting or street lamps going out. In order to ensure the normal lighting of urban roads, street lamps need to be inspected manually regularly to detect faults and repair them. However, traditional inspection methods have problems such as poor real-time performance, low efficiency, and high labor costs.

[0003] At present, some patented technologies have attempted to achieve intelligent management by installing light-sensing sensors on street lamps to monitor the lighting status of street lamps in real time. Although this method improves efficiency, it is difficult to promote and apply it widely due to the high cost of sensor equipment and the fact that intelligent street lamp management is not popular in many areas. Another method is to use a single image for automatic recognition, but this method relies solely on a single image, which will lead to low accuracy in street lamp under-brightness detection, i.e., judgment of street lamp brightness deficiency, due to problems such as perspective angle, distance, and occlusion.

[0004] Therefore, it is necessary to design a new method to realize automatic detection of street lamp lighting status, effectively improve the inspection efficiency, overcome the shortcomings of traditional inspection methods, and solve the problem of low accuracy caused by single image detection. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a street lamp under-lighting detection method and system based on dynamic patrol data.

[0006] To achieve the above object, the present invention adopts the following technical solution: a street lamp under-lighting detection method based on dynamic patrol data, comprising:

[0007] Obtain street light videos collected by a device installed on a moving vehicle over a period of time;

[0008] Performing illumination adjustment on each frame of the street light video to obtain an adjustment result;

[0009] Input all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the detection frame of the street lamp and the corresponding under-lighting state;

[0010] Assign an ID to the detection frame of each frame and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame to update or confirm the current recognition result of the detection frame to obtain the final recognition result corresponding to each detection frame;

[0011] Output each of the detection boxes and the corresponding final recognition results.

[0012] A further technical solution is: performing illumination adjustment on each frame of the street light video to obtain an adjustment result, including:

[0013] Each frame of the street light video is input into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

[0014] A further technical solution is: inputting each frame of the street light video into the illumination adjustment model to perform illumination adjustment to obtain an adjustment result, including:

[0015] Input each frame of the street light video into the illumination adjustment model, and use the local branch network to perform point-by-point linear transformation on each frame of the street light video to obtain a transformation result;

[0016] The global branch network fully connects each frame of the street light video to generate global information, and uses a cross-attention mechanism to combine the global information with the transformation result to generate a color correction matrix and a gamma correction value. The color correction matrix and the gamma correction value are used to adjust each frame of the street light video to obtain an adjustment result.

[0017] A further technical solution is: inputting all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, including:

[0018] The adjustment result is input into the street lamp under-lighting detection model, the adjustment result is divided into windows of fixed size through a convolution layer, the divided pictures are spliced ​​in the depth direction, and the spliced ​​results are linearly transformed through a fully connected layer to obtain a feature map;

[0019] Performing feature cyclic shift on the feature map to obtain a temporary window;

[0020] Applying masked multi-head self-attention calculation to the temporary window to obtain an attention score, and updating the feature map using the attention score;

[0021] Performing a reverse cyclic shift operation on the feature map to restore the feature to an original position in the adjustment result to obtain a processed feature map;

[0022] A decoupling head is used to predict the street light lack-lighting category and position for the processed feature map, and overlapping bounding boxes are removed by non-maximum suppression to obtain recognition results for all frame images.

[0023] A further technical solution is: performing a characteristic cyclic shift on the characteristic map to obtain a temporary window includes:

[0024] Each window corresponding to the feature map is translated to the left by half of its own size, and then translated upward by half of its own size to obtain a temporary window.

[0025] A further technical solution is: assigning an ID to the detection frame of each frame and performing ID matching, calculating the IoU value between the detection frames with the same ID, and using the recognition result of the next frame to update or confirm the current recognition result of the detection frame to obtain the final recognition result corresponding to each detection frame, including:

[0026] Assigning an ID to the detection frame of each frame of the image;

[0027] Performing ID matching on the detection frames of all the frames to determine the detection frames with the same ID in each frame;

[0028] Select the first frame as the current frame;

[0029] The IoU value between each detection frame in the current frame and the detection frame with the same ID in a frame after the current frame;

[0030] Update or confirm the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain an intermediate recognition result corresponding to each detection frame;

[0031] Determine whether the next frame of the current frame is the last frame of the current frame;

[0032] If the frame image after the current frame image is not the last frame image, the frame image after the current frame image is used as the current frame image, and the IoU value of each detection frame in the current frame image and the detection frame with the same ID in the frame image after the current frame image is performed;

[0033] If the frame image following the current frame image is the last frame image, the intermediate recognition result corresponding to each of the detection frames is used as the final recognition result corresponding to each of the detection frames.

[0034] A further technical solution is: updating or confirming the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain an intermediate recognition result corresponding to each detection frame includes:

[0035] When the IoU value is greater than the set threshold, the recognition result corresponding to the detection frame of the frame following the current frame is used as the intermediate recognition result corresponding to each detection frame; when the IoU value is not greater than the set threshold, the detection frame and the corresponding recognition result corresponding to the IoU value are discarded.

[0036] The present invention also provides a street lamp under-lighting detection system based on dynamic patrol data, comprising:

[0037] A video acquisition unit, used to acquire street lamp videos collected by a device installed on a moving vehicle over a period of time;

[0038] An adjustment unit, used for adjusting the illumination of each frame of the street light video to obtain an adjustment result;

[0039] A recognition unit, used for inputting all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the detection frame of the street lamp and the corresponding under-lighting state;

[0040] An updating unit, configured to assign an ID to the detection frame of each frame and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame to update or confirm the current recognition result of the detection frame to obtain a final recognition result corresponding to each detection frame;

[0041] The output unit is used to output each of the detection frames and the corresponding final recognition results.

[0042] Its further technical solution is: the adjustment unit is used to input each frame of the street light video into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

[0043] Its further technical solution is: the adjustment unit includes:

[0044] A transformation subunit, used for inputting each frame of the street light video into the illumination adjustment model, and performing point-by-point linear transformation on each frame of the street light video by a local branch network to obtain a transformation result;

[0045] The picture adjustment subunit is used to generate global information by fully connecting each frame of the street light video through the global branch network, and combine the global information with the transformation result by using the cross attention mechanism to generate a color correction matrix and a gamma correction value, and use the color correction matrix and the gamma correction value to adjust each frame of the street light video to obtain an adjustment result.

[0046] Compared with the prior art, the present invention has the following beneficial effects: by collecting videos of street lamps during vehicle driving and adjusting the illumination of video frames, a clearer street lamp image can be obtained. Then, each frame of the image is input into the street lamp under-lighting detection model to identify the position and under-lighting state of the street lamp, and determine the detection frame of each street lamp and its under-lighting state. In order to improve the accuracy, an ID is assigned to each detection frame and ID matching is performed, and the IoU value between different frames is calculated to ensure the consistency of the target in the video sequence. By updating or confirming the recognition result of each detection frame, the accurate position and lighting state of each street lamp are finally obtained. This method can effectively and automatically detect the lighting state of street lamps, significantly improve the inspection efficiency, solve the shortcomings of traditional manual inspection and single-frame image detection, and improve the accuracy of detection.

[0047] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of a flow chart of a street lamp under-lighting detection method based on dynamic patrol data provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a picture before enhancement provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of an enhanced picture provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a characteristic graph provided by an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a recognition result of a picture at a first moment provided by an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of a recognition result of a picture at a second moment provided by an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of a recognition result of a picture at a third moment provided by an embodiment of the present invention;

[0056] Figure 8 A schematic diagram of a recognition result of a picture at a fourth moment provided by an embodiment of the present invention;

[0057] Fig. 9 A schematic diagram of a recognition result of a picture at a fifth moment provided by an embodiment of the present invention;

[0058] Fig.10 A schematic diagram of a recognition result of a picture at a sixth moment provided by an embodiment of the present invention;

[0059] Fig.11 A schematic block diagram of a street lamp under-lighting detection system based on dynamic patrol data provided by an embodiment of the present invention;

[0060] Fig.12 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0063] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0064] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0065] See also Figure 1 , Figure 1 A schematic flow chart of a street lamp under-lighting detection method based on dynamic patrol data provided by an embodiment of the present invention. The street lamp under-lighting detection method based on dynamic patrol data is applied in a server, which interacts with a device and a terminal installed on a moving vehicle. By performing frame processing and lighting adjustment on the street lamp video in the dynamic patrol data, the quality of each frame of the image is improved, thereby effectively identifying the street lamp position and lighting status. First, the image details are enhanced by lighting preprocessing and adjustment models, and the detection accuracy in low-light or high-light environments is improved. Then, the street lamp position and under-lighting status are identified for each frame of the image using a deep learning model, and the consistency and accuracy of detection across frames are ensured by ID matching and IoU calculation between images. Finally, redundant information is removed by non-maximum suppression and adaptive update algorithms, and the overall detection accuracy is improved. This method overcomes the problems of low efficiency of traditional manual inspections and difficulty in identifying local lighting status, and realizes efficient and accurate automatic detection of street lamp lighting status.

[0066] Figure 1 FIG. 1 is a flow chart of a method for detecting street lamp under-lighting based on dynamic patrol data provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S150.

[0067] S110: Obtain street lamp videos collected by a device installed on a moving vehicle over a period of time.

[0068] In this embodiment, the device is installed on a moving vehicle to collect street light information in real time, including taking street light pictures, videos and GPS data; through the device collection during the vehicle's driving process, street light information in a larger area can be covered, which is particularly suitable for cities or regions with long patrol routes and wide distribution. Compared with manual patrols, vehicle collection is not only faster, but also can continuously obtain data, improving patrol efficiency. The device can collect video and image data in real time, and ensure accurate spatiotemporal labeling of the data through GPS positioning. In this way, the specific location and status of each street light can be accurately recorded, avoiding the problems of missed inspections and inaccurate data in manual patrols.

[0069] By collecting street light videos during vehicle driving, we can not only obtain the status of street lights at a single moment, but also capture the working status of street lights under different environmental lighting and weather changes. For example, the lighting conditions of street lights in different conditions such as morning and evening, sunny and cloudy, rainy and snowy, etc. This provides more dimensions for subsequent analysis and ensures the accuracy of the detection results.

[0070] Through continuous video data, the lighting status of each street lamp can be detected in real time, such as whether there is a lack of light, flickering or over-brightness. Compared with a single static image, this dynamic monitoring can effectively reduce the probability of misjudgment and improve the accuracy of fault detection.

[0071] S120: Performing illumination adjustment on each frame of the street light video to obtain an adjustment result.

[0072] In this embodiment, the adjustment result refers to an image obtained after each frame of the image in the street lamp video has been preprocessed and the illumination has been adjusted.

[0073] Specifically, each frame of the street light video is input into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

[0074] In this embodiment, images collected at night usually have problems of low light, overexposure, and underexposure. The collected image data is preprocessed and adaptive lighting adjustment is performed. Specifically, the lighting adjustment model formed by training the Transformer-based image lighting adjustment network is used to adjust the lighting of the image, aiming to solve the problem of poor performance of images under different lighting conditions, especially for image collection in dynamic environments, such as street light videos taken during vehicle driving. The network is mainly divided into two core modules: PEM (Partial Enhancement Module) and GPM (Global Parameter Module). These two modules work together to enhance the local features of the image and optimize the global lighting conditions.

[0075] In one embodiment, the above-mentioned step S120 may include steps S121-S122.

[0076] S121. Input each frame of the street light video into the illumination adjustment model, and use the local branch network to perform point-by-point linear transformation on each frame of the street light video to obtain a transformation result.

[0077] In this embodiment, the transformation result refers to the result obtained by linearly transforming each frame of the image point by point.

[0078] Specifically, PEM focuses on the local features of the image and performs a point-by-point linear transformation on the feature map of the input image through a series of 1*1 convolutional layers. This transformation can enhance the characteristics of specific pixels, thereby better preserving and highlighting important details in the image.

[0079] After the input image is preprocessed, the feature map is sent to PEM. The 1*1 convolution layer inside PEM generates a predicted multiplication map and addition map by performing independent linear transformation on each pixel of the feature map. These two maps are used as queries (Q) for subsequent feature fusion.

[0080] Through the PEM module, local details in the image are significantly enhanced, which is especially important for recognition tasks (such as license plate recognition, pedestrian detection, etc.).

[0081] S122. The global branch network fully connects each frame of the street light video to generate global information, and uses a cross-attention mechanism to combine the global information with the transformation result to generate a color correction matrix and a gamma correction value. The color correction matrix and the gamma correction value are used to adjust each frame of the street light video to obtain an adjustment result.

[0082] In this embodiment, GPM is responsible for capturing and generating global illumination adjustment parameters, including color correction matrix and gamma correction value. These parameters are used to adjust the overall illumination conditions of the image so that the image remains consistent and high quality in different environments.

[0083] Starting from each frame, each frame first passes through a fully connected layer to generate a value vector (V), which represents the value vector of global information. Then, another fully connected layer generates a key vector (K). K and V are combined with the query (Q) generated by PEM through a cross-attention mechanism to form the final color correction matrix and gamma correction value.

[0084] The color correction matrix and gamma correction values ​​generated by PEM and GPM are applied to the original input image. Specifically, the original image is reshaped into a form suitable for these parameters, and then the color and brightness of the image are finely adjusted through matrix operations and gamma correction.

[0085] The final output is a lighting-adjusted image, which has better lighting uniformity and visual effects while maintaining the original details.

[0086] The global illumination adjustment parameters generated by the GPM module can effectively improve the overall lighting conditions of the image, so that the image can maintain high quality in different environments.

[0087] like Figure 2After each frame of the picture shown in FIG. 1 is processed by the above process, the adjustment result can be obtained, such as Figure 3 As shown in the figure, the Transformer-based architecture enables the model to maintain high efficiency when processing large-scale image data, which is suitable for real-time or near-real-time application scenarios. The network structure is flexible, and the specific parameters of PEM and GPM can be adjusted according to different application scenarios to adapt to different lighting conditions and image types. Through local and global dual adjustments, the image performance in various complex environments is more stable, reducing the problems of false detection and missed detection caused by lighting changes.

[0088] In summary, the image illumination adjustment network not only technically realizes the effective optimization of image illumination conditions, but also provides strong support for practical applications, especially in image processing tasks in dynamic environments.

[0089] S130. Input all the frame images in the adjustment result into a street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the detection frame of the street lamp and the corresponding under-lighting state.

[0090] In this embodiment, the detection frame refers to the position information of each street lamp, and the lack-lighting state refers to the on / off state of the street lamp in each detection frame.

[0091] In one embodiment, the above-mentioned step S130 may include steps S151 to S155.

[0092] S151. Input the adjustment result into a street lamp under-lighting detection model, divide the adjustment result into windows of fixed size through a convolutional layer, splice the split images in the depth direction, and perform linear transformation on the spliced ​​results through a fully connected layer to obtain a feature map.

[0093] In this embodiment, the feature map refers to an intermediate representation obtained after convolution, concatenation and linear transformation, such as Figure 4 shown.

[0094] The preprocessed image is input into the street light under-lighting detection model. The model first divides the image into small windows of fixed size through a series of convolutional layers. Then, these small windows are spliced ​​together in the depth dimension to form a new feature map. Finally, a fully connected layer is used to perform a linear transformation on the spliced ​​feature map, with the goal of reducing the spatial dimensions of the feature map (including width and height) while increasing the number of channels (depth).

[0095] This step helps improve the model's ability to capture local details of the image, while reducing the amount of data for subsequent processing and improving computing efficiency.

[0096] S152: Perform feature cyclic shift on the feature map to obtain a temporary window.

[0097] In this embodiment, the temporary window refers to a window obtained after characteristic cyclic shift.

[0098] Specifically, each window corresponding to the feature map is translated to the left by half of the size of the window itself, and then translated upward by half of the size of the window itself, so as to obtain a temporary window.

[0099] For the feature map obtained in the previous step, each window is shifted to the left by half of its own size, and then shifted upward by half of its own size. This shift operation causes the originally non-adjacent features to be rearranged into the same window, thus forming a temporary window.

[0100] In this way, the range of feature interactions within the local window can be effectively expanded, allowing the model to calculate self-attention in a wider area and thus better understand the relationship between different regions.

[0101] S153. Perform masked multi-head self-attention calculation on the temporary window to obtain an attention score, and use the attention score to update the feature map.

[0102] In this embodiment, a masked multi-head self-attention mechanism is applied to the temporary window formed by step S152 to calculate the attention scores between each feature point, and update the feature map accordingly. Specifically, a mask is applied to the calculated attention score matrix. An appropriate mask is applied to the attention score matrix to prevent certain unnecessary information from flowing. This step ensures that the model can focus on important feature areas more effectively while avoiding unnecessary computational overhead. The Softmax function is applied to the masked attention scores to obtain a standardized attention distribution matrix, in which each element represents the degree of attention of a feature point to all other feature points. The value vector is weighted and summed using the standardized attention distribution matrix to generate a new feature representation. This step actually recombines the information of each feature point according to its importance to form part of the updated feature map. The above steps are performed in one "head". In the multi-head self-attention mechanism, multiple such "heads" are run in parallel, each with an independent W. Q , W K and W V The weight matrix is ​​used to capture the relationship between feature points from different angles. The output feature vectors of all heads are concatenated and then passed through an additional linear transformation W O Integrate into a new feature vector. This new feature vector is the feature representation of the corresponding position in the updated feature map, thereby achieving the updated feature map.

[0103] Using masked multi-head self-attention can not only improve the model's ability to understand the local structure of the image, but also ensure that the model focuses on more important feature areas and improves detection accuracy.

[0104] S154. Perform a reverse cyclic shift operation on the feature map to restore the feature to its original position in the adjustment result to obtain a processed feature map.

[0105] In this embodiment, the processed feature map refers to the result obtained by performing a reverse cyclic shift operation on the updated feature map.

[0106] Specifically, after completing the self-attention calculation, the opposite operation to step S152 is performed on the feature map, that is, the features are restored to their relative positions in the original image, thereby obtaining the processed feature map.

[0107] This step ensures that the spatial structure of the feature map is consistent with the original image, which facilitates subsequent feature parsing and target positioning.

[0108] S155. Use a decoupling head to predict the street light lack-lighting category and position for the processed feature map, and remove overlapping bounding boxes through non-maximum suppression to obtain recognition results for all frame images.

[0109] In this embodiment, the decoupled head is used to perform classification and regression tasks on the processed feature map, that is, to predict the category and precise location of the street light. The non-maximum suppression (NMS) algorithm is used to remove duplicate bounding boxes and select the most appropriate prediction result as the final output.

[0110] The design of the decoupling head improves the prediction accuracy and robustness of the model, while NMS ensures the uniqueness and accuracy of the output results and avoids misjudgment caused by multiple detections.

[0111] The method of this embodiment combines advanced image processing technology and deep learning models, especially by introducing the self-attention mechanism in the Transformer architecture, which significantly improves the adaptability and detection accuracy of the model in complex scenarios. In addition, through the clever processing of feature maps, such as cyclic displacement and reverse displacement, the expression ability and generalization performance of the model are further enhanced. This method is not only suitable for street light under-lighting detection, but can also be widely used in other tasks that require fine target detection and classification.

[0112] S140, assigning an ID to the detection frame of each frame of the image and performing ID matching, calculating the IoU value between the detection frames with the same ID, and using the recognition result of the next frame of the image to update or confirm the current recognition result of the detection frame to obtain a final recognition result corresponding to each detection frame.

[0113] In this embodiment, after continuously tracking and updating the target in multiple frames of images, the target position corresponding to the detection frame (i.e., the coordinates of the detection frame) and the state of the target (for example, whether the light is on or off) are finally determined. Therefore, the final recognition result corresponding to each detection frame is the recognition result of the detection frame obtained after continuously tracking and updating the target in multiple frames of images.

[0114] In this process, the system processes each frame of the video sequence to achieve continuous tracking of the target and update of the recognition results, thereby improving the accuracy and robustness of target recognition.

[0115] In one embodiment, the above-mentioned step S140 may include steps S141 to S148.

[0116] S141. Assign an ID to the detection frame of each frame of the image.

[0117] In this embodiment, each detection frame is ensured to be uniquely identified to facilitate subsequent tracking and matching. When a target is detected for the first time, a unique ID may be assigned to each detection frame.

[0118] S142: Perform ID matching on the detection frames of all the frames to determine the detection frame with the same ID in each frame.

[0119] In this embodiment, detection frames belonging to the same object in consecutive frames are identified. By comparing the detection frame IDs in different frames, detection frames with the same ID are found. These detection frames are considered to be the representation of the same object at different time points.

[0120] S143, selecting the first frame image as the current frame image.

[0121] S144. Calculate the IoU value between each detection box in the current frame and the detection box with the same ID in a frame after the current frame.

[0122] In this embodiment, the IoU value between the detection boxes with the same ID in the current frame and the next frame is calculated to measure the degree of change of the target position between the two frames.

[0123] S145. Update or confirm the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain an intermediate recognition result corresponding to each detection frame.

[0124] In this embodiment, the intermediate recognition result corresponding to each detection frame refers to the result obtained by updating or confirming the recognition result corresponding to the detection frame of the current frame image using the recognition result corresponding to the detection frame of the following frame image.

[0125] When the IoU value is greater than the set threshold, the recognition result corresponding to the detection frame of the frame following the current frame is used as the intermediate recognition result corresponding to each detection frame; when the IoU value is not greater than the set threshold, the detection frame and the corresponding recognition result corresponding to the IoU value are discarded.

[0126] Specifically, when the IoU value is greater than the set threshold, the detection boxes in the two frames are considered to represent the same target, and the recognition result of the latter frame is used to update the recognition result of the current frame.

[0127] When the IoU value is not greater than the set threshold, it is considered that the detection frames in the two frames may not represent the same target, or the target has changed significantly. In this case, the detection frame and the corresponding recognition result are discarded. This effectively solves practical problems such as repeated recognition of unlit street lights and recognition angles.

[0128] S146, determining whether the next frame of the current frame of the picture is the last frame of the picture;

[0129] S147, if the frame image after the current frame image is not the last frame image, take the frame image after the current frame image as the current frame image, and execute step S144;

[0130] S148. If the frame image following the current frame image is the last frame image, the intermediate recognition result corresponding to each of the detection frames is used as the final recognition result corresponding to each of the detection frames.

[0131] In this embodiment, after all frames in the video sequence are processed, the final recognition result of each detection frame is the intermediate recognition result after multiple updates or confirmations.

[0132] In summary, firstly, a unique ID is assigned to each detection frame in each frame involved in the video captured by the device for subsequent tracking.

[0133] Next, the IoU value between the detection boxes with the same ID in the current frame and the previous frame is calculated; when the IoU value is greater than the preset threshold, the two detection boxes are considered to be matched, their IDs are retained and tracking continues; when the IoU value is less than or equal to the preset threshold, it is considered that the two detection boxes may not belong to the same target and the detection result is discarded.

[0134] Finally, the detection results of subsequent frames are used to update or confirm the recognition results of the current frame; especially when the target position changes or there is occlusion, the recognition accuracy is improved by fusing multi-frame information.

[0135] For example, see Figures 5 to 10 At the initial moment, due to the long distance between the device and the target object, the street lights in the distance (ID 6 and 8) are displayed as not lit due to the perspective problem. As the vehicle drives, the distance between the device and these street lights gradually decreases. At the third moment, the angle of the street light with ID 6 changes, and the detection result is updated accordingly, corrected to the lighted state. At the fourth moment, the street lights with ID 7 and 8 are blocked by road signs, resulting in inability to correctly identify them. At the sixth moment, by calculating the IoU value between the detection frame and the trajectory frame, that is, the intersection over union ratio, it is determined that the detection frames at the fourth moment and the sixth moment are the same target, and the matching detection frame IDs (7 and 8) are retained, and the detection result of the street light with ID 8 is corrected to the lighted state due to the angle change. By judging a series of continuous frames, this method effectively avoids the inaccuracy of judging the state of street lights with a single frame image, and also prevents the problem of repeated identification of street lights without lighting.

[0136] The entire update or confirmation process is as follows:

[0137] First moment: Detection frame ID 6: initially identified as lack of light.

[0138] Detection frame ID 8: Initially identified as lack of light.

[0139] Second moment: Detection frame ID 6: continues to be identified as a lack of light state.

[0140] Detection frame ID 8: Continue to identify it as a lack of light state.

[0141] Moment 3: Detection frame ID 6: As the distance between the device and the target object approaches and the angle changes, the recognition result is corrected to the light-on state.

[0142] Detection frame ID 8: Continue to identify it as a lack of light state.

[0143] Fourth moment: Detection frame ID 6: continues to be identified as a light-on state.

[0144] Detection frame ID 7: Initially recognized as lack of light (blocked by road signs).

[0145] Detection frame ID 8: Continues to be identified as lack of light (blocked by road signs).

[0146] Fifth moment: Detection frame ID 6: continues to be identified as being in the light-on state.

[0147] Detection frame ID 7: Continues to be identified as lack of light (blocked by road signs).

[0148] Detection frame ID 8: Continues to be identified as lack of light (blocked by road signs).

[0149] The sixth moment: Detection frame ID 6: continues to be identified as the light-on state.

[0150] Detection box ID 7: The IoU value between the detection box and the trajectory box meets the threshold, and is ultimately identified as a lack of light state.

[0151] Detection box ID 8: The IoU value between the detection box and the trajectory box meets the threshold and is finally recognized as a light-on state (due to angle changes).

[0152] Final recognition result summary:

[0153] Detection frame ID 6: Finally recognized as the light-on state.

[0154] Detection frame ID 7: Finally identified as lack of light.

[0155] Detection frame ID 8: Finally recognized as the light-on state.

[0156] Through the above process, we can see how the recognition result of each detection frame is gradually corrected and determined as time and device position change. This method effectively improves the accuracy of street light status judgment and reduces misidentification.

[0157] It can be seen that the method of this embodiment achieves high-precision recognition of the state of street lamps by analyzing continuous image frames within a period of time, using ID tracking and IoU calculation. This method not only improves the accuracy of recognition, but also effectively solves the common misrecognition problem in practical applications, and has broad application prospects. Specifically, through multi-frame information fusion, the misrecognition problem caused by perspective angle and occlusion is effectively solved, and the accuracy of the model's judgment on the state of street lamps is improved. The use of ID tracking detection frame avoids repeated recognition of the same target and improves the efficiency of the system. This method can maintain high recognition performance in a variety of complex scenarios (such as different weather conditions and different time periods).

[0158] S150: Output each of the detection frames and the corresponding final recognition results.

[0159] The specific position of the detection frame and the corresponding final recognition result are output to facilitate modification by relevant departments.

[0160] The above-mentioned street lamp under-lighting detection method based on dynamic patrol data can obtain a clearer street lamp image by collecting videos of street lamps during vehicle driving and adjusting the lighting of video frames. Then, each frame of the image is input into the street lamp under-lighting detection model to identify the position and under-lighting state of the street lamp, and determine the detection frame of each street lamp and its under-lighting state. In order to improve the accuracy, an ID is assigned to each detection frame and ID matching is performed, and the IoU value between different frames is calculated to ensure the consistency of the target in the video sequence. By updating or confirming the recognition result of each detection frame, the accurate position and lighting state of each street lamp are finally obtained. This method can effectively and automatically detect the lighting state of street lamps, significantly improve patrol efficiency, solve the shortcomings of traditional manual patrol and single-frame image detection, and improve the accuracy of detection.

[0161] Fig.11 FIG. 3 is a schematic block diagram of a street lamp under-lighting detection system 300 based on dynamic patrol data provided by an embodiment of the present invention. Fig.11 As shown, corresponding to the above street lamp under-lighting detection method based on dynamic patrol data, the present invention also provides a street lamp under-lighting detection system 300 based on dynamic patrol data. The street lamp under-lighting detection system 300 based on dynamic patrol data includes a unit for executing the above street lamp under-lighting detection method based on dynamic patrol data, and the system can be configured in a server. Specifically, please refer to Fig.11 The street lamp under-lighting detection system 300 based on dynamic patrol data includes a video acquisition unit 301 , an adjustment unit 302 , a recognition unit 303 , an update unit 304 and an output unit 305 .

[0162] The video acquisition unit 301 is used to acquire a street lamp video collected by a device installed on a moving vehicle within a period of time; the adjustment unit 302 is used to adjust the lighting of each frame of the street lamp video to obtain an adjustment result; the recognition unit 303 is used to input all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the street lamp detection frame and the corresponding under-lighting state; the update unit 304 is used to assign an ID to the detection frame of each frame image and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame image to update or confirm the current recognition result of the detection frame to obtain the final recognition result corresponding to each detection frame; the output unit 305 is used to output each detection frame and the corresponding final recognition result.

[0163] In one embodiment, the adjustment unit 302 is used to input each frame of the street light video into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

[0164] In one embodiment, the adjusting unit 302 includes:

[0165] A transformation subunit, used for inputting each frame of the street light video into the illumination adjustment model, and performing point-by-point linear transformation on each frame of the street light video by a local branch network to obtain a transformation result;

[0166] The picture adjustment subunit is used to generate global information by fully connecting each frame of the street light video through the global branch network, and combine the global information with the transformation result by using the cross attention mechanism to generate a color correction matrix and a gamma correction value, and use the color correction matrix and the gamma correction value to adjust each frame of the street light video to obtain an adjustment result.

[0167] In one embodiment, the identification unit 303 includes:

[0168] A feature map determination subunit is used to input the adjustment result into a street lamp under-lighting detection model, divide the adjustment result into windows of fixed size through a convolution layer, splice the split images in the depth direction, and perform linear transformation on the spliced ​​results through a fully connected layer to obtain a feature map;

[0169] A cyclic shift subunit, used for performing a characteristic cyclic shift on the characteristic map to obtain a temporary window;

[0170] A multi-head self-attention calculation subunit, configured to perform masked multi-head self-attention calculation on the temporary window to obtain an attention score, and update the feature map using the attention score;

[0171] A reverse cyclic shift operation subunit, used to perform a reverse cyclic shift operation on the feature map to restore the feature to its original position in the adjustment result to obtain a processed feature map;

[0172] The decoupling subunit is used to use the decoupling head to predict the street lamp lack-lighting category and position on the processed feature map, and remove overlapping bounding boxes by non-maximum suppression to obtain recognition results of all frame images.

[0173] In one embodiment, the cyclic shift subunit is used to shift each window corresponding to the feature map to the left by half of the size of the window itself, and then shift it upward by half of the size of the window itself, so as to obtain a temporary window.

[0174] In one embodiment, the updating unit 304 includes:

[0175] An ID allocation subunit, used to allocate an ID to the detection frame of each frame of the picture;

[0176] A matching subunit, configured to perform ID matching on the detection frames of all the frames to determine the detection frames with the same ID in each frame;

[0177] A selection subunit is used to select the first frame image as the current frame image;

[0178] A calculation subunit, configured to calculate an IoU value between each detection frame in the current frame image and the detection frame having the same ID in a frame image after the current frame image;

[0179] An updating subunit, used to update or confirm the recognition results corresponding to all the detection frames of the current frame image according to the IoU value, so as to obtain an intermediate recognition result corresponding to each detection frame;

[0180] A judging subunit, used to judge whether the next frame of the current frame of the picture is the last frame of the picture;

[0181] A current frame picture determination subunit is used for, if the frame picture after the current frame picture is not the last frame picture, taking the frame picture after the current frame picture as the current frame picture, and performing the IoU value of each detection frame in the current frame picture and the detection frame with the same ID in the frame picture after the current frame picture;

[0182] The result determination subunit is used to use the intermediate recognition result corresponding to each detection frame as the final recognition result corresponding to each detection frame if the frame following the current frame is the last frame.

[0183] In one embodiment, the updating subunit is used to use the recognition result corresponding to the detection frame of the frame image after the current frame image as the intermediate recognition result corresponding to each detection frame when the IoU value is greater than the set threshold; when the IoU value is not greater than the set threshold, the detection frame corresponding to the IoU value and the corresponding recognition result are discarded.

[0184] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned street light under-lighting detection system 300 based on dynamic patrol data and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0185] The street lamp under-lighting detection system 300 based on dynamic patrol data can be implemented in the form of a computer program. Fig.12 Runs on the computer device shown.

[0186] See also Fig.12 , Fig.12 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0187] See also Fig.12 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0188] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a street lamp under-lighting detection method based on dynamic patrol data.

[0189] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0190] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a street lamp under-lighting detection method based on dynamic patrol data.

[0191] The network interface 505 is used to communicate with other devices over the network. Fig.12 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0192] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0193] Obtain street light videos collected by a device installed on a moving vehicle over a period of time;

[0194] Performing illumination adjustment on each frame of the street light video to obtain an adjustment result;

[0195] Input all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the detection frame of the street lamp and the corresponding under-lighting state;

[0196] Assign an ID to the detection frame of each frame and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame to update or confirm the current recognition result of the detection frame to obtain the final recognition result corresponding to each detection frame;

[0197] Output each of the detection boxes and the corresponding final recognition results.

[0198] In one embodiment, when the processor 502 implements the step of adjusting the illumination of each frame of the street light video to obtain an adjustment result, the processor 502 specifically implements the following steps:

[0199] Each frame of the street light video is input into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

[0200] In one embodiment, when the processor 502 implements the step of inputting each frame of the street light video into the illumination adjustment model to perform illumination adjustment to obtain an adjustment result, the processor 502 specifically implements the following steps:

[0201] Each frame of the street light video is input into the illumination adjustment model, and the local branch network performs point-by-point linear transformation on each frame of the street light video to obtain a transformation result; the global branch network fully connects each frame of the street light video to generate global information, and uses a cross-attention mechanism to combine the global information with the transformation result to generate a color correction matrix and a gamma correction value, and uses the color correction matrix and the gamma correction value to adjust each frame of the street light video to obtain an adjustment result.

[0202] In one embodiment, when the processor 502 implements the step of inputting all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state to obtain the recognition results of all the frame images, the processor 502 specifically implements the following steps:

[0203] The adjustment result is input into the street light under-lighting detection model, the adjustment result is divided into windows of fixed size through the convolution layer, the divided pictures are spliced ​​in the depth direction, and the spliced ​​result is linearly transformed through the fully connected layer to obtain a feature map; the feature map is subjected to feature cyclic shift to obtain a temporary window; the temporary window is subjected to masked multi-head self-attention calculation to obtain an attention score, and the feature map is updated using the attention score; the feature map is subjected to a reverse cyclic shift operation to restore the feature to its original position in the adjustment result to obtain a processed feature map; the processed feature map is subjected to a decoupling head to predict the street light under-lighting category and position, and overlapping bounding boxes are removed through non-maximum suppression to obtain recognition results of all frame images.

[0204] In one embodiment, when the processor 502 implements the step of performing a feature cyclic shift on the feature map to obtain a temporary window, the processor 502 specifically implements the following steps:

[0205] Each window corresponding to the feature map is translated to the left by half of its own size, and then translated upward by half of its own size to obtain a temporary window.

[0206] In one embodiment, the processor 502 implements the steps of assigning an ID to the detection frame of each frame of the picture and performing ID matching, calculating the IoU value between the detection frames with the same ID, and updating or confirming the current recognition result of the detection frame using the recognition result of the next frame of the picture to obtain the final recognition result corresponding to each detection frame, and specifically implements the following steps:

[0207] Assign an ID to the detection frame of each frame image; perform ID matching on the detection frames of all the frame images to determine the detection frames with the same ID in each frame image; select the first frame image as the current frame image; calculate the IoU value of each detection frame in the current frame image and the detection frame with the same ID in the frame image after the current frame image; update or confirm the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain the intermediate recognition result corresponding to each detection frame; determine whether the frame image after the current frame image is the last frame image; if the frame image after the current frame image is not the last frame image, take the frame image after the current frame image as the current frame image, and execute the IoU value of each detection frame in the current frame image and the detection frame with the same ID in the frame image after the current frame image; if the frame image after the current frame image is the last frame image, take the intermediate recognition result corresponding to each detection frame as the final recognition result corresponding to each detection frame.

[0208] In one embodiment, when the processor 502 implements the step of updating or confirming the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain the intermediate recognition result corresponding to each detection frame, the processor 502 specifically implements the following steps:

[0209] When the IoU value is greater than the set threshold, the recognition result corresponding to the detection frame of the frame following the current frame is used as the intermediate recognition result corresponding to each detection frame; when the IoU value is not greater than the set threshold, the detection frame and the corresponding recognition result corresponding to the IoU value are discarded.

[0210] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0211] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0212] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0213] Obtain a street lamp video collected by a device installed on a moving vehicle within a period of time; perform lighting adjustment on each frame of the street lamp video to obtain an adjustment result; input all the frame images in the adjustment result into a street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the street lamp detection frame and the corresponding under-lighting state; assign an ID to the detection frame of each frame image and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame image to update or confirm the current recognition result of the detection frame, so as to obtain the final recognition result corresponding to each detection frame; output each detection frame and the corresponding final recognition result.

[0214] In one embodiment, when the processor executes the computer program to implement the step of adjusting the illumination of each frame of the street light video to obtain an adjustment result, the processor specifically implements the following steps:

[0215] Each frame of the street light video is input into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

[0216] In one embodiment, when the processor executes the computer program to implement the step of inputting each frame of the street light video into the illumination adjustment model to perform illumination adjustment to obtain an adjustment result, the processor specifically implements the following steps:

[0217] Each frame of the street light video is input into the illumination adjustment model, and the local branch network performs point-by-point linear transformation on each frame of the street light video to obtain a transformation result; the global branch network fully connects each frame of the street light video to generate global information, and uses a cross-attention mechanism to combine the global information with the transformation result to generate a color correction matrix and a gamma correction value, and uses the color correction matrix and the gamma correction value to adjust each frame of the street light video to obtain an adjustment result.

[0218] In one embodiment, when the processor executes the computer program to implement the step of inputting all frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state to obtain the recognition results of all frame images, the processor specifically implements the following steps:

[0219] The adjustment result is input into the street light under-lighting detection model, the adjustment result is divided into windows of fixed size through the convolution layer, the divided pictures are spliced ​​in the depth direction, and the spliced ​​result is linearly transformed through the fully connected layer to obtain a feature map; the feature map is subjected to feature cyclic shift to obtain a temporary window; the temporary window is subjected to masked multi-head self-attention calculation to obtain an attention score, and the feature map is updated using the attention score; the feature map is subjected to a reverse cyclic shift operation to restore the feature to its original position in the adjustment result to obtain a processed feature map; the processed feature map is subjected to a decoupling head to predict the street light under-lighting category and position, and overlapping bounding boxes are removed through non-maximum suppression to obtain recognition results of all frame images.

[0220] In one embodiment, when the processor executes the computer program to implement the step of performing a feature cyclic shift on the feature map to obtain a temporary window, the processor specifically implements the following steps:

[0221] Each window corresponding to the feature map is translated to the left by half of its own size, and then translated upward by half of its own size to obtain a temporary window.

[0222] In one embodiment, when the processor executes the computer program to implement the steps of assigning an ID to the detection frame of each frame of the picture and performing ID matching, calculating the IoU value between the detection frames with the same ID, and updating or confirming the current recognition result of the detection frame using the recognition result of the next frame of the picture to obtain the final recognition result corresponding to each detection frame, the following steps are specifically implemented:

[0223] Assign an ID to the detection frame of each frame image; perform ID matching on the detection frames of all the frame images to determine the detection frames with the same ID in each frame image; select the first frame image as the current frame image; calculate the IoU value of each detection frame in the current frame image and the detection frame with the same ID in the frame image after the current frame image; update or confirm the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain the intermediate recognition result corresponding to each detection frame; determine whether the frame image after the current frame image is the last frame image; if the frame image after the current frame image is not the last frame image, take the frame image after the current frame image as the current frame image, and execute the IoU value of each detection frame in the current frame image and the detection frame with the same ID in the frame image after the current frame image; if the frame image after the current frame image is the last frame image, take the intermediate recognition result corresponding to each detection frame as the final recognition result corresponding to each detection frame.

[0224] In one embodiment, when the processor executes the computer program to implement the step of updating or confirming the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain the intermediate recognition result corresponding to each detection frame, the following steps are specifically implemented:

[0225] When the IoU value is greater than the set threshold, the recognition result corresponding to the detection frame of the frame following the current frame is used as the intermediate recognition result corresponding to each detection frame; when the IoU value is not greater than the set threshold, the detection frame and the corresponding recognition result corresponding to the IoU value are discarded.

[0226] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0227] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0228] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0229] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the system of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0231] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A street lamp under-lighting detection method based on dynamic patrol data, characterized in that: include: Obtain street light videos collected by a device installed on a moving vehicle over a period of time; Performing illumination adjustment on each frame of the street light video to obtain an adjustment result; Input all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the detection frame of the street lamp and the corresponding under-lighting state; Assign an ID to the detection frame of each frame and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame to update or confirm the current recognition result of the detection frame to obtain the final recognition result corresponding to each detection frame; Output each of the detection frames and the corresponding final recognition results; The step of inputting all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state to obtain the recognition results of all the frame images includes: The adjustment result is input into the street lamp under-lighting detection model, the adjustment result is divided into windows of fixed size through a convolution layer, the divided pictures are spliced ​​in the depth direction, and the spliced ​​results are linearly transformed through a fully connected layer to obtain a feature map; Performing feature cyclic shift on the feature map to obtain a temporary window; Applying masked multi-head self-attention calculation to the temporary window to obtain an attention score, and updating the feature map using the attention score; Performing a reverse cyclic shift operation on the feature map to restore the feature to an original position in the adjustment result to obtain a processed feature map; A decoupling head is used to predict the street light lack-lighting category and position for the processed feature map, and overlapping bounding boxes are removed by non-maximum suppression to obtain recognition results for all frame images.

2. The street lamp under-lighting detection method based on dynamic patrol data according to claim 1, characterized in that: The step of performing illumination adjustment on each frame of the street light video to obtain an adjustment result includes: Each frame of the street light video is input into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

3. The street lamp under-lighting detection method based on dynamic patrol data according to claim 2, characterized in that: The step of inputting each frame of the street light video into the illumination adjustment model to perform illumination adjustment to obtain an adjustment result includes: Input each frame of the street light video into the illumination adjustment model, and use the local branch network to perform point-by-point linear transformation on each frame of the street light video to obtain a transformation result; The global branch network fully connects each frame of the street light video to generate global information, and uses a cross-attention mechanism to combine the global information with the transformation result to generate a color correction matrix and a gamma correction value. The color correction matrix and the gamma correction value are used to adjust each frame of the street light video to obtain an adjustment result.

4. The street lamp under-lighting detection method based on dynamic patrol data according to claim 1, characterized in that: The performing a feature cyclic shift on the feature map to obtain a temporary window includes: Each window corresponding to the feature map is translated to the left by half of its own size, and then translated upward by half of its own size to obtain a temporary window.

5. The street lamp under-lighting detection method based on dynamic patrol data according to claim 1, characterized in that: The step of assigning an ID to the detection frame of each frame and performing ID matching, calculating the IoU value between the detection frames with the same ID, and updating or confirming the current recognition result of the detection frame using the recognition result of the next frame to obtain the final recognition result corresponding to each detection frame includes: Assigning an ID to the detection frame of each frame of the image; Performing ID matching on the detection frames of all the frames to determine the detection frames with the same ID in each frame; Select the first frame as the current frame; The IoU value between each detection frame in the current frame and the detection frame with the same ID in a frame after the current frame; Update or confirm the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain an intermediate recognition result corresponding to each detection frame; Determine whether the next frame of the current frame is the last frame of the current frame; If the frame image after the current frame image is not the last frame image, the frame image after the current frame image is used as the current frame image, and the IoU value of each detection frame in the current frame image and the detection frame with the same ID in the frame image after the current frame image is performed; If the frame image following the current frame image is the last frame image, the intermediate recognition result corresponding to each of the detection frames is used as the final recognition result corresponding to each of the detection frames.

6. The street lamp under-lighting detection method based on dynamic inspection data according to claim 5, characterized in that: The updating or confirming the recognition results corresponding to all the detection frames of the current frame image according to the IoU value to obtain the intermediate recognition result corresponding to each detection frame includes: When the IoU value is greater than the set threshold, the recognition result corresponding to the detection frame of the frame following the current frame is used as the intermediate recognition result corresponding to each detection frame; when the IoU value is not greater than the set threshold, the detection frame and the corresponding recognition result corresponding to the IoU value are discarded.

7. The street lamp under-lighting detection system based on dynamic patrol data is characterized by: include: A video acquisition unit, used to acquire street lamp videos collected by a device installed on a moving vehicle over a period of time; An adjustment unit, used for adjusting the illumination of each frame of the street light video to obtain an adjustment result; A recognition unit, used for inputting all the frame images in the adjustment result into the street lamp under-lighting detection model to identify the street lamp position and under-lighting state, so as to obtain the recognition results of all the frame images, wherein the recognition results include the detection frame of the street lamp and the corresponding under-lighting state; An updating unit, configured to assign an ID to the detection frame of each frame and perform ID matching, calculate the IoU value between the detection frames with the same ID, and use the recognition result of the next frame to update or confirm the current recognition result of the detection frame to obtain a final recognition result corresponding to each detection frame; An output unit, used to output each of the detection frames and the corresponding final recognition result; The identification unit comprises: A feature map determination subunit is used to input the adjustment result into a street lamp under-lighting detection model, divide the adjustment result into windows of fixed size through a convolution layer, splice the split images in the depth direction, and perform linear transformation on the spliced ​​results through a fully connected layer to obtain a feature map; A cyclic shift subunit, used for performing a characteristic cyclic shift on the characteristic map to obtain a temporary window; A multi-head self-attention calculation subunit, configured to perform masked multi-head self-attention calculation on the temporary window to obtain an attention score, and update the feature map using the attention score; A reverse cyclic shift operation subunit, used to perform a reverse cyclic shift operation on the feature map to restore the feature to its original position in the adjustment result to obtain a processed feature map; The decoupling subunit is used to use the decoupling head to predict the street lamp lack-lighting category and position on the processed feature map, and remove overlapping bounding boxes by non-maximum suppression to obtain recognition results of all frame images.

8. The street lamp under-lighting detection system based on dynamic patrol data according to claim 7, characterized in that: The adjustment unit is used to input each frame of the street light video into the lighting adjustment model for lighting adjustment to obtain an adjustment result; wherein the lighting adjustment model includes a local branch network for enhancing local feature details and a global branch network for generating a color correction matrix and a gamma correction value and adjusting each frame of the street light video.

9. The street lamp under-lighting detection system based on dynamic patrol data according to claim 8, characterized in that: The adjustment unit comprises: A transformation subunit, used for inputting each frame of the street light video into the illumination adjustment model, and performing point-by-point linear transformation on each frame of the street light video by a local branch network to obtain a transformation result; The picture adjustment subunit is used to generate global information by fully connecting each frame of the street light video through the global branch network, and combine the global information with the transformation result by using the cross attention mechanism to generate a color correction matrix and a gamma correction value, and use the color correction matrix and the gamma correction value to adjust each frame of the street light video to obtain an adjustment result.

Images