Image recognition-based street lamp lighting monitoring method, system, terminal, medium and lighting system

Abstract

This invention provides a street lighting monitoring method, system, terminal, medium, and lighting system based on image recognition. It involves segmenting street lighting areas from real-time acquired street lighting scene images, extracting street lighting mask images from these areas, filtering the effective street lighting areas within the mask images, identifying faulty streetlights within multiple street lighting blocks divided by these effective areas, and finally uploading the location information of each faulty streetlight to a monitoring and control center for monitoring. This invention, through processing and segmenting real-time street lighting scene images, can accurately identify faulty streetlights on roads, effectively utilize electrical resources, minimize energy waste, improve urban infrastructure management, and effectively save manpower, material resources, and financial resources.

Description

Technical Field

[0001] This invention relates to the field of lighting, and in particular to a street lighting monitoring method, system, terminal, medium, and lighting system based on image recognition. Background Technology

[0002] With the promotion and development of smart city construction across various regions, road lighting systems have gradually gained importance. Traditional lighting systems typically operate on a zone-by-zone basis, with streetlights switched on and off according to their latitude and longitude and varying weather conditions. These systems maintain a fixed brightness, requiring constant on-site inspections and maintenance to check for malfunctions or damage. This wastes significant lighting, human, financial, and material resources. In today's energy-scarce and information-driven world, traditional street lighting systems can no longer meet current demands. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a street light monitoring method, system, terminal, medium and lighting system based on image recognition, so as to solve the above-mentioned problems of the prior art.

[0004] To achieve the above and other related objectives, this invention provides a street lighting monitoring method based on image recognition. The method includes: acquiring real-time street lighting scene images; segmenting the street lighting scene images into street lighting regions and extracting street lighting mask images from the street lighting regions; filtering valid street lighting regions in the street lighting mask images and identifying faulty streetlights in multiple street lighting blocks divided by the valid street lighting regions; locating each faulty streetlight and generating corresponding faulty streetlight location information; and uploading the faulty streetlight location information to a monitoring and control center for monitoring.

[0005] In one embodiment of the present invention, the step of segmenting the street lamp scene image into street lamp regions and segmenting a street lamp mask image from the street lamp regions includes: obtaining a feature map corresponding to the street lamp scene image based on a feature extraction network; generating one or more detection boxes corresponding to the street lamp regions based on the feature map of the street lamp scene image based on a target detection framework; and predicting instance masks of the images within each detection box based on a semantic segmentation model to output the corresponding street lamp mask image; wherein, the street lamp regions in the street lamp mask image are RGB color images, and the non-street lamp regions are black images.

[0006] In one embodiment of the present invention, the step of filtering the effective street light areas in the street light mask image and identifying the faulty street lights in the multiple street light blocks divided by the effective street light areas includes: filtering the effective street light areas in the street light mask image based on the acquisition environment information of the street light scene image; dividing the effective street light areas into multiple street light blocks according to the number of street lights; characterizing the street light brightness value in each street light block; and identifying the faulty street lights in each street light block based on the brightness value of each street light.

[0007] In one embodiment of the present invention, the description of the street light brightness value of each street light in each street light block includes: transferring each street light block from the RGB color space to the YUV color space, and extracting the Y component therein to describe the street light brightness value of each street light block.

[0008] In one embodiment of the present invention, the step of determining the faulty streetlights in each streetlight block based on the brightness values ​​of each streetlight includes: determining whether the streetlight lighting status of each streetlight block is normal based on the streetlight brightness values ​​of each streetlight block and the average brightness value of all streetlight blocks calculated from the streetlight brightness values; and determining the faulty streetlights in the streetlight blocks with abnormal lighting status based on the streetlight brightness values ​​of each streetlight in the streetlight blocks with abnormal lighting status and the streetlights in their adjacent blocks.

[0009] In one embodiment of the present invention, determining whether the street lighting status of each street light block is normal based on the street light brightness value of each street light block and the average brightness of all street light blocks calculated from the street light brightness values ​​includes: calculating the brightness deviation of each street light block based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness values; comparing the brightness deviation of each street light block with a set street light block threshold; if the brightness deviation of a street light block is less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be normal; if the brightness deviation of a street light block is not less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be abnormal.

[0010] In one embodiment of the present invention, the step of determining the streetlights with lighting faults in a streetlight block with abnormal lighting conditions based on the streetlight brightness values ​​of each streetlight in the streetlight block with abnormal lighting conditions and its adjacent blocks includes: statistically analyzing the brightness values ​​of all streetlights in the streetlight block with abnormal lighting conditions and its adjacent blocks, and calculating the average brightness value of all streetlights in the streetlight block with abnormal lighting conditions and the brightness value of each streetlight in the streetlight block with abnormal lighting conditions; calculating the brightness deviation from the mean for each streetlight in the streetlight block with abnormal lighting conditions; comparing the brightness deviation from the mean for each streetlight with a set streetlight threshold; if the brightness deviation from the mean for a streetlight is less than the set streetlight threshold, the corresponding streetlight brightness is normal and it is determined not to be a streetlight with lighting faults; if the brightness deviation from the mean for a streetlight is not less than the set streetlight threshold, the corresponding streetlight is off and it is determined to be a streetlight with lighting faults.

[0011] In one embodiment of the present invention, locating each faulty street light and generating faulty street light information includes: traversing the street light mask image from left to right, sequentially numbering all street light blocks and street lights in the street light mask image; and generating lighting faulty street light location information based on the street light determined to be a lighting faulty street light and the number of the street light block in which it is located.

[0012] In one embodiment of the present invention, the environmental information collected includes: road width, street light spacing, and rotation angle of the image acquisition device for collecting street light scene images.

[0013] To achieve the above and other related objectives, this invention provides a street light monitoring system based on image recognition. The system includes: an image acquisition module for acquiring street light scene images captured in real-time by an image acquisition device; a street light region segmentation module connected to the image acquisition module for segmenting street light regions from the street light scene images and extracting street light mask images from the street light regions; a lighting on / off discrimination module connected to the street light region segmentation module for filtering valid street light regions in the street light mask images and identifying faulty street lights in multiple street light blocks divided by the valid street light regions; a street light positioning module connected to the street light region segmentation module and the lighting on / off discrimination module for locating each faulty street light and generating corresponding faulty street light location information; and an information uploading module connected to the street light positioning module for uploading the faulty street light location information to a monitoring and control center for monitoring.

[0014] To achieve the above and other related objectives, the present invention provides a street light monitoring terminal based on image recognition, comprising: one or more memory and one or more processors; the one or more memory is used to store a computer program; the one or more processors are connected to the memory and are used to run the computer program to execute the street light monitoring method based on image recognition.

[0015] To achieve the above and other related objectives, the present invention provides a computer-readable storage medium storing a computer program, which is executed by one or more processors to perform the image recognition-based street lighting monitoring method.

[0016] To achieve the above and other related objectives, the present invention provides a street lighting system, comprising: an image acquisition device, an edge node, an information uploading device, and a central processing unit; wherein, the image acquisition device is used to acquire street lighting scene images in real time; the edge node is communicatively connected to the image acquisition device, and is used to segment the street lighting scene images into street lighting regions, and segment street lighting mask images from the street lighting regions; filter valid street lighting regions in the street lighting mask images, and identify streetlights with lighting faults in multiple street lighting blocks divided by the valid street lighting regions; locate each streetlight with lighting faults and generate corresponding streetlight location information; the information uploading device is communicatively connected to the edge node and the central processing unit, and is used to upload the streetlight location information with lighting faults to the central processing unit for monitoring.

[0017] As described above, this invention is a street lighting monitoring method, system, terminal, medium, and lighting system based on image recognition, which has the following beneficial effects: This invention segments street lighting regions from real-time acquired street lighting scene images, extracts street lighting mask images from these regions, filters valid street lighting regions within the mask images, identifies faulty streetlights in multiple street lighting blocks divided by the valid regions, and finally uploads the location information of each faulty streetlight to the monitoring and control center for monitoring. This invention, through processing and partitioning real-time street lighting scene images, can accurately identify faulty streetlights on roads, effectively utilize power resources, minimize energy waste, improve urban infrastructure management, and effectively save manpower, material resources, and financial resources. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart of a street light monitoring method based on image recognition according to an embodiment of the present invention.

[0019] Figure 2 The diagram shown is a flowchart illustrating a streetlight mask image segmentation method according to an embodiment of the present invention.

[0020] Figure 3 The diagram shown is a flowchart illustrating a street light fault identification method according to an embodiment of the present invention.

[0021] Figure 4 The diagram shown is a flowchart illustrating a street light fault identification method according to an embodiment of the present invention.

[0022] Figure 5 The diagram shows a flowchart of generating location information of a street light with lighting failure in one embodiment of the present invention.

[0023] Figure 6 The diagram shown is a structural schematic of a street lighting monitoring system based on image recognition according to an embodiment of the present invention.

[0024] Figure 7 The diagram shown is a schematic representation of a street lighting system according to an embodiment of the present invention.

[0025] Figure 8 The diagram shown is a schematic representation of a street lighting system according to an embodiment of the present invention.

[0026] Figure 9 The diagram shown is a schematic representation of an application method of a street lighting system according to an embodiment of the present invention.

[0027] Figure 10 The diagram shows a flowchart illustrating how the brightness discrimination module of this invention identifies faulty streetlights.

[0028] Figure 11 The diagram shown is a structural schematic of a street light monitoring terminal based on image recognition according to an embodiment of the present invention. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0030] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0031] Throughout this specification, when it is said that a part is "connected" to another part, this includes not only "direct connection" but also "indirect connection" by placing other elements in between. Furthermore, when it is said that a part "includes" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather means that other constituent elements may also be included.

[0032] The terms "first," "second," and "third," etc., used herein are for the purpose of describing various parts, components, regions, layers, and / or segments, but are not limiting. These terms are used only to distinguish one part, component, region, layer, or segment from others. Therefore, the "first part," "component," "region," "layer," or "segment" described below may refer to a "second part," "component," "region," "layer," or "segment" without departing from the scope of this invention.

[0033] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0034] This invention discloses a street lighting monitoring method based on image recognition. The method involves segmenting real-time street lighting scene images into street lighting regions, extracting street lighting mask images from these regions, filtering the effective street lighting regions within the mask images, identifying faulty streetlights within multiple street lighting blocks divided from the effective regions, and finally uploading the location information of each faulty streetlight to a monitoring and control center for monitoring. This invention, through processing and segmenting real-time street lighting scene images, can accurately identify faulty streetlights on roads, effectively utilize electrical resources, minimize energy waste, improve urban infrastructure management, and effectively save manpower, material resources, and financial resources.

[0035] The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement it. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0036] like Figure 1 This is a flowchart illustrating a street light monitoring method based on image recognition, as shown in an embodiment of the present invention.

[0037] The method includes:

[0038] Step S1: Acquire real-time street light scene images.

[0039] In one embodiment, the acquired street light scene image can be acquired in real time by an image acquisition device. The street light scene image has multiple street lights arranged on both sides of the road, with a certain spacing between the street lights, and each road has a certain width.

[0040] The installation method and shooting angle of the image acquisition equipment are not limited. It can be a fixedly installed image acquisition device in different locations or a mobile image acquisition device. Fixed image acquisition devices include road monitoring cameras used for road surveillance, while mobile image acquisition devices include drones and image acquisition devices mounted on mobile vehicles. The image acquisition device can be any device with image acquisition capabilities, such as a camera.

[0041] Step S2: Perform street lamp region segmentation on the street lamp scene image, and extract the street lamp mask image from the street lamp region.

[0042] In one embodiment, since the light sources in the real-time street light scene images captured within the shooting range of the image acquisition device are not only street lights, but may also include light sources formed by the interior lights of shops along the street, the exterior sign lights, road vehicle lights, decorative lights, indicator lights, etc., which may cause deviations in the judgment of street light illumination conditions, it is necessary to process the real-time street light scene images.

[0043] like Figure 2 As shown, step S2 includes:

[0044] Step S21: Based on the feature extraction network, obtain the feature map corresponding to the input street light scene image;

[0045] Step S22: Based on the target detection framework, generate one or more detection boxes corresponding to the street light region according to the feature map of the street light scene image; that is, input the feature map of the street light scene image into the target detection framework to generate detection boxes for the street light ROI region;

[0046] Step S23: Based on the semantic segmentation model, predict the instance mask of the image within each detection box to output the corresponding street lamp mask image; wherein, the street lamp area in the street lamp mask image is an RGB color image, and the non-street lamp area is a black image. That is, the output image is black except for the street lamps (excluding the lamp poles), and the street lamps (excluding the lamp poles) retain RGB color.

[0047] Step S3: Filter the valid street light areas in the street light mask image, and identify the faulty street lights in the multiple street light blocks divided by the valid street light areas.

[0048] In one embodiment, such as Figure 3 As shown, step S3 includes:

[0049] Step S31: Based on the acquisition environment information of the street light scene image, filter the effective street light areas in the street light mask image;

[0050] Step S32: Divide the effective street light area into multiple street light blocks according to the number of street lights;

[0051] Specifically, considering that traversing rectangular block pixels is less complex than directly traversing irregular street light pixels, this application first divides the valid street light area into blocks to determine whether there are any street lights that are out in each street light block. Then, it traverses the irregular street light pixels within the valid street light area with problems to find the faulty street lights. Since it is only necessary to traverse the street lights within the valid street light area with problems, the computational complexity can be reduced to a certain extent.

[0052] Therefore, the effective street light area is divided into n street light blocks according to the number of street lights. If there is a remainder, the remaining street lights are divided into a separate street light block. For example, every three street lights are divided into a street light block.

[0053] Step S33: Characterize the street light brightness value in each street light block;

[0054] Step S34: Determine the faulty streetlights in each streetlight block based on the brightness values ​​of each streetlight.

[0055] In one embodiment, since the shooting angle and position of the real-time street light scene image are not limited, how to more accurately identify a row of street lights with "non-equal spacing, non-equal size, and non-equal brightness values" is a key problem that needs to be solved in this application; the collected environmental information includes: road width, street light spacing, and rotation angle of the image acquisition device for collecting street light scene images; the effective street light area is filtered according to the road width, street light spacing, and rotation angle of the image acquisition device to exclude street lights that are far away from the image acquisition device.

[0056] In one specific embodiment, characterizing the street light brightness value of each street light in each street light block includes: transferring each street light block from the RGB color space to the YUV color space, and extracting the Y component therein to characterize the street light brightness value of each street light block.

[0057] In one embodiment, the step of determining the faulty streetlights in each streetlight block based on the brightness values ​​of each streetlight includes:

[0058] The street lighting status of each street light block is determined based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value.

[0059] Based on the street light brightness values ​​of the street light blocks with abnormal lighting conditions and their adjacent blocks, the faulty street lights in the street light blocks with abnormal lighting conditions are identified.

[0060] In one specific embodiment, such as Figure 4 As shown, the method of determining whether the street lighting status of each street light block is normal based on the street light brightness value of each street light block and the average brightness of all street light blocks calculated from the street light brightness value includes:

[0061] Based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the brightness values ​​of each street light block, the brightness deviation from the mean of each street light block is calculated; specifically, the deviation from the mean here is the difference between the street light brightness value of one street light block and the average brightness value of all street light blocks.

[0062] The brightness deviation of each street light block is compared with the set threshold for street light blocks;

[0063] If the brightness deviation of the street light block is less than the set street light block threshold, then the street light lighting status of the corresponding street light block is determined to be normal.

[0064] If the brightness deviation of a street light block is not less than the set street light block threshold, then the street light lighting status of the corresponding street light block is determined to be abnormal.

[0065] In one embodiment, such as Figure 4As shown, the streetlights with lighting faults in the streetlight block with abnormal lighting status, determined based on the streetlight brightness values ​​of each streetlight in the streetlight block with abnormal lighting status and its adjacent blocks, include:

[0066] The brightness values ​​of all streetlights in the streetlight block with abnormal lighting status and its adjacent blocks are statistically analyzed, and the average brightness value of all streetlights in the streetlight block and its adjacent blocks, as well as the brightness value of each streetlight in the streetlight block with abnormal lighting status, are calculated. Specifically, if streetlight block i is a non-edge block, the brightness values ​​of all streetlights in streetlight block i, streetlight block i-1, and streetlight block i+1 are statistically analyzed, and the brightness value of each streetlight in streetlight block i and the average brightness value of all streetlights in streetlight block i are calculated respectively. If street light block i is an edge block, for the leftmost first street light block, only the average brightness value of all street lights in the first and second street light blocks needs to be calculated; for the rightmost street light block x, only the average brightness value of all street lights in street light block x and street light block x-1 needs to be calculated. It should be noted that, in order to take into account the special case that all street lights in street light block i are completely off, the statistics of street light blocks and adjacent street light blocks are based on the special case that all street lights in street light blocks are completely off.

[0067] Calculate the luminance deviation from the mean for each street light in a street light block with abnormal lighting conditions; specifically, the deviation from the mean here is the difference between the luminance value of one street light in a street light block with abnormal lighting conditions and the average luminance value of all street lights in the street light block and its adjacent blocks.

[0068] The brightness deviation of each street light from the average value is compared with the set street light threshold.

[0069] If the brightness deviation of a street light is less than the set street light threshold, then the brightness of the corresponding street light is normal, and the corresponding street light is determined not to be a lighting fault street light.

[0070] If the brightness deviation of a street light is not less than the set street light threshold, the corresponding street light will turn off, and the corresponding street light will be determined to be a lighting fault street light.

[0071] Step S4: Locate each faulty street light and generate corresponding location information for the faulty street light.

[0072] In one embodiment, step S4 includes:

[0073] Traverse the street light mask image from left to right, and number all street light blocks and street lights in the street light mask image sequentially;

[0074] The location information of the streetlights that are identified as having lighting malfunctions is generated based on the streetlight number of each streetlight block in which it is located.

[0075] Specifically, such as Figure 5 As shown, step S4 includes:

[0076] Obtain the street lamp mask image after instance segmentation;

[0077] Traverse the street light mask image from left to right, and number all street light blocks and street lights in the street light mask image sequentially;

[0078] Obtain the streetlights identified as having lighting malfunctions and the streetlight block number they belong to;

[0079] The location information of the streetlights that are identified as having lighting malfunctions is generated based on the streetlights and the streetlight block number in which they are located.

[0080] Step S5: Upload the location information of the faulty streetlights to the monitoring and control center for monitoring.

[0081] In one embodiment, after locating the faulty street light and generating its location information, the location information is uploaded to the monitoring and control center. The monitoring and control center then analyzes and reports the status of the faulty street light to staff for subsequent maintenance.

[0082] Similar to the above embodiments, the present invention provides a street light monitoring system based on image recognition.

[0083] The following specific embodiments are provided in conjunction with the accompanying drawings:

[0084] like Figure 6 This invention presents a schematic diagram of a street light monitoring system based on image recognition, as described in an embodiment of the present invention.

[0085] The system includes:

[0086] Image acquisition module 61 is used to acquire street light scene images captured in real time by image acquisition device;

[0087] The street light region segmentation module 62 is connected to the image acquisition module 61 and is used to segment the street light scene image into a street light region and segment out a street light mask image from the street light region.

[0088] The lighting on / off discrimination module 63 is connected to the street light area segmentation module 62 and is used to filter the valid street light area in the street light mask image and identify the lighting fault street lights that exist in the multiple street light blocks divided by the valid street light area.

[0089] The street light positioning module 64 is connected to the street light area segmentation module 62 and the lighting on / off discrimination module 63, and is used to locate each street light with lighting failure and generate corresponding lighting failure street light location information.

[0090] The information upload module 65 is connected to the street light positioning module 64 and is used to upload the location information of the street light with lighting failure to the monitoring and control center for monitoring.

[0091] It should be noted that, as should be understood Figure 6 The division of modules in the system embodiment is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be implemented entirely in hardware; or some modules can be implemented through processing element calls in software, while others are implemented in hardware.

[0092] For example, each module can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0093] Since the implementation principle of the image recognition-based street lighting monitoring system has been described in the foregoing embodiments, it will not be repeated here.

[0094] In one embodiment, the streetlight region segmentation module 62 is used to obtain a feature map corresponding to the input streetlight scene image based on a feature extraction network; generate one or more detection boxes corresponding to the streetlight region based on the feature map of the streetlight scene image based on a target detection framework; and predict the instance mask of the image within each detection box based on a semantic segmentation model to output the corresponding streetlight mask image; wherein, the streetlight region in the streetlight mask image is an RGB color image, and the non-streetlight region is a black image.

[0095] In one embodiment, the lighting on / off discrimination module 63 is used to filter the effective street light area in the street light mask image based on the acquisition environment information of the street light scene image; divide the effective street light area into multiple street light blocks according to the number of street lights; characterize the street light brightness value in each street light block; and determine the faulty street lights in each street light block based on the brightness value of each street light.

[0096] In one embodiment, characterizing the street light brightness value of each street light in each street light block includes: transferring each street light block from the RGB color space to the YUV color space, and extracting the Y component therein to characterize the street light brightness value of each street light block.

[0097] In one embodiment, the step of determining the faulty streetlights in each streetlight block based on the brightness values ​​of each streetlight includes: determining whether the streetlight lighting status of each streetlight block is normal based on the streetlight brightness values ​​of each streetlight block and the average brightness value of all streetlight blocks calculated from the streetlight brightness values ​​of each streetlight; and determining the faulty streetlights in the streetlight blocks with abnormal lighting status based on the streetlight brightness values ​​of each streetlight in the streetlight blocks with abnormal lighting status and the streetlights of their adjacent blocks.

[0098] In one embodiment, determining whether the street lighting status of each street light block is normal based on the street light brightness value of each street light block and the average brightness of all street light blocks calculated from the street light brightness values ​​includes: calculating the brightness deviation of each street light block based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness values; comparing the brightness deviation of each street light block with a set street light block threshold; if the brightness deviation of a street light block is less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be normal; if the brightness deviation of a street light block is not less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be abnormal.

[0099] In one embodiment, determining the faulty streetlights in a streetlight block with abnormal lighting conditions based on the streetlight brightness values ​​of each streetlight in the streetlight block with abnormal lighting conditions and its adjacent blocks includes: statistically analyzing the brightness values ​​of all streetlights in the streetlight block with abnormal lighting conditions and its adjacent blocks, and calculating the average brightness value of all streetlights in the streetlight block with abnormal lighting conditions and the brightness value of each streetlight in the streetlight block with abnormal lighting conditions; calculating the brightness deviation from the average value of each streetlight in the streetlight block with abnormal lighting conditions; comparing the brightness deviation from the average value of each streetlight with a set streetlight threshold; if the brightness deviation from the average value of a streetlight is less than the set streetlight threshold, the corresponding streetlight brightness is normal, and the corresponding streetlight is determined not to be a faulty streetlight; if the brightness deviation from the average value is not less than the set streetlight threshold, the corresponding streetlight is off, and the corresponding streetlight is determined to be a faulty streetlight.

[0100] In one embodiment, the street light positioning module is used to traverse the street light mask image from left to right, and sequentially number all street light blocks and street lights in the street light mask image; and generate lighting fault street light location information based on the street light determined to be a lighting fault street light and the street light block number in which it is located.

[0101] like Figure 7 A schematic diagram of a street lighting system according to an embodiment of the present invention is shown.

[0102] The system includes: an image acquisition device 1, an edge node 2, an information uploading device 3, and a central processing unit 4;

[0103] The image acquisition device 1 is used to acquire street light scene images in real time;

[0104] The edge node 2 is communicatively connected to the image acquisition device 1, and is used to segment the street light scene image into street light regions and segment street light mask images from the street light regions; filter the valid street light regions in the street light mask images, and identify the lighting fault street lights that exist in the multiple street light blocks divided by the valid street light regions; locate each lighting fault street light and generate corresponding lighting fault street light location information;

[0105] The information uploading device 3 is communicatively connected to the edge node 2 and the central processing unit 4, and is used to upload the location information of the faulty streetlights to the central processing unit for monitoring. The central processing unit 4 is a processing device with data processing capabilities, which processes and analyzes the uploaded data of the location information of the faulty streetlights, thereby enabling the monitoring of the lighting status of each streetlight in the image.

[0106] It should be noted that, in order to reduce the bandwidth and latency losses caused by network transmission and multi-level forwarding, the edge nodes of this application are responsible for handling some data processing and data storage at the network edge.

[0107] In a preferred embodiment, the edge node 2 is equipped with an image acquisition module, a street light region segmentation module, a lighting on / off discrimination module, and a street light positioning module; this enables the following: Figure 6 The image acquisition module 61, street light region segmentation module 62, lighting on / off discrimination module 63, and street light positioning module 64 are shown; the information uploading device 3 realizes the following: Figure 6 The functions of the information upload module 65 shown are therefore not elaborated upon here.

[0108] This invention sets up the main data processing and data storage at the edge nodes, which can effectively reduce the bandwidth and latency losses caused by network transmission and multi-level forwarding, and reduce request response time.

[0109] To better illustrate the above-mentioned street lighting system, the present invention provides the following specific embodiments.

[0110] Example 1: A street lighting system. Figure 8 This is a schematic diagram of a street lighting system based on image recognition.

[0111] The system includes: an image acquisition module, an edge node processor, an information upload module, and a central processing unit;

[0112] The system includes an image acquisition module for acquiring real-time street light scene images; an edge node processor for segmenting the input street light scene images into mask images for identifying faulty street lights and locating their location information; and an information upload module for uploading faulty street light information to the central processing unit for data processing and analysis.

[0113] Specifically, the edge node processor includes a street light segmentation module, a brightness / darkness discrimination module, and a street light localization module. The street light segmentation module segments the real-time street light scene image acquired by the image acquisition module into a street light mask image. The brightness / darkness discrimination module filters the valid street light areas in the street light mask image, divides the valid street light areas into n street light blocks, determines whether a faulty street light exists in each block, and further identifies the faulty street lights within the faulty street light blocks. The street light localization module locates the position of the faulty street light and generates faulty street light information.

[0114] like Figure 9 As shown, the application methods of street lighting systems include:

[0115] The image acquisition module acquires real-time street light scene images; the street light scene images are input to the street light segmentation module to segment out street light mask images; the street light mask images are input to the brightness and darkness discrimination module to identify faulty street lights; the street light positioning module acquires the positioning information of the faulty street lights; and the information upload module uploads the faulty street light information.

[0116] Among them, such as Figure 10 As shown, the brightness discrimination module uses the following methods to identify faulty streetlights:

[0117] Obtain the street lamp mask image after instance segmentation;

[0118] After obtaining the street light mask image after instance segmentation, the effective street light area is filtered according to the road width, street light spacing and the rotation angle of the image acquisition device;

[0119] The effective street light area is divided into n street light blocks (n≥3) according to the number of street lights. If there is a remainder, the remaining street lights are divided into a separate street light block.

[0120] Each street light block is transferred from the RGB color space to the YUV color space, and the Y component is extracted to represent the street light brightness value. The average brightness of each street light block and the average brightness of all blocks are calculated respectively.

[0121] First, determine whether the brightness deviation of each street light block i is less than the set street light block threshold. If the brightness deviation of street light block i is less than the set street light block threshold, then determine that all street lights in street light block i are normal. If the brightness deviation of street light block i is greater than or equal to the set street light block threshold, then proceed to the next step of processing.

[0122] The average brightness of all streetlights in streetlight block i, streetlight block i-1, and streetlight block i+1 is calculated (the average brightness of streetlight block i and its left and right streetlight blocks i-1 is calculated here, and streetlight block i+1 is calculated to account for the special case where all streetlights in streetlight block i are completely off). The average brightness of each streetlight (irregular shape) in the streetlight block and the average brightness of all streetlights in the streetlight block are calculated respectively.

[0123] Each street light j is checked to see if its brightness deviation from the average is less than the set street light threshold. If the brightness deviation of street light j is less than the set street light threshold, the street light is considered normal. If the brightness deviation of street light j is greater than or equal to the set street light threshold, the street light is considered faulty. This information is used to locate the faulty street light and generate faulty street light information.

[0124] Therefore, this embodiment has the following advantages:

[0125] 1. By processing and partitioning real-time street light scene images, faulty street lights on the road can be accurately identified, effectively utilizing power resources, minimizing energy waste, improving the management level of urban infrastructure, and effectively saving manpower, material resources, and financial resources.

[0126] 2. Setting up the main data processing and storage on edge nodes can effectively reduce the bandwidth and latency losses caused by network transmission and multi-level forwarding, and reduce request response time.

[0127] like Figure 11 A schematic diagram of the structure of the image recognition-based street lighting monitoring terminal 80 in an embodiment of the present invention is shown.

[0128] The image recognition-based street light monitoring terminal 80 includes a memory 81 and a processor 82. The memory 81 stores a computer program; the processor 82 runs the computer program to implement, for example... Figure 1 The image recognition-based street light monitoring method is described above.

[0129] Optionally, the number of memories 81 can be one or more, and the number of processors 82 can be one or more. Figure 11Each example is taken as an instance.

[0130] Optionally, the processor 82 in the image recognition-based street light monitoring terminal 80 will perform the following actions: Figure 1 The steps described involve loading one or more instructions corresponding to the process of an application into memory 81, and then having the processor 82 run the application stored in the first memory 81, thereby achieving the following: Figure 1 The various functions of the image recognition-based street light monitoring method.

[0131] Optionally, the memory 81 may include, but is not limited to, high-speed random access memory and non-volatile memory. For example, one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices; the processor 82 may include, but is not limited to, a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0132] Optionally, the processor 82 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0133] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed, implements as follows: Figure 1The illustrated method is a street light monitoring method based on image recognition. The computer-readable storage medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Read-Only Optical Disk Memory), magneto-optical disks, ROMs (Read-Only Memory), RAMs (Random Access Memory), EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable Programmable Read-Only Memory), magnetic cards or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product not connected to a computer device or a component used with a computer device.

[0134] In summary, the image recognition-based street lighting monitoring method, system, terminal, medium, and lighting system of this invention segment street lighting regions from real-time acquired street lighting scene images, extract street lighting mask images from these regions, filter valid street lighting regions within the mask images, identify faulty streetlights within multiple street lighting blocks divided by these valid regions, and finally upload the location information of each faulty streetlight to the monitoring and control center for monitoring. This invention, through processing and segmenting real-time street lighting scene images, can accurately identify faulty streetlights on roads, effectively utilize electrical resources, minimize energy waste, improve urban infrastructure management, and effectively save manpower, material resources, and financial resources. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0135] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A street light monitoring method based on image recognition, characterized in that, The method includes: Acquire real-time images of streetlight scenes; The street light scene image is segmented into street light regions, and a street light mask image is segmented from the street light regions. The process involves filtering the effective streetlight areas in the streetlight mask image and identifying faulty streetlights within multiple streetlight blocks divided from the effective streetlight areas. This filtering includes: filtering the effective streetlight areas in the streetlight mask image based on the acquisition environment information of the streetlight scene image; dividing the effective streetlight areas into multiple streetlight blocks according to the number of streetlights; characterizing the brightness value of the streetlights in each streetlight block; and identifying faulty streetlights in each streetlight block based on the brightness values. The method of determining the faulty streetlights in each streetlight block based on the brightness value of each streetlight includes: determining whether the streetlight lighting status of each streetlight block is normal based on the streetlight brightness value of each streetlight block and the average brightness value of all streetlight blocks calculated from the streetlight brightness value of each streetlight; and determining the faulty streetlights in the streetlight block with abnormal lighting status based on the streetlight brightness value of each streetlight in the streetlight block with abnormal lighting status and the streetlight brightness value of each streetlight in its adjacent blocks. The method of determining whether the street lighting status of each street light block is normal based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value of each street light block includes: calculating the brightness deviation of each street light block based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value of each street light block; comparing the brightness deviation of each street light block with a set street light block threshold; if the brightness deviation of a street light block is less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be normal; if the brightness deviation of a street light block is not less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be abnormal. The method for determining the faulty streetlights in a streetlight block with abnormal lighting conditions based on the streetlight brightness values ​​of each streetlight in the streetlight block with abnormal lighting conditions and its adjacent blocks includes: statistically analyzing the brightness values ​​of all streetlights in the streetlight block with abnormal lighting conditions and its adjacent blocks, and calculating the average brightness value of all streetlights in the streetlight block with abnormal lighting conditions and the brightness value of each streetlight in the streetlight block with abnormal lighting conditions; calculating the brightness deviation from the average value of each streetlight in the streetlight block with abnormal lighting conditions; comparing the brightness deviation from the average value of each streetlight with a set streetlight threshold; if the brightness deviation from the average value of a streetlight is less than the set streetlight threshold, the corresponding streetlight brightness is normal, and the corresponding streetlight is determined not to be a faulty streetlight; if the brightness deviation from the average value of a streetlight is not less than the set streetlight threshold, the corresponding streetlight is off, and the corresponding streetlight is determined to be a faulty streetlight. Locate each faulty street light and generate corresponding location information for that street light; The location information of the streetlights with lighting malfunctions is uploaded to the monitoring and control center for monitoring.

2. The street light monitoring method based on image recognition according to claim 1, characterized in that, The step of segmenting the street light scene image into street light regions and segmenting a street light mask image from the street light regions includes: Based on the feature extraction network, feature maps corresponding to the input street light scene image are obtained; Based on the target detection framework, one or more detection boxes corresponding to the street light region are generated according to the feature map of the street light scene image. Based on the semantic segmentation model, the instance mask of the image within each detection box is predicted to output the corresponding street lamp mask image; wherein, the street lamp area in the street lamp mask image is an RGB color image, and the non-street lamp area is a black image.

3. The street light monitoring method based on image recognition according to claim 1, characterized in that, The street light brightness values ​​characterizing each street light in each street light block include: Each street light block is transferred from the RGB color space to the YUV color space, and the Y component is extracted to represent the street light brightness value of each street light block.

4. The street light monitoring method based on image recognition according to claim 1, characterized in that, The process of locating each faulty street light and generating corresponding location information for that street light includes: Traverse the street light mask image from left to right, and number all street light blocks and street lights in the street light mask image sequentially; The location information of the streetlights that are identified as having lighting malfunctions is generated based on the streetlight number of each streetlight block in which it is located.

5. The street light monitoring method based on image recognition according to claim 1, characterized in that, The environmental information collected includes: road width, street light spacing, and the rotation angle of the image acquisition device used to collect street light scene images.

6. A street light monitoring system based on image recognition, characterized in that, The system includes: The image acquisition module is used to acquire street light scene images captured in real time by the image acquisition device; A street light region segmentation module, connected to the image acquisition module, is used to segment the street light scene image into street light regions and segment out a street light mask image from the street light regions. A lighting on / off discrimination module, connected to the street light area segmentation module, is used to filter valid street light areas in the street light mask image and identify faulty street lights in multiple street light blocks divided by the valid street light areas. The filtering of valid street light areas in the street light mask image and the identification of faulty street lights in multiple street light blocks divided by the valid street light areas include: filtering valid street light areas in the street light mask image based on the acquisition environment information of the street light scene image; dividing the valid street light areas into multiple street light blocks according to the number of street lights; characterizing the brightness value of street lights in each street light block; and determining the faulty street lights in each street light block based on the brightness values ​​of each street light. The method of determining the faulty streetlights in each streetlight block based on the brightness value of each streetlight includes: determining whether the streetlight lighting status of each streetlight block is normal based on the streetlight brightness value of each streetlight block and the average brightness value of all streetlight blocks calculated from the streetlight brightness value of each streetlight; and determining the faulty streetlights in the streetlight block with abnormal lighting status based on the streetlight brightness value of each streetlight in the streetlight block with abnormal lighting status and the streetlight brightness value of each streetlight in its adjacent blocks. The method of determining whether the street lighting status of each street light block is normal based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value of each street light block includes: calculating the brightness deviation of each street light block based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value of each street light block; comparing the brightness deviation of each street light block with a set street light block threshold; if the brightness deviation of a street light block is less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be normal; if the brightness deviation of a street light block is not less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be abnormal. The method for determining the faulty streetlights in a streetlight block with abnormal lighting conditions based on the streetlight brightness values ​​of each streetlight in the streetlight block with abnormal lighting conditions and its adjacent blocks includes: statistically analyzing the brightness values ​​of all streetlights in the streetlight block with abnormal lighting conditions and its adjacent blocks, and calculating the average brightness value of all streetlights in the streetlight block with abnormal lighting conditions and the brightness value of each streetlight in the streetlight block with abnormal lighting conditions; calculating the brightness deviation from the average value of each streetlight in the streetlight block with abnormal lighting conditions; comparing the brightness deviation from the average value of each streetlight with a set streetlight threshold; if the brightness deviation from the average value of a streetlight is less than the set streetlight threshold, the corresponding streetlight brightness is normal, and the corresponding streetlight is determined not to be a faulty streetlight; if the brightness deviation from the average value of a streetlight is not less than the set streetlight threshold, the corresponding streetlight is off, and the corresponding streetlight is determined to be a faulty streetlight. The street light positioning module is connected to the street light area segmentation module and the lighting on / off discrimination module, and is used to locate each street light with lighting failure and generate corresponding street light location information. The information upload module is connected to the street light positioning module and is used to upload the location information of the street light with lighting failure to the monitoring and control center for monitoring.

7. A street light monitoring terminal based on image recognition, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The device contains a computer program that is executed by one or more processors to perform the method as described in any one of claims 1 to 5.

9. A street lighting system, characterized in that, The system includes: Image acquisition equipment, edge nodes, information uploading devices, and central processing unit; The image acquisition device is used to acquire street light scene images in real time; The edge node is communicatively connected to the image acquisition device and is used to segment the streetlight scene image into streetlight regions and extract streetlight mask images from the streetlight regions; filter the valid streetlight regions in the streetlight mask images and identify the faulty streetlights in the multiple streetlight blocks divided by the valid streetlight regions; wherein, the filtering of the valid streetlight regions in the streetlight mask images and the identification of the faulty streetlights in the multiple streetlight blocks divided by the valid streetlight regions includes: filtering the valid streetlight regions in the streetlight mask images based on the acquisition environment information of the streetlight scene image; dividing the valid streetlight regions into multiple streetlight blocks according to the number of streetlights; characterizing the streetlight brightness value in each streetlight block; and identifying the faulty streetlights in each streetlight block based on the brightness value of each streetlight. The method of determining the faulty streetlights in each streetlight block based on the brightness value of each streetlight includes: determining whether the streetlight lighting status of each streetlight block is normal based on the streetlight brightness value of each streetlight block and the average brightness value of all streetlight blocks calculated from the streetlight brightness value of each streetlight; and determining the faulty streetlights in the streetlight block with abnormal lighting status based on the streetlight brightness value of each streetlight in the streetlight block with abnormal lighting status and the streetlight brightness value of each streetlight in its adjacent blocks. The method of determining whether the street lighting status of each street light block is normal based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value of each street light block includes: calculating the brightness deviation of each street light block based on the street light brightness value of each street light block and the average brightness value of all street light blocks calculated from the street light brightness value of each street light block; comparing the brightness deviation of each street light block with a set street light block threshold; if the brightness deviation of a street light block is less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be normal; if the brightness deviation of a street light block is not less than the set street light block threshold, then the street lighting status of the corresponding street light block is determined to be abnormal. The method for determining the faulty streetlights in a streetlight block with abnormal lighting conditions based on the streetlight brightness values ​​of each streetlight in the block and its adjacent blocks includes: statistically analyzing the brightness values ​​of all streetlights in the block and its adjacent blocks, and calculating the average brightness value of all streetlights in the block and its adjacent blocks, as well as the brightness value of each streetlight in the block; calculating the brightness deviation from the average for each streetlight in the block; comparing the brightness deviation of each streetlight with a set streetlight threshold; if the brightness deviation is less than the set streetlight threshold, the corresponding streetlight is considered to have normal brightness and is not a faulty streetlight; if the brightness deviation is not less than the set streetlight threshold, the corresponding streetlight is considered to be off and is a faulty streetlight; and locating each faulty streetlight and generating corresponding faulty streetlight location information. The information uploading device is communicatively connected to the edge node and the central processing unit, and is used to upload the location information of the faulty street light to the central processing unit for monitoring.

Images