Image data mining method, device, equipment, roadside equipment, edge computing platform

Abstract

The present disclosure provides an image data mining method, which relates to the fields of intelligent transportation and autonomous driving, and in particular, to the fields of computer vision or image processing. The image data mining method includes acquiring a first frame image and a second frame image that are continuously captured and contain traffic lights; determining whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first differential direction; determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second differential direction; when the state of the lamp head of the target color is lit, determining whether the second frame image is the target image according to the inter-region difference between the first region including the lamp head of the first color, the second region including the lamp head of the second color, and the first target region including the lamp head of the target color in the second frame image. The present disclosure also provides an image data mining device and roadside equipment.

Description

[0001] The present disclosure is a divisional application of the Chinese invention patent with application number 202011009094.0, whose application date is September 23, 2020, and its invention name is Image data mining method, device, equipment and roadside equipment. Technical Field

[0002] The present disclosure relates to the field of intelligent transportation or autonomous driving, specifically, to the field of computer vision or image processing, and more specifically, to an image data mining method, device, equipment, storage medium, roadside equipment and edge computing platform. Background Art

[0003] In intelligent transportation, the driving environment is sensed by using the vehicle-road cooperative technology that combines roadside sensors and vehicle-mounted sensors. Roadside sensors are installed on the roadside and collect image data with a fixed sensing area. The image data collected by the roadside sensors can be used to construct a training data set to train and establish a neural network model to recognize the color of traffic lights. However, since the yellow light data accounts for a small proportion in the training data set, it will affect the recognition effect of the established neural network model, which will lead to the failure of yellow traffic light recognition. Summary of the invention

[0004] In view of this, the present disclosure provides an image data mining method, apparatus, device and storage medium.

[0005] A first aspect of the present disclosure provides an image data mining method, comprising:

[0006] Acquire an image including a traffic light, wherein the traffic light includes a plurality of lamp heads respectively having a first color, a second color, and a target color;

[0007] Determining in the image a first area including lamp heads of the first color, a second area including lamp heads of the second color, and a target area including lamp heads of the target color, wherein the first area, the second area, and the target area have the same shape and size as each other;

[0008] respectively acquiring a first differential value indicating a difference between the target area and the first area, a second differential value indicating a difference between the target area and the second area, and a third differential value indicating a difference between the first area and the second area; and

[0009] Whether the image is a target image is determined based on the first difference value, the second difference value, and the third difference value, and if it is determined that the image is a target image, the target image is stored as result image data of image data mining.

[0010] A second aspect of the present disclosure provides an image data mining method, comprising:

[0011] Acquire a first frame image and a second frame image that are continuously captured and include a traffic light, wherein the traffic light includes a plurality of lamp heads having a first color, a second color, and a target color respectively;

[0012] Determine whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction;

[0013] When it is determined that the lamp cap of the target color is lit, setting the state of the lamp cap of the target color to be lit;

[0014] determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second difference direction;

[0015] When it is determined that the lamp cap of the target color is extinguished, setting the state of the lamp cap of the target color to extinguished;

[0016] When the lamp head of the target color is on, determining whether the second frame image is a target image according to an inter-region difference between a first region including the lamp head of the first color, a second region including the lamp head of the second color, and a first target region including the lamp head of the target color in the second frame image; and

[0017] In the case where it is determined that the second frame image is a target image, the target image is stored as result image data of image data mining.

[0018] A third aspect of the present disclosure provides an image data mining device, comprising:

[0019] An image acquisition module is configured to acquire an image including a traffic light, wherein the traffic light includes a plurality of lamp heads respectively having a first color, a second color, and a target color;

[0020] an area determination module, configured to determine, in the image, a first area including a lamp head of the first color, a second area including a lamp head of the second color, and a target area including a lamp head of the target color, wherein the first area, the second area, and the target area have the same shape and size as each other;

[0021] a differential value acquisition module configured to respectively acquire a first differential value indicating a difference between the target area and the first area, a second differential value indicating a difference between the target area and the second area, and a third differential value indicating a difference between the first area and the second area; and

[0022] The judgment and storage module is configured to determine whether the image is a target image based on the first differential value, the second differential value and the third differential value, and if the image is determined to be a target image, store the target image as result image data of image data mining.

[0023] A fourth aspect of the present disclosure provides an image data mining device, comprising:

[0024] a memory storing program instructions; and

[0025] The processor is configured to execute the program instructions to perform the image data mining method provided according to the first aspect of the present disclosure.

[0026] A fifth aspect of the present disclosure provides an image data mining device, comprising:

[0027] An image acquisition module is configured to acquire a first frame image and a second frame image continuously captured containing a traffic light, wherein the traffic light comprises a plurality of lamp heads having a first color, a second color and a target color respectively;

[0028] A first lighting judgment module is configured to determine whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along a first difference direction;

[0029] A first state setting module is configured to set the state of the lamp head of the target color to be lit when it is determined that the lamp head of the target color is lit;

[0030] an extinguishing judgment module, configured to determine whether the lamp head of the target color is extinguished according to an inter-frame difference between the first frame image and the second frame image along a second difference direction;

[0031] A second state setting module is configured to set the state of the lamp head of the target color to be extinguished when it is determined that the lamp head of the target color is extinguished;

[0032] A second lighting judgment module is configured to determine whether the second frame image is a target image according to an inter-region difference between a first region including the lamp head of the first color, a second region including the lamp head of the second color, and a first target region including the lamp head of the target color in the second frame image when the state of the lamp head of the target color is lit; and

[0033] The result data storage module is configured to store the target image as result image data of image data mining when the second frame image is determined to be the target image.

[0034] A sixth aspect of the present disclosure provides an image data mining device, comprising:

[0035] a memory storing program instructions; and

[0036] The processor is configured to execute the program instructions to perform the image data mining method provided according to the second aspect of the present disclosure.

[0037] A seventh aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the image data mining method as described above.

[0038] An eighth aspect of the present disclosure provides a roadside device, including:

[0039] a memory storing program instructions; and

[0040] The processor is configured to execute the program instructions to perform the image data mining method provided according to the first aspect and the second aspect of the present disclosure.

[0041] A ninth aspect of the present disclosure provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0042] The tenth aspect of the present disclosure provides an edge computing platform, comprising: a memory storing program instructions; and a processor configured to execute the program instructions to execute the image data mining method provided by the first and second aspects of the present disclosure.

[0043] According to the embodiments of the present disclosure, by obtaining a plurality of differential values ​​representing the differences between the regions respectively including the lamp caps of the first color, the lamp caps of the second color, and the lamp caps of the target color, and determining whether the lamp caps of the target color in the image are in the lit state according to the plurality of differential values, the image data containing the traffic lights is screened, thereby realizing fully automatic data mining of the traffic light image data, thereby saving labor costs and time for data annotation of the image data. In addition, by adding lighting inspection and extinguishing inspection for the lamp caps of the target color and other than the target color, the accuracy of data mining is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0045] Figure 1 The system architecture of intelligent traffic vehicle-road collaboration according to an embodiment of the present disclosure is schematically shown;

[0046] Figure 2 The flowchart of the image data mining method according to the embodiment of the present disclosure is schematically shown;

[0047] Figure 3 An example of inter-region difference of the image data mining method according to an embodiment of the present disclosure is schematically shown;

[0048] Figure 4 A flowchart of an image data mining method according to another embodiment of the present disclosure is schematically shown;

[0049] Figure 5 An example of inter-frame difference of an image data mining method according to another embodiment of the present disclosure is schematically shown;

[0050] Figure 6 The execution process of the image data mining method according to the embodiment of the present disclosure is schematically shown;

[0051] Figure 7 A block diagram of an image data mining device according to an embodiment of the present disclosure is schematically shown;

[0052] Figure 8 A block diagram schematically shows an image data mining device according to another embodiment of the present disclosure; and

[0053] Fig. 9 A block diagram of an electronic device suitable for performing image data mining according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0055] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0056] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0057] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0058] When training and establishing a neural network model for light color recognition using a training data set constructed using image data collected by roadside sensors, the yellow light data accounts for a relatively small proportion in the training data set, thereby affecting the recognition effect of the established neural network model. The disclosed embodiment proposes an image data mining method for automatically filtering image data of yellow light on from image data collected using roadside sensors to supplement the training data set, thereby improving the recognition effect of the established neural network model.

[0059] The intelligent transportation vehicle-road collaborative system uses communication technology to achieve real-time information interaction between vehicles, between vehicles and roads, between vehicles and people, and between vehicles and the network. Figure 1 The system architecture of intelligent traffic vehicle-road collaboration according to an embodiment of the present disclosure is schematically shown. Figure 1 As shown, the vehicle-mounted device is set in the automatic driving or assisted driving vehicle 20, and can communicate with the server device 40 to send the driving information of the vehicle to the server device 40 in real time and receive real-time road condition information from the server device. The roadside device 30 is set up on the roadside, including a roadside perception device 301 and a roadside computing device 302. The roadside perception device 301 (such as a roadside camera) is connected to a roadside computing device 302 (such as a roadside computing unit RSCU), and the roadside computing device 302 is connected to the server device 40. The server device 40 can communicate with the automatic driving or assisted driving vehicle 20 in various ways. In another system architecture, the roadside perception device 301 itself includes a computing function, and the roadside perception device 301 can be directly connected to the server device 40. The above connection can be wired or wireless. In this application, the server device 40 can be, for example, a cloud control platform, a vehicle-road collaborative management platform, a central subsystem, an edge computing platform, a cloud computing platform, etc.

[0060] The roadside perception device 301 is used to capture an image of a traffic light including a plurality of lamp heads having red (first color), green (second color) and yellow (target color), respectively. Therefore, the positions of the traffic light and the lamp heads of the traffic light in the image are fixed. Figure 1As shown, in each captured frame of the image, the positions of the red light 11, the green light 12 and the yellow light 13 are the same. Therefore, the color of the lamp head can be determined according to the position of the lamp head. Figure 1 As shown, block 130 (in Figure 1 The position of the area 130 including the yellow light 13 surrounded by the yellow light 13 (indicated by a dotted line in FIG. 1 ) relative to the entire frame image represents the yellow light 13. Similarly, the position of the area 110 including the red light 11 relative to the entire frame image represents the red light 11, and the position of the area 120 including the green light 12 relative to the entire frame image represents the green light 12.

[0061] Figure 2 The flowchart of the image data mining method 200 according to the embodiment of the present disclosure is schematically shown. Figure 2 As shown, the image data mining method 200 includes the following steps:

[0062] In step S210, an image including a traffic light is acquired.

[0063] According to an embodiment, the captured image containing the traffic light is generally a color image. Each pixel in the color image can generally be represented by three color components, namely, a red (R) component, a green (G) component, and a blue (B) component. That is, each pixel corresponds to a three-element array, and each element in the array takes a value between [0,255]. In the process of determining whether the yellow light (the lamp head of the target color) in the image is in a lit state, in order to reduce the amount of calculation, a grayscale image corresponding to the color image is used for processing. In a grayscale image, each pixel has only one sampling value, which can represent different degrees of depth of any color, or different brightness of different colors. A specific algorithm can be used to convert the captured color image into a depth image.

[0064] In step S220, a first region including lamp heads of a first color, a second region including lamp heads of a second color, and a target region including lamp heads of a target color are determined in the image.

[0065] According to an embodiment, the first area may be an area including a red light (a lamp head of a first color). The second area may be an area including a green light (a lamp head of a second color). The determined first area, second area, and target area have the same shape and size as each other. That is, the determined first area, second area, and target area are rectangular areas with equal length and width, and the first area, second area, and target area include the same number of pixels.

[0066] In step S230 , a first differential value indicating a difference between the target area and the first area, a second differential value indicating a difference between the target area and the second area, and a third differential value indicating a difference between the first area and the second area are respectively acquired.

[0067] According to an embodiment, respectively obtaining a first differential value representing the difference between a target area and a first area, a second differential value representing the difference between the target area and the second area, and a third differential value representing the difference between the first area and the second area may further include obtaining differential areas according to the difference in grayscale values ​​of pixels in two areas, between the target area and the first area, between the target area and the second area, and between the first area and the second area, respectively, and binarizing and normalizing the obtained differential areas to obtain the first differential value, the second differential value, and the third differential value.

[0068] In step S240, whether the image is a target image is determined according to the first difference value, the second difference value, and the third difference value, and if it is determined that the image is a target image, the target image is stored as result image data of image data mining.

[0069] According to an embodiment, determining whether an image is a target image based on the first differential value, the second differential value, and the third differential value may further include comparing the absolute value of the first differential value, the absolute value of the second differential value, and the absolute value of the third differential value with the first threshold value and the second threshold value, respectively, and determining the image as a target image when the absolute value of the first differential value and the absolute value of the second differential value are both greater than the first threshold value and the absolute value of the third differential value is less than the second threshold value. After the target image is determined, the target image may be selected and the selected target image may be stored in a specified set. After a certain number of target images are stored in the specified set, the set may be added to the training data set to supplement the yellow light (lamp head of the target color) data.

[0070] It is easy to understand that the above-mentioned image data mining method 200 can be used not only for mining yellow light data, but also for mining red light data and green light data, as long as the red light or green light is used as the lamp head of the target color, and the lamp heads of the other two colors except the red light or green light are used as the lamp heads of the first color and the lamp heads of the second color respectively.

[0071] According to the image data mining method of the embodiment of the present disclosure, the regions including the lamp head of the first color, the lamp head of the second color and the lamp head of the target color can be analyzed by acquiring the first differential value, the second differential value and the third differential value, and whether the lamp head of the target color is lit can be judged based on the difference between the regions, thereby realizing data mining of the lamp head of the specified color. The operation is simple and the execution efficiency is high, which can significantly save labor costs and save time for data annotation of image data.

[0072] Figure 3 An example of the inter-region difference of the image data mining method according to the embodiment of the present disclosure is schematically shown. Figure 3 The process of obtaining the first difference value, the second difference value, and the third difference value, and identifying the target image according to the first difference value, the second difference value, and the third difference value is described in detail. In order to simplify the description, the red light is used as the lamp head of the first color, the green light is used as the lamp head of the second color, and the yellow light is used as the lamp head of the target color. In addition, if Figure 3 As shown, region 310 is a first region including the red light 31, region 320 is a second region including the green light 32, and region 330 is a third region including the yellow light 33, and only three pixels are shown in each region for illustration.

[0073] According to an embodiment, a difference is first performed between any two regions, and the difference may not distinguish directions. For example, the difference between the target region 330 and the first region 310 may be calculated by subtracting the grayscale value of the pixel 3111 in the first region 310 from the grayscale value of the pixel 3311 in the target region 330, subtracting the grayscale value of the pixel 3112 in the first region 310 from the grayscale value of the pixel 3312 in the target region 330, subtracting the grayscale value of the pixel 3113 in the first region 310 from the grayscale value of the pixel 3313 in the target region 330, and so on. It can also be obtained in the following manner: subtract the grayscale value of pixel 3311 in the target area 330 from the grayscale value of pixel 3111 in the first area 310, subtract the grayscale value of pixel 3312 in the target area 330 from the grayscale value of pixel 3112 in the first area 310, subtract the grayscale value of pixel 3313 in the target area 330 from the grayscale value of pixel 3113 in the first area 310... The above differential operation is performed for each pixel in the target area 330 and the first area 310, and the above differential operation is performed between the target area 330 and the second area 320 and the first area 310 and the second area 320 respectively. Through the above differential operation, three differential areas can be obtained respectively, and the pixels in each differential area are the difference between the grayscale values ​​of the pixels in the corresponding two areas.

[0074] Next, binarization processing is performed on each obtained differential region, specifically including setting a threshold value THB for binarization processing, and comparing the values ​​of the pixels in the differential region with the threshold value THB respectively. The values ​​of the pixels whose values ​​are greater than the threshold value THB are set to "0", and the values ​​of the pixels whose values ​​are less than or equal to the threshold value THB are set to "255".

[0075] Next, the normalization process of the binary-processed differential region specifically includes adding the values ​​of each pixel in each binary-processed differential region, and using the sum obtained by the addition as the first differential value, the second differential value, and the third differential value. According to another embodiment, the sum of the obtained pixel values ​​can also be divided by the number of pixels contained in the differential region, that is, the number of pixels contained in the first region 310, the second region 320, and the target region 330, and the obtained quotient is used as the first differential value, the second differential value, and the third differential value. The first differential value, the second differential value, and the third differential value can generally characterize the differences between the first region 310, the second region 320, and the target region 330.

[0076] Next, the absolute value of the first differential value, the absolute value of the second differential value, and the absolute value of the third differential value are compared with the set thresholds respectively. Compared with the unlit lamp head, each pixel in the lit lamp head has a larger grayscale value, so the two differential values ​​in which the lit lamp head participates will have a larger value, and the other differential value in which the unlit lamp head participates will have a smaller value. Thresholds can be set separately, for example, the first threshold and the second threshold are set. According to an embodiment, an image in which the absolute value of the first differential value and the absolute value of the second differential value are both greater than the first threshold and the absolute value of the third differential value is less than the second threshold can be determined as a target image. According to an embodiment, the first threshold and the second threshold can be the same value, so an image in which the absolute value of the first differential value and the absolute value of the second differential value are both greater than the absolute value of the third differential value can be determined as a target image. According to another embodiment, the first threshold can be k times the second threshold, so an image in which the absolute value of the first differential value and the absolute value of the second differential value are k times greater than the absolute value of the third differential value can be determined as a target image.

[0077] It is easy to understand that the first threshold and the second threshold can be set according to an empirical value, or can be set according to the characteristics of the sensor, or can be set according to the requirements of the inspection error or the contrast requirements, etc., as long as the difference between the gray values ​​caused by the lighting of the lamp head can be screened out. The embodiment of the present disclosure does not limit the setting of the first threshold and the second threshold and the setting of the conditions for screening the target image using the first threshold and the second threshold.

[0078] The image data mining method 200 according to the embodiment of the present disclosure is easy to implement and has high execution efficiency, but in the process of executing the operation, it may cause false detection due to the lack of necessary inspection and guarantee means. For example, if the frame surrounding the lamp head is not tight enough, resulting in the formation of a region including several pixels of other background areas other than the lamp head, it is impossible to eliminate such interference by judging only through one frame of image, so false detection may occur.

[0079] Figure 4 The flowchart of the image data mining method 400 according to another embodiment of the present disclosure is schematically shown. Figure 4 As shown, the image data mining method 400 includes the following steps:

[0080] In step S410, a first frame image and a second frame image containing a traffic light that are captured continuously are acquired.

[0081] According to an embodiment, the continuously captured images may be continuous image frames captured by a roadside sensor or continuous image frames obtained from captured video data. The change in the state of the lamp head can be used to determine the change in the state of the lamp head between two continuously captured frames of images, thereby inferring whether the lamp head is on or off. For example, if the grayscale value of the pixel in the area where the yellow light is located changes significantly during the period from the first frame of image to the second frame of image, it is determined that the yellow light is on or off during the period from the first frame of image to the second frame of image.

[0082] In step S420, it is determined whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction.

[0083] According to an embodiment, the first differential direction refers to a direction of differential from the second frame image to the first frame image. According to an embodiment, determining whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first differential direction further includes determining a first target area and a second target area including the lamp head of the target color in the second frame image and the first frame image, respectively, wherein the first target area and the second target area have the same shape and size as each other. The first differential target area is obtained according to the difference between the grayscale values ​​of each pixel in the first target area and each pixel in the second target area, and the first differential target area is binarized and normalized to obtain a target lighting differential value. The target lighting differential value is compared with the first lighting threshold, and when the target lighting differential value is greater than the first lighting threshold, it is determined that the lamp head of the target color is lit.

[0084] In step S430, when it is determined that the lamp head of the target color is lit, the state of the lamp head of the target color is set to be lit.

[0085] According to an embodiment, the state of the lamp head of the target color refers to the static state of the lamp head of the target color in a frame image, which is different from the dynamic state of the lamp head of the target color being lit. For example, the yellow light is lit in the first frame image and remains lit in the second frame image, then it can be determined that the state of the yellow light in the first frame image and the state in the second frame image are both lit, but obviously the yellow light did not light up during the period from the first frame image to the second frame image. The state of the lamp head of the target color being lit means that the lamp head of the target color is lit in the current frame (e.g., the second frame), and can be lit or off in the previous frame (e.g., the first frame).

[0086] In step S440, it is determined whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second difference direction.

[0087] According to an embodiment, the second differential direction refers to a direction of differential from the first frame image to the second frame image. According to an embodiment, determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second differential direction further includes determining a first target area and a second target area including the lamp head of the target color in the second frame image and the first frame image, respectively, and the first target area and the second target area have the same shape and size as each other. The second differential target area is obtained according to the difference between the grayscale value of each pixel in the second target area and the grayscale value of each pixel in the first target area, and the second differential target area is binarized and normalized to obtain a target extinguishing differential value. The target extinguishing differential value is compared with the first extinguishing threshold value, and when the target extinguishing differential value is greater than the first extinguishing threshold value, it is determined that the lamp head of the target color is extinguished.

[0088] In step S450, when it is determined that the lamp head of the target color is extinguished, the state of the lamp head of the target color is set to extinguished.

[0089] According to an embodiment, the state of the lamp head of the target color being off means that the lamp head of the target color is off in the current frame (eg, the second frame), and may be on or off in the previous frame (eg, the first frame).

[0090] In step S460, when the state of the lamp head of the target color is lit, whether the second frame image is the target image is determined based on the inter-area difference between the first area including the lamp head of the first color, the second area including the lamp head of the second color, and the first target area including the lamp head of the target color in the second frame image.

[0091] According to an embodiment, determining whether the second frame image is a target image based on the inter-regional difference between the first region including the lamp head of the first color, the second region including the lamp head of the second color, and the first target region including the lamp head of the target color in the second frame image further includes obtaining a differential region based on the difference in the grayscale value of each pixel in the two regions between the first target region and the first region, between the first target region and the second region, and between the first region and the second region, and performing binarization and normalization on the obtained differential region to obtain a first differential value, a second differential value, and a third differential value. The absolute value of the first differential value, the absolute value of the second differential value, and the absolute value of the third differential value are compared with the first threshold value and the second threshold value, respectively, and when the absolute value of the first differential value and the absolute value of the second differential value are both greater than the first threshold value and the absolute value of the third differential value is less than the second threshold value, the image is determined to be a target image. The operation of determining the target image and the setting of the first threshold value and the second threshold value in this embodiment can be obtained by referring to the method 200 in the aforementioned embodiment, and will not be repeated here.

[0092] In step S470, when it is determined that the second frame image is the target image, the target image is stored as result image data of image data mining.

[0093] According to an embodiment, a target image may be selected and stored in a designated set. If the designated set does not store the required number of target images, the process returns to step S410 to start processing the next frame of image. After a certain number of target images are stored in the designated set, the set may be added to the training data set to supplement the yellow light (the lamp head of the target color) data.

[0094] According to the image data mining method of the embodiment of the present disclosure, by determining whether the lamp head of the target color is lit or extinguished based on the inter-frame difference between two frames of images captured continuously, it is possible to avoid false detection caused by the fact that the lamp head of the target color is not lit due to the inclusion of several pixels of other background areas other than the lamp head in the formed area. In the case of adding the light color lighting check and the light color extinguishing check at the same time, it is helpful to determine the period of time during which the lamp head of the target color remains lit, and the light color lighting check and the light color extinguishing check can be mutually verified, so that in the case of a false detection in the light color lighting check, the false detection can be discovered as early as possible by using the light color extinguishing check. For example, when the yellow light is not lit, it is falsely detected as lit by the light color lighting check. After several steps, it is detected that the green light is extinguished by the light color extinguishing check, and it can be determined that the previously determined yellow light is lit is a false detection. And it is easy to find the image that is stored incorrectly between determining that the yellow light is lit and determining that the green light is extinguished.

[0095] Figure 5An example of inter-frame difference of an image data mining method according to another embodiment of the present disclosure is schematically shown. Figure 5 As shown, 530_1 is the second target area including the lamp head 53_1 of the target color in the first frame image, and 530_2 is the first target area including the lamp head 53_2 of the target color in the second frame image. The dotted line L1 with an arrow indicates the first differential direction. It is realized based on the difference of the grayscale value of each pixel in the first target area 530_2 minus the grayscale value of each pixel in the second target area 530_1 according to the inter-frame difference between the first frame image and the second frame image along the first differential direction. The dotted line L2 with an arrow indicates the second differential direction. It is realized based on the difference of the grayscale value of each pixel in the second target area 530_1 minus the grayscale value of each pixel in the first target area 530_2 according to the inter-frame difference between the first frame image and the second frame image along the second differential direction.

[0096] It can be seen from the method 400 according to the above embodiment that the operation of determining whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction in step S420, and the operation of determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second difference direction in step S440, are only judged for the lamp head of the target color (such as yellow light), and are not judged for lamp heads of other colors (such as red light and green light). In this case, if the yellow light is not correctly detected to be extinguished when the yellow light is extinguished, it is possible to cause frequent invalid operations, thereby increasing the amount of calculation. If the detection of the red light and the green light is added at the same time, then even if the yellow light is not detected to be extinguished when the yellow light is extinguished, it can still be judged whether the yellow light has been extinguished or the previous yellow light is a false detection based on the detected red light or green light. That is, the reliability of the inspection can be further improved by the lighting inspection and extinguishing inspection for lamp heads of other colors.

[0097] According to an embodiment, the lighting inspection for a red light (lamp head of the first color) and a green light (lamp head of the second color) may include: determining a first area and a third area including the lamp head of the first color and a second area and a fourth area including the lamp head of the second color in the second frame image and the first frame image, respectively, and the first area, the second area, the third area and the fourth area have the same shape and size as the first target area and the second target area. Between the first area and the third area and between the second area and the fourth area, respectively, a differential area is obtained according to the difference in the grayscale value of each pixel in the two areas, and the obtained differential area is binarized and normalized to obtain a first lighting differential value and a second lighting differential value. The first lighting differential value and the second lighting differential value are compared with the second lighting threshold respectively, and when the target lighting differential value is greater than the first lighting threshold and the first lighting differential value and the second lighting differential value are both smaller than the second lighting threshold, it is determined that the lamp head of the target color is lit; the target lighting differential value, the first lighting differential value and the second lighting differential value are compared with the second lighting threshold, the third lighting threshold and the fourth lighting threshold respectively, and when the first lighting differential value is greater than the third lighting threshold and the target lighting differential value and the second lighting differential value are both smaller than the second lighting threshold, it is determined that the lamp head of the first color is lit; when the second lighting differential value is greater than the fourth lighting threshold and the target lighting differential value and the first lighting differential value are both smaller than the second lighting threshold, it is determined that the lamp head of the second color is lit.

[0098] According to an embodiment, the extinguishing inspection for red and green lights may include: determining a first area and a third area including a lamp head of a first color and a second area and a fourth area including a lamp head of a second color in a second frame image and a first frame image, respectively, and the first area, the second area, the third area and the fourth area have the same shape and size as the first target area and the second target area. Between the third area and the first area and between the fourth area and the second area, respectively, a differential area is obtained according to the difference in the grayscale value of each pixel in the two areas, and the obtained differential area is binarized and normalized to obtain a first extinguishing differential value and a second extinguishing differential value. The first extinguishing differential value and the second extinguishing differential value are compared with the second extinguishing threshold value, respectively, and when the target extinguishing differential value is greater than the first extinguishing threshold value and the first extinguishing differential value and the second extinguishing differential value are both less than the second extinguishing threshold value, it is determined that the lamp head of the target color is extinguished. The target extinction differential value, the first extinction differential value and the second extinction differential value are compared with the second extinction threshold, the third extinction threshold and the fourth extinction threshold respectively, and when the first extinction differential value is greater than the third extinction threshold and the target extinction differential value and the second extinction differential value are both smaller than the second extinction threshold, it is determined that the lamp head of the first color is extinguished; when the second extinction differential value is greater than the fourth extinction threshold and the target extinction differential value and the first extinction differential value are both smaller than the second extinction threshold, it is determined that the lamp head of the second color is extinguished.

[0099] According to the above embodiment, it is possible to further determine whether the lamp head of the first color and the lamp head of the second color are lit or extinguished. According to an embodiment, method 400 further includes setting the state of the lamp head of the target color to extinguished when it is determined that the lamp head of the first color or the second color is lit. According to an embodiment, method 400 further includes querying the state of the lamp head of the target color when it is determined that no lamp head is lit, and performing operations according to the query results. When the state of the lamp head of the target color is lit, the operation of step S460 in method 400 may continue to be performed. When the state of the lamp head of the target color is extinguished, the processing of the next frame of image may be started directly, or when the lamp color extinguishing check has not been performed, the processing of the next frame of image may be started after the lamp color extinguishing check is performed.

[0100] In addition, it should be noted that, for the identified false detection, if the corresponding image has been stored, you can choose to delete the stored image, or you can keep the image as a normal data image.

[0101] The lighting threshold and the extinguishing threshold involved in each embodiment of the present disclosure can be obtained by the following method: obtain a plurality of frames of images continuously captured containing traffic lights. Between any two frames of images in the plurality of frames, the lighting threshold for determining whether the lamp head of the first color, the second color, and the target color is lit is obtained by calculating the inter-frame difference of the two frames of images along the first differential direction. Between any two frames of images in the plurality of frames, the extinguishing threshold for determining whether the lamp head of the first color, the second color, and the target color is extinguished is obtained by calculating the inter-frame difference of the two frames of images along the second differential direction.

[0102] Figure 6 The execution process of the image data mining method according to the embodiment of the present disclosure is schematically shown. Figure 6As shown, the image data mining method is started, and at step S601, the method starts initialization to obtain multiple thresholds required during the execution of the method. According to an embodiment, the initialization operation may include obtaining the specific position of each lamp head in the traffic light in the image, for example, the coordinates of the upper left corner point and the lower right corner point of the area including the lamp head, and the number of pixels n_pixel in the determined area may be obtained. In the first frame image, the initial state of each lamp head is unknown, and the historical lighting light color is initialized to unknown. Next, multiple lighting thresholds and extinguishing thresholds are obtained through threshold initialization processing. The threshold initialization processing may include setting an initial value. First, the number of frames init_waiting_frame_number of the image required in the threshold initialization processing is set. According to an embodiment, the number of frames init_waiting_frame_number may be set to include one or more complete traffic light cycles, or may be set to a fixed time including multiple traffic light cycles, such as 10 minutes, but the embodiments of the present disclosure are not limited thereto. Then, set the red light on threshold red_max_diff_sum1=0, the green light on threshold green_max_diff_sum1=0, and the yellow light on threshold yellow_max_diff_sum1=0; set the red light off threshold red_max_diff_sum2=0, the green light off threshold green_max_diff_sum2=0, and the yellow light off threshold yellow_max_diff_sum2=0. Then, set the yellow light secondary check flag Flag to 0. The threshold initialization process also includes: acquiring a first frame image; performing inter-frame difference on the first frame image and the second frame image along the first differential direction for the determined area including the lamp head, and binarizing and normalizing the obtained differential image to obtain the first differential threshold, the second differential threshold, and the third differential threshold for the red light, the green light, and the yellow light, respectively. The maximum value among the obtained first differential threshold, the second differential threshold and the third differential threshold is obtained, recorded as max_diff_sum_current1, and the color of the lamp head corresponding to the differential threshold is recorded according to the positions of the lamp heads of the three colors. The differential threshold max_diff_sum_current1 is assigned to the lighting threshold of the lamp head of the corresponding color. Next, for the determined area including the lamp head, inter-frame difference is performed on the first frame image and the second frame image along the second differential direction, and the obtained differential image is binarized and normalized to obtain the first differential threshold, the second differential threshold and the third differential threshold of the red light, the green light and the yellow light, respectively. The maximum value among the obtained first differential threshold, the second differential threshold and the third differential threshold is obtained, recorded as max_diff_sum_current2, and the color of the lamp head corresponding to the differential threshold is recorded according to the positions of the lamp heads of the three colors.The difference threshold max_diff_sum_current2 is assigned to the extinguishing threshold of the lamp head of the corresponding color. The above threshold initialization process is repeatedly performed between any two frames of multiple frames until the processing of init_waiting_frame_number frames is completed, and the lighting threshold and extinguishing threshold of the red light, green light and yellow light are obtained respectively.

[0103] It should be noted that in the above threshold initialization process, the maximum value among the first differential threshold, the second differential threshold and the third differential threshold is selected as max_diff_sum_current1 or max_diff_sum_current2. However, according to the embodiment, other values ​​can be selected as max_diff_sum_current1 or max_diff_sum_current2, for example, the minimum value among the first differential threshold, the second differential threshold and the third differential threshold is selected. The embodiment of the present disclosure does not limit the selection of the differential threshold, as long as the desired screening result can be achieved. For example, multiple differential thresholds calculated based on any two frames of images in multiple frames can be first stored, and then the median or a value that meets other set conditions can be selected from the multiple differential thresholds. In this embodiment, the maximum value among the first differential threshold, the second differential threshold and the third differential threshold is selected as max_diff_sum_current1 or max_diff_sum_current2, which can make the subsequent detection of the light color lighting test or the light color extinguishing test using the lighting threshold or the extinguishing threshold more reliable.

[0104] Next, after the initialization process is completed, at step S602, a light color lighting test is performed. Next, at step S603, firstly, it is determined whether a lamp head is lit according to the result of the light color lighting test. If the judgment result is yes, then at step S604, it is continued to determine whether the lighted light is a yellow light. If the judgment result is still yes, it means that the yellow light is lit, so at step S605, the value of the flag used to indicate the state of the yellow light is set to 1. Next, at step S606, a secondary check is performed on the current frame (i.e., the second frame of the two frames of images captured continuously) to confirm again whether the state of the yellow light is lit. At step S607, it is determined whether the secondary check passes according to the result of the secondary check. If it passes, it means that the yellow light in the current frame (i.e., the second frame) is in the state of lighting, and then at step S608, the current frame is saved. If the secondary check fails, that is, the judgment result at step S607 is no, the current frame is not saved. In order to further improve the reliability of the light color test, regardless of whether the current frame is saved, the process proceeds to step S609 to perform a light color off test. Next, at step S610, it is determined whether the yellow light is off according to the result of the light color off test. If the result of the determination is yes, it indicates that the test result is unreliable, and one or both of the light color on test and the light color off test may have a false detection. In step S611, the value of the flag bit Flag is set to 0, and the next frame is processed at step S612. If the result of the determination at step S610 is no, it can be considered that the test result is reliable, and there is no need to set the flag bit Flag. The process can proceed directly to step S612 and the next frame is processed. In addition, if the result of the determination at step S604 is no, it means that the red light or the green light is on, so the flag bit is set to 0 at step S613, and the process continues to step S609 to perform a light color off test. On the other hand, if the result of the judgment at step S603 is no, it means that all lamp heads maintain their original status, so the value of the flag used to indicate the yellow light status is further judged at step S614. If the value of Flag is 1, it means that the yellow light remains lit, and then the process proceeds to step S606 to continue to perform the secondary verification process on the current frame. If the value of Flag is 0, it means that the yellow light remains off, and then the process proceeds to step S609 to perform the light color off test.

[0105] According to the embodiments of the present disclosure, while detecting the yellow light state based on the light color on test, multiple means such as secondary verification and light color off test are used to avoid the increase in calculation amount due to false detection, thereby achieving fast and accurate automatic extraction of image data.

[0106] Figure 7 The block diagram of the image data mining device 700 according to the embodiment of the present disclosure is schematically shown. Figure 7As shown, the image data mining device 700 includes an image acquisition module 710 , a region determination module 720 , a differential value acquisition module 730 and a judgment and storage module 740 .

[0107] According to an embodiment, the image acquisition module 710 is configured to acquire an image containing a traffic light. The region determination module 720 is configured to determine in the image a first region including a lamp head of a first color, a second region including a lamp head of a second color, and a target region including a lamp head of a target color. The differential value acquisition module 730 is configured to respectively acquire a first differential value representing the difference between the target region and the first region, a second differential value representing the difference between the target region and the second region, and a third differential value representing the difference between the first region and the second region. The judgment and storage module 740 is configured to determine whether the image is a target image based on the first differential value, the second differential value, and the third differential value, and if the image is determined to be a target image, store the target image as the result image data of image data mining.

[0108] The specific operations of the above functional modules can be obtained by referring to the operation steps of the image data mining method 200 in the aforementioned embodiment, which will not be repeated here.

[0109] Figure 8 The block diagram of an image data mining device according to another embodiment of the present disclosure is schematically shown. Figure 8 As shown, the image data mining device 800 includes an image acquisition module 810, a first lighting judgment module 820, a first state setting module 830, an extinguishing judgment module 840, a second state setting module 850, a second lighting judgment module 860 and a result data storage module 870.

[0110] According to an embodiment, the image acquisition module 810 is configured to acquire a first frame image and a second frame image containing a traffic light that are captured continuously. The first lighting judgment module 820 is configured to determine whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first differential direction. The first state setting module 830 is configured to set the state of the lamp head of the target color to lighting when it is determined that the lamp head of the target color is lit. The extinguishing judgment module 840 is configured to determine whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second differential direction. The second state setting module 850 is configured to set the state of the lamp head of the target color to extinguished when it is determined that the lamp head of the target color is extinguished. The second lighting judgment module 860 is configured to determine whether the second frame image is the target image according to the inter-region difference between the first region including the lamp head of the first color, the second region including the lamp head of the second color, and the first target region including the lamp head of the target color in the second frame image when the state of the lamp head of the target color is lit. The result data storage module 870 is configured to store the target image as result image data of the image data mining when the second frame image is determined to be the target image.

[0111] The specific operations of the above functional modules can be obtained by referring to the operation steps of the image data mining method 400 in the aforementioned embodiment, which will not be repeated here.

[0112] Fig. 9 The block diagram of an electronic device 900 suitable for performing image data mining according to an embodiment of the present disclosure is schematically shown. Fig. 9 The electronic device shown is used to execute the image data mining method according to an embodiment of the present disclosure.

[0113] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901 and a memory 902. The processor 901 can perform various appropriate actions and processes according to the program or instructions stored in the memory 902. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0114] The processor 901 and the memory 902 are connected to each other via a bus. The processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in the memory 902. It should be noted that the program may also be stored in one or more storage devices other than the memory 902. The processor 901 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more storage devices.

[0115] According to an embodiment of the present disclosure, the electronic device 900 may further include an input device 903 and an output device 904, which are also connected to the bus. In addition, the electronic device 900 may further include one or more of the following components: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN card, a modem, etc.

[0116] The executor of the image data mining method according to the embodiment of the present disclosure may be a separate image data mining device or a roadside device. A separate image data mining device may acquire images offline from a storage device, or may acquire images online from a shooting device. Roadside devices may include, for example, roadside perception devices with computing functions, roadside computing devices connected to roadside perception devices, server devices connected to roadside computing devices, or server devices directly connected to roadside perception devices. In the present application, server devices may include, for example, cloud control platforms, vehicle-road collaborative management platforms, central subsystems, edge computing platforms, cloud computing platforms, and the like.

[0117] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0118] The present disclosure also provides a computer-readable storage medium and a computer program product. The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the processor 901, the method according to the embodiment of the present disclosure is implemented. The computer program product includes a computer program, and the computer program can implement the method of any of the above embodiments when executed by the processor.

[0119] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0120] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0122] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image data mining method, comprising: Acquire a first frame image and a second frame image that are continuously captured and include a traffic light, wherein the traffic light includes a plurality of lamp heads having a first color, a second color, and a target color respectively; Determine whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction; When it is determined that the lamp cap of the target color is lit, setting the state of the lamp cap of the target color to be lit; determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second difference direction; When it is determined that the lamp cap of the target color is extinguished, setting the state of the lamp cap of the target color to extinguished; When the lamp head of the target color is on, determining whether the second frame image is a target image according to the inter-region difference between a first region including the lamp head of the first color, a second region including the lamp head of the second color, and a first target region including the lamp head of the target color in the second frame image; as well as In the case of determining that the second frame image is a target image, storing the target image as result image data of image data mining; Wherein, determining whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction includes: Determine a first target area and a second target area including a lamp head of the target color in the second frame image and the first frame image respectively, wherein the first target area and the second target area have the same shape and size as each other; Obtaining a first differential target area according to a difference between the grayscale values ​​of each pixel in the first target area and each pixel in the second target area, and performing binarization and normalization processing on the first differential target area to obtain a target lighting differential value; and The target lighting difference value is compared with a first lighting threshold, and when the target lighting difference value is greater than the first lighting threshold, it is determined that the lamp head of the target color is lit.

2. The method according to claim 1, further comprising: When it is determined that the lamp head of the first color or the second color is lit, the state of the lamp head of the target color is set to be off.

3. The method according to claim 1, further comprising: When it is determined that no lamp head is lit, the state of the lamp head of the target color is queried.

4. The method according to claim 1, wherein: Determining whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction also includes: Determine in the second frame image and the first frame image respectively a first area and a third area including the lamp head of the first color and a second area and a fourth area including the lamp head of the second color, wherein the first area, the second area, the third area and the fourth area have the same shape and size as the first target area and the second target area; respectively obtaining a differential region between the first region and the third region and between the second region and the fourth region according to the difference between the grayscale values ​​of each pixel in the two regions, and performing binarization and normalization processing on the obtained differential region to obtain a first lighting differential value and a second lighting differential value; and The first lighting differential value and the second lighting differential value are compared with the second lighting threshold respectively, and when the target lighting differential value is greater than the first lighting threshold and the first lighting differential value and the second lighting differential value are both less than the second lighting threshold, it is determined that the lamp head of the target color is lit.

5. The method according to claim 1, wherein: Determining whether the second frame image is a target image according to the inter-region difference between a first region including the lamp head of the first color, a second region including the lamp head of the second color, and a first target region including the lamp head of the target color in the second frame image comprises: respectively obtaining differential regions between the first target region and the first region, between the first target region and the second region, and between the first region and the second region according to the difference in the grayscale value of each pixel in the two regions, and performing binarization and normalization processing on the obtained differential regions to obtain a first differential value, a second differential value, and a third differential value; and The absolute value of the first differential value, the absolute value of the second differential value, and the absolute value of the third differential value are compared with a first threshold and a second threshold, respectively, and when the absolute value of the first differential value and the absolute value of the second differential value are both greater than the first threshold and the absolute value of the third differential value is less than the second threshold, the image is determined to be a target image.

6. The method according to claim 1, wherein: Determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second difference direction includes: Determine a first target area and a second target area including a lamp head of the target color in the second frame image and the first frame image respectively, wherein the first target area and the second target area have the same shape and size as each other; Obtaining a second differential target area according to the difference between the grayscale values ​​of each pixel in the second target area and each pixel in the first target area, and performing binarization and normalization processing on the second differential target area to obtain a target extinguishing differential value; and The target extinguishing difference value is compared with a first extinguishing threshold value, and when the target extinguishing difference value is greater than the first extinguishing threshold value, it is determined that the lamp head of the target color is extinguished.

7. The method according to claim 6, wherein: Determining whether the lamp head of the target color is extinguished according to the inter-frame difference between the first frame image and the second frame image along the second difference direction further includes: Determine in the second frame image and the first frame image respectively a first area and a third area including the lamp head of the first color and a second area and a fourth area including the lamp head of the second color, wherein the first area, the second area, the third area and the fourth area have the same shape and size as the first target area and the second target area; respectively obtaining differential regions between the third region and the first region and between the fourth region and the second region according to the difference between the grayscale values ​​of the pixels in the two regions, and performing binarization and normalization processing on the obtained differential regions to obtain a first extinguishing differential value and a second extinguishing differential value; and The first extinction differential value and the second extinction differential value are compared with the second extinction threshold value respectively, and when the target extinction differential value is greater than the first extinction threshold value and the first extinction differential value and the second extinction differential value are both less than the second extinction threshold value, it is determined that the lamp head of the target color is extinguished.

8. The method according to claim 1, before determining whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along the first difference direction, further comprising: Acquire a plurality of frames of images continuously captured containing a traffic light, wherein the traffic light comprises a plurality of lamp heads respectively having a first color, a second color, and a target color; Between any two frames of the plurality of images, a lighting threshold for determining whether the lamp heads of the first color, the second color, and the target color are lit is obtained by calculating an inter-frame difference between the two frames of images along a first difference direction; and Between any two frames of the multiple images, an extinguishing threshold for determining whether the lamp heads of the first color, the second color and the target color are extinguished is obtained by calculating an inter-frame difference between the two frames of images along a second differential direction.

9. An image data mining device, comprising: An image acquisition module is configured to acquire a first frame image and a second frame image continuously captured containing a traffic light, wherein the traffic light comprises a plurality of lamp heads having a first color, a second color and a target color respectively; A first lighting judgment module is configured to determine whether the lamp head of the target color is lit according to the inter-frame difference between the first frame image and the second frame image along a first difference direction; A first state setting module is configured to set the state of the lamp head of the target color to be lit when it is determined that the lamp head of the target color is lit; an extinguishing judgment module, configured to determine whether the lamp head of the target color is extinguished according to an inter-frame difference between the first frame image and the second frame image along a second difference direction; A second state setting module is configured to set the state of the lamp head of the target color to be extinguished when it is determined that the lamp head of the target color is extinguished; A second lighting judgment module is configured to determine whether the second frame image is a target image according to the difference between the first area including the lamp head of the first color, the second area including the lamp head of the second color, and the first target area including the lamp head of the target color in the second frame image when the state of the lamp head of the target color is lit; as well as A result data storage module is configured to store the target image as result image data of image data mining when the second frame image is determined to be a target image. Wherein, the first lighting judgment module is further configured as follows: Determine a first target area and a second target area including a lamp head of the target color in the second frame image and the first frame image respectively, wherein the first target area and the second target area have the same shape and size as each other; Obtaining a first differential target area according to a difference between the grayscale values ​​of each pixel in the first target area and each pixel in the second target area, and performing binarization and normalization processing on the first differential target area to obtain a target lighting differential value; as well as The target lighting difference value is compared with a first lighting threshold, and when the target lighting difference value is greater than the first lighting threshold, it is determined that the lamp head of the target color is lit.

10. An image data mining device, comprising: A memory storing program instructions; as well as A processor is configured to execute the program instructions to perform the image data mining method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the image data mining method according to any one of claims 1 to 8 when executed by a processor.

12. A roadside device, comprising: A memory storing program instructions; as well as A processor is configured to execute the program instructions to perform the image data mining method according to any one of claims 1 to 8.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

14. An edge computing platform, comprising: A memory storing program instructions; as well as A processor is configured to execute the program instructions to perform the image data mining method according to any one of claims 1 to 8.

Images