Image fusion method and device, electronic equipment and storage medium

Abstract

The invention provides an image fusion method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a bright frame image and a dark frame image from a monitoring video stream, and enabling the exposure intensity of the bright frame image to be greater than the exposure intensity of the dark frame image; determining whether the first lamp eye area feature of the bright frame image is matched with the second lamp eye area feature of the dark frame image; under the condition that the first lamp eye area feature and the second lamp eye area feature are not matched, filling a lamp eye area in the target image by using a non-lighted single-eye signal lamp image to obtain a processed target bright frame image and a processed target dark frame image; the target image is a bright frame image or a dark frame image; and fusing the target bright frame image and the target dark frame image to obtain a signal lamp enhanced image. According to the technical scheme provided by the invention, the problem of wrong fusion caused by time domain difference when a signal lamp region is enhanced by adopting an image fusion mode of differential exposure parameters can be solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image fusion method, device, electronic equipment and storage medium. Background Art

[0002] In the field of road traffic monitoring, image fusion technology can be used to enhance the traffic light area in the road monitoring image to restore the traffic light, so as to solve the problem of overexposure of the traffic light area caused by increasing the exposure parameters due to weak fill light conditions.

[0003] At present, in the process of signal light image fusion, it is necessary to obtain two frames of images with different exposure parameters from the video images collected by the monitoring equipment, and then directly fuse the signal light area of ​​the two frames of images, and refine the outline of the signal light by differentiating the exposure parameters. This method requires obtaining two frames of images with different exposure parameters, and inevitably there will be differences in the time domain, resulting in misfusion of the signal light area, making the fused signal light area abnormal, which is not conducive to the collection of indicators such as the traffic evidence chain. Summary of the invention

[0004] The present invention provides an image fusion method, an apparatus, an electronic device and a storage medium, which are used to solve the problem of mis-fusion caused by time domain differences when a signal light area is enhanced by using an image fusion method with differentiated exposure parameters in the prior art.

[0005] The present invention provides an image fusion method, comprising:

[0006] Acquire a bright frame image and a dark frame image from a surveillance video stream, wherein the exposure intensity of the bright frame image is greater than the exposure intensity of the dark frame image;

[0007] Determine whether a first lamp eye region feature of the bright frame image matches a second lamp eye region feature of the dark frame image;

[0008] In the case where the first eye region feature and the second eye region feature do not match, the eye region in the target image is filled with the target non-eye region image to obtain a processed target bright frame image and a target dark frame image; wherein the target non-eye region image is an image of a single-eye signal light without lighting, and the target image is the bright frame image or the dark frame image;

[0009] The target bright frame image and the target dark frame image are fused to obtain a signal light enhanced image.

[0010] According to an image fusion method provided by the present invention, the method of filling the lamp eye area in the target image with the target non-lamp eye area image comprises:

[0011] The lamp group area of ​​the lamp group where the lamp eye area in the target image is located is equally divided according to the number of lamp eyes in the lamp group area to obtain the lamp eye area and at least one non-lamp eye area;

[0012] Determine a non-lamp eye area in the at least one non-lamp eye area as the target non-lamp eye area image;

[0013] The target non-lamp eye area image is used to fill the lamp eye area in the target image.

[0014] According to an image fusion method provided by the present invention, determining whether the first light eye region feature of the bright frame image matches the second light eye region feature of the dark frame image includes:

[0015] Determining a first signal light region in the bright frame image and a second signal light region in the dark frame image;

[0016] generating a first highlight area mask of the first signal light area and a second highlight area mask of the second signal light area;

[0017] Determine whether the first eye region feature and the second eye region feature match based on the first highlight region mask and the second highlight region mask; wherein the first highlight region mask represents the first eye region feature, and the second highlight region mask represents the second eye region feature.

[0018] According to an image fusion method provided by the present invention, determining whether the first lamp eye region feature and the second lamp eye region feature match based on the first highlight region mask and the second highlight region mask includes:

[0019] Determine a first highlight pixel number of the first highlight area mask and a second highlight pixel number of the second highlight area mask;

[0020] When the number of the first highlight pixels is equal to the number of the second highlight pixels, determining that the first eye region feature matches the second eye region feature;

[0021] When the number of the first highlight pixels is less than the number of the second highlight pixels, determining that the first eye region feature and the second eye region feature do not match;

[0022] Among them, the first light eye area feature is the number of the first highlight pixels, and the second light eye area feature is the number of the second highlight pixels.

[0023] According to an image fusion method provided by the present invention, the determining of the first number of highlight pixels of the first highlight area mask and the second number of highlight pixels of the second highlight area mask includes:

[0024] Determine the number of highlighted pixels in the second highlighted area mask, obtain the number of the second highlighted pixels, and record the position coordinates of each of the highlighted pixels;

[0025] The pixel points at the position coordinates are traversed in the first highlight area mask, and the number of highlight pixel points among the traversed pixel points is determined to obtain the first number of highlight pixel points.

[0026] According to an image fusion method provided by the present invention, generating a first highlight area mask of the first signal light area and a second highlight area mask of the second signal light area includes:

[0027] Taking the first signal light area and the second signal light area as target signal light areas respectively, and determining the maximum color channel value of each pixel in the target signal light area;

[0028] When the maximum color channel value is greater than or equal to a preset brightness threshold, the brightness value of the pixel is assigned to the highest order brightness value, or when the maximum color channel value is less than the preset brightness threshold, the brightness value of the pixel is assigned to the lowest order brightness value;

[0029] A highlight area mask corresponding to the target signal light area is generated based on the re-assigned pixel points.

[0030] An image fusion method provided by the present invention also includes:

[0031] When the first eye-of-the-light region feature matches the second eye-of-the-light region feature, the bright frame image and the dark frame image are fused to obtain the signal light enhanced image.

[0032] The present invention also provides an image fusion device, comprising:

[0033] An image acquisition module, used to acquire a bright frame image and a dark frame image from a surveillance video stream, wherein the exposure intensity of the bright frame image is greater than the exposure intensity of the dark frame image;

[0034] A feature matching module, used to determine whether a feature of a first lamp eye region of the bright frame image matches a feature of a second lamp eye region of the dark frame image;

[0035] An image filling module, used for filling the eye area in the target image with the target non-eye area image when the first eye area feature and the second eye area feature do not match, to obtain a processed target bright frame image and a target dark frame image; wherein the target non-eye area image is an image of a single-eye signal light without lighting, and the target image is the bright frame image or the dark frame image;

[0036] The image fusion module is used to fuse the target bright frame image and the target dark frame image to obtain a signal light enhanced image.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned image fusion methods is implemented.

[0038] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image fusion method described above is implemented.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image fusion method described above is implemented.

[0040] The image fusion method, device, electronic device and storage medium provided by the present invention first obtain a bright frame image and a dark frame image from a monitoring video stream to obtain two frames of differentially exposed images before and after, and then compare the features of the eye area of ​​the bright frame image and the dark frame image. If the features of the eye area of ​​the two frames do not match, indicating that there is a difference in the light position or light number of the signal light between the bright frame image and the dark frame image of differential exposure, then the eye area of ​​any one of the bright frame image and the dark frame image is filled with a single-eye signal light image that is not lit to obtain a processed target bright frame image and target dark frame image, and then the target bright frame image and the target dark frame image are fused to obtain a signal light enhanced image. In this way, when there is a difference in the eye area of ​​the two frames of images with different exposure parameters before and after due to the time domain difference, only the light information of the eye area of ​​one of the frames is retained during image fusion, thereby avoiding erroneous fusion caused by the time domain difference. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flowchart of an image fusion method provided by an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of the principle of the image fusion method provided by an embodiment of the present invention;

[0044] Figure 3 is a structural schematic diagram of an image fusion device provided by an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that the serial numbers assigned to the objects described in the present invention, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning.

[0048] In the process of using two frames of images with different exposure parameters to enhance the traffic lights in the image by image fusion, due to the inevitable difference in the time domain between the two frames, the positions of the traffic lights in the two frames may not be in the same position, the numbers of lights may be different, or the patterns of lights may be different. For example, when the red and green lights in a light group are switched, one frame may have a red light and the other may have a green light, or the positions of the lights in the two frames may be the same but the numbers displayed may be different. In this case, if the two frames are directly fused, the different information of lights in the two frames may cause the traffic light area to be misfused, abnormally bright areas, etc., which may cause abnormalities in the enhanced traffic light area, i.e., abnormal frames may appear after fusion, which is not conducive to the collection of indicators such as the traffic evidence chain.

[0049] Based on this, an embodiment of the present invention provides an image fusion method, which obtains two frames of images with different exposure parameters from a surveillance video stream to obtain a bright frame image and a dark frame image, and then compares the features of the light eye area of ​​the two frames of images. If they do not match, it means that the light positions, light numbers or light patterns of the signal lights in the two frames of images are different. At this time, the light eye area in the bright frame image is filled with a single-eye signal light image that is not lit, and the dark frame image is used as the target dark frame image to be fused with the target bright frame image obtained after the light frame image is filled; or, the light eye area in the dark frame image is filled with a single-eye signal light image that is not lit, and the bright frame image is used as the target bright frame image to be fused with the target dark frame image obtained after the dark frame image is filled. In this way, the problem of misfusion caused by time domain differences is eliminated.

[0050] Combine the following Figure 1-Figure 2 The image fusion method of the present invention is described. The image fusion method can be applied to electronic devices such as terminal devices or servers. Among them, the terminal device may include a monitoring device or a mobile phone, computer, vehicle-mounted device, tablet computer, etc. that is communicatively connected to the monitoring device; the server may include an independent server, a cluster server or a cloud server, etc., which can be communicatively connected to the monitoring device. The image fusion method can also be applied to an image fusion device set in an electronic device such as a terminal device or a server, and the image fusion device can be implemented by software, hardware, or a combination of both.

[0051] Figure 1 The schematic diagram of the process of the image fusion method provided by the embodiment of the present invention is exemplified. Figure 1 As shown, the image fusion method may include the following steps 110 to 140.

[0052] Step 110: Acquire a bright frame image and a dark frame image from the surveillance video stream.

[0053] The exposure intensity of the bright frame image is greater than the exposure intensity of the dark frame image.

[0054] Specifically, taking traffic monitoring at an intersection as an example, the camera of a monitoring device (such as a camera) can collect video images of the intersection in real time to obtain a monitoring video stream. The monitoring device is a differentiated image acquisition device that can collect video images under different exposure parameters. For example, dual-channel exposure parameters can be set, one channel sets a first exposure parameter to image a bright frame image; the other channel sets a second exposure parameter to image a dark frame image. The first exposure parameter is greater than the second exposure parameter. Exemplarily, the exposure parameters may include at least one of shutter and gain.

[0055] The electronic device can obtain the bright frame image and the dark frame image from the monitoring video stream to obtain the video images with different exposure parameters of the previous and next frames, and enhance the image of the traffic light area in the image through subsequent image fusion to clearly restore the state of the traffic light.

[0056] Step 120: Determine whether the first lamp eye region feature of the bright frame image matches the second lamp eye region feature of the dark frame image.

[0057] After the electronic device obtains the bright frame image and the dark frame image with differential exposure parameters, the area images where the signal lights are located in the two frames of images can be extracted respectively to obtain the first signal light area in the bright frame image and the second signal light area in the dark frame image. The first signal light area and the second signal light area are used as regions of interest (ROI) for subsequent image processing.

[0058] Next, the eye region feature of the first signal light region is extracted to obtain the first eye region feature; the eye region feature of the second signal light region is extracted to obtain the second eye region feature. The eye region feature may include at least one of a brightness feature, a color feature, and a binary brightness gradient.

[0059] After obtaining the first lamp eye region feature and the second lamp eye region feature, the electronic device may compare the first lamp eye region feature and the second lamp eye region feature to determine whether the two match.

[0060] Step 130: When the first eye region feature and the second eye region feature do not match, the eye region in the target image is filled with the target non-eye region image to obtain a processed target bright frame image and a target dark frame image.

[0061] The target non-light eye area image is an image of a single-eye signal light without the light on, and the target image is a bright frame image or a dark frame image.

[0062] If the first eye region feature and the second eye region feature do not match, it means that the light positions, light numbers, or light patterns of the traffic lights between the bright frame image and the dark frame image are different. If the two frames are directly fused, incorrect fusion will occur. For example, the traffic light in the bright frame image displays the number 4, while the traffic light in the dark frame image displays the number 5. If the bright frame image and the dark frame image are directly fused, the number displayed by the traffic light in the fused image will be abnormal, such as becoming 9, resulting in the appearance of an abnormal traffic light frame. In view of this, in an embodiment of the present invention, the eye region in the bright frame image or the dark frame image can be filled with a single-eye traffic light image that is not lit, and then the two frames are fused to repair the abnormal traffic light frame.

[0063] Specifically, the traffic light may be a light group including at least two light eyes, or may be a light group including only one light eye (ie, the red, green and yellow lights share one light eye).

[0064] For the case where the traffic light is a group of lights, in an exemplary embodiment, a frame of video image without a light on can be pre-collected as a reference frame, from which a single-eye traffic light image without the light on is extracted to obtain a target non-eye area image. When it is determined that the first eye area feature and the second eye area feature do not match, the target non-eye area image can be used to fill the eye area in the bright frame image or the dark frame image.

[0065] For the case where the traffic light is a group of lights, in another embodiment, the traffic light area in the target image can be divided into equal parts according to the number of light eyes, and any unlit area divided into equal parts is determined as a target non-light eye area image, and the light eye area in the target image is filled with the target non-light eye area image.

[0066] For the case where a traffic light has only one eye, a frame of video image without the light on can be pre-captured as a reference frame, from which an image of the eye area can be extracted to obtain a single-eye traffic light image without the light on. The single-eye traffic light image without the light on can be used to fill the eye area in the bright frame image or the dark frame image.

[0067] Exemplarily, when the first eye-light area feature and the second eye-light area feature match, it means that the lighting conditions between the acquired bright frame image and the dark frame image are the same. At this time, the bright frame image and the dark frame image can be directly fused to enhance the signal light in the image and obtain a signal light enhanced image.

[0068] Step 140: Fusing the target bright frame image and the target dark frame image to obtain a signal light enhanced image.

[0069] The electronic device may use the target non-lamp eye area image to fill the lamp eye area in the bright frame image to obtain the target bright frame image, and then use the acquired dark frame image as the target dark frame image, fuse the target dark frame image and the target bright frame image to obtain the signal light enhanced image. Alternatively, the target non-lamp eye area image may be used to fill the lamp eye area in the dark frame image to obtain the target dark frame image, and then use the acquired bright frame image as the target bright frame image, fuse the target dark frame image and the target bright frame image to obtain the signal light enhanced image.

[0070] It can be understood that the obtained signal light enhanced image is a fused image obtained by enhancing the signal light in the video image through fusion, and the clarity of the signal light is improved through fusion.

[0071] The image fusion method provided by the embodiment of the present invention first obtains a bright frame image and a dark frame image from a monitoring video stream to obtain two frames of differentially exposed images before and after, and then compares the features of the eye area of ​​the bright frame image and the dark frame image. If the features of the eye area of ​​the two do not match, it means that there is a difference in the light position or light number of the signal light between the bright frame image and the dark frame image of differential exposure, and then the eye area of ​​any one of the bright frame image and the dark frame image is filled with a single-eye signal light image that is not lit to obtain a processed target bright frame image and target dark frame image, and then the target bright frame image and the target dark frame image are fused to obtain a signal light enhanced image. In this way, when there is a difference in the eye area of ​​the two frames of images before and after with different exposure parameters due to time domain differences, only the light information of the eye area of ​​one of the frames is retained during image fusion, thereby avoiding misfusion caused by time domain differences.

[0072] based on Figure 1 In the image fusion method of the corresponding embodiment, in an exemplary embodiment, according to the standard specifications of the traffic light, the actual size of each eye (such as the width and height of the light frame, the size of the light-emitting unit, etc.) is basically the same, and only one type of eye is lit in the same light group at the same time, and the features of other non-lighting areas in the image are basically the same. Based on this, the traffic light area in the target image can be divided into equal parts according to the number of eyes, so as to obtain at least two equally divided areas, and any non-lighting area in these equally divided areas is used as the target non-eye area image to fill the eye area in the target image to obtain the image information of the original non-lighting state. Specifically, filling the eye area in the target image with the target non-eye area image may include: dividing the light group area of ​​the light group where the eye area in the target image is located according to the number of eyes in the light group area, so as to obtain the eye area and at least one non-eye area; determining one non-eye area in the at least one non-eye area as the target non-eye area image; and filling the eye area in the target image with the target non-eye area image.

[0073] For example, Figure 2 The schematic diagram of the principle of the image fusion method provided by the embodiment of the present invention is exemplarily shown. Figure 2As shown, taking the case where a light group of a signal light is included in a bright frame image and a dark frame image, and the target image is a bright frame image as an example, after the electronic device obtains the bright frame image and the dark frame image, it can determine the first signal light area 21 in the bright frame image and the second signal light area 23 in the dark frame image. Wherein, the number of lights on in the first signal light area 21 and the second signal light area 23 is different, and the first signal light area 21 can be equally divided according to the number of lights, and equal areas A, equal areas B and equal areas C are obtained. Wherein, the equal areas C are the lights, and the lights in the equal areas A and the equal areas B are not in the state of being lit. Either the equal areas A and the equal areas B can be used as the target non-light area image to fill the equal areas C in the bright frame image. At this time, the first signal light area 21 in the bright frame image can be corrected to the signal light area 22 in the target bright frame image. Afterwards, the electronic device uses the dark frame image as the target dark frame image and fuses it with the target bright frame image obtained after the filling process to obtain the signal light enhanced image. At this time, the signal light areas in the bright frame image and the dark frame image are fused into a signal light area 24 in the signal light enhanced image.

[0074] In this way, the restoration effect based on the dark frame eye area can be guaranteed, avoiding the problem of false fusion caused by time domain differences, and avoiding the problem of overexposure of the fused signal light due to the bright frame being too bright. Moreover, by equal filling, the eye area can be filled with other non-eye areas in the same frame image, so that the filled area can maintain the same or similar image characteristics as other areas, ensuring the overall visual effect of the signal light area, and then ensuring the overall visual effect of the fused image.

[0075] Exemplarily, the filling algorithm used when performing image filling may include flood filling, but is not limited thereto. Flood filling can be used to retain the details of the original non-lit area as much as possible in a specific connected domain.

[0076] based on Figure 1 The image fusion method of the corresponding embodiment, in an exemplary embodiment, determining whether the first light eye area feature of the bright frame image and the second light eye area feature of the dark frame image match may include: determining the first signal light area in the bright frame image and the second signal light area in the dark frame image; generating a first highlight area mask of the first signal light area and a second highlight area mask of the second signal light area; determining whether the first light eye area feature and the second light eye area feature match based on the first highlight area mask and the second highlight area mask.

[0077] Among them, the first highlight area mask can characterize the highlight pixels in the first signal light area, and the area formed by these highlight pixels is the eye area in the first signal light area. Based on this, the first highlight area mask can be used to characterize the first eye area characteristics of the bright frame image; the second highlight area mask can characterize the highlight pixels in the second signal light area, and the area formed by these highlight pixels is the eye area in the second signal light area. Based on this, the second highlight area mask can be used to characterize the second eye area characteristics of the dark frame image.

[0078] based on Figure 1 The image fusion method of the corresponding embodiment, in an exemplary embodiment, determining whether the first light eye area feature of the bright frame image and the second light eye area feature of the dark frame image match may include: determining the first signal light area in the bright frame image and the second signal light area in the dark frame image; determining the first color of the first light eye area in the first signal light area and the second color of the second light eye area in the second signal light area; when the first color and the second color are different, determining that the first light eye area feature of the bright frame image and the second light eye area feature of the dark frame image do not match; when the first color and the second color are the same, generating a first highlight area mask for the first signal light area and a second highlight area mask for the second signal light area; determining whether the first light eye area feature and the second light eye area feature match based on the first highlight area mask and the second highlight area mask.

[0079] Exemplarily, generating a first highlight area mask for a first signal light area and a second highlight area mask for a second signal light area may include: taking the first signal light area and the second signal light area as target signal light areas respectively, and determining the maximum color channel value of the pixel for each pixel in the target signal light area; assigning the brightness value of the pixel to the highest order brightness value when the maximum color channel value is greater than or equal to a preset brightness threshold, or assigning the brightness value of the pixel to the lowest order brightness value when the maximum color channel value is less than the preset brightness threshold; and generating a highlight area mask corresponding to the target signal light area based on the re-assigned pixels. Among them, the pixel assigned the highest order brightness value is the highlight pixel, and the pixel assigned the lowest order brightness value is the non-highlight pixel.

[0080] Specifically, for an image, it may have multiple color channels, and the number of color channels depends on its color mode. For a traffic light, its color mode may be an RGB mode, and each pixel in the image may include a red channel (R), a green channel (G), and a blue channel (B).

[0081] Taking the RGB color channel as an example, for the first signal light area, each pixel in the first signal light area can be traversed to determine the maximum value of the RGB channel of each pixel to obtain the maximum color channel value of the pixel. Specifically, the maximum color channel value Max of each pixel in the first signal light area brigh It can be determined according to the following expression (1):

[0082] Max bright =Max(R bright , G bright , B brigh ) (1)

[0083] Among them, R bright , G brigh and B bright Indicates the channel values ​​of the three RGB channels of the pixel points in the first signal light area.

[0084] Determine the maximum color channel value Max of the pixel in the first signal light area bright After that, the Max brigh Compared with the preset brightness threshold thresh, if it is greater than or equal to thresh, the pixel point is determined to be the eye area of ​​the first signal light area, that is, the highlight area, and its brightness value can be assigned to the highest order brightness value 255, and the pixel point is determined to be a highlight pixel point; if Max bright If the value is less than thresh, the pixel is determined to be a non-eye area in the first signal light area, that is, a non-highlight area, and its brightness value can be assigned the lowest-order brightness value of 0. The pixel is determined to be a non-highlight pixel. Specifically, it can be expressed as the following expression (2):

[0085]

[0086] Among them, Mask brigh Represents the brightness value of the pixel points in the first signal light area after re-assignment. Based on this, the brightness values ​​of all the pixel points in the first signal light area are re-assigned to obtain the first highlight area mask of the first signal light area.

[0087] Similarly, the maximum color channel value Max of each pixel in the second signal light area dark It can be determined according to the following expression (3):

[0088] Max dark =Max(R dark , G dark , B dark ) (3)

[0089] Among them, R dark , Gdark and B dark Indicates the channel values ​​of the three RGB channels of the pixel points in the first signal light area.

[0090] Determine the maximum color channel value Max of the pixel point in the second signal light area dark After that, the Max dark Compared with the preset brightness threshold thresh, if it is greater than or equal to thresh, the pixel point is determined to be the eye area of ​​the second signal light area, that is, the highlight area, and its brightness value can be assigned to the highest order brightness value 255, and the pixel point is determined to be a highlight pixel point; if Max dark If the value is less than thresh, the pixel is determined to be a non-eye area in the second signal light area, that is, a non-highlight area, and its brightness value can be assigned the lowest-order brightness value of 0. The pixel is determined to be a non-highlight pixel. Specifically, it can be expressed as the following expression (4):

[0091]

[0092] Among them, Mask dark Represents the brightness value of the pixel points in the second signal light area after reassignment. Based on this, the brightness values ​​of all the pixel points in the second signal light area are reassigned to obtain the second highlight area mask of the second signal light area.

[0093] Exemplarily, the above-mentioned preset brightness threshold thresh can be determined based on the brightness information of the signal light area of ​​the dark frame sample image in different scenes. For example, the same exposure parameters as the dark frame image can be used to obtain video images in different scenes, obtain multiple dark frame sample images, extract the brightness information of the signal light area in each dark frame sample image, and determine the preset brightness threshold thresh based on the brightness information. For example, the brightness value of each pixel in the signal light area can be determined, and the minimum value of these brightness values ​​is determined as the preset brightness threshold thresh.

[0094] Since the exposure parameter of the bright frame image is greater than that of the dark frame image, combined with the distribution characteristics of the highlight area masks in the bright frame image and the dark frame image, it can be seen that for the first highlight area mask corresponding to the bright frame image and the second highlight area mask corresponding to the dark frame image, the pixels at the same coordinate position in the image have the following characteristics:

[0095] If the first highlight area mask and the second highlight area mask match, then for the same coordinate position, if the second highlight area mask corresponding to the dark frame image is a highlight pixel at the coordinate, then the first highlight area mask corresponding to the bright frame image must also be a highlight pixel at the coordinate, and the brightness value of the pixel of the first highlight area mask at the coordinate is greater than the brightness value of the pixel of the second highlight area mask at the coordinate; and the total number of highlight pixels in the first highlight area mask is greater than the total number of highlight pixels in the second highlight area mask. Conversely, if the first highlight area mask and the second highlight area mask match, then for the pixel at the same coordinate position, if the second highlight area mask is a highlight pixel at the coordinate, then the first highlight area mask is not necessarily a highlight pixel at the coordinate, for example, when a digital signal light jumps, some highlight pixels will match and some will not match.

[0096] Based on this, in an exemplary embodiment, determining whether the first eye region feature and the second eye region feature match based on the first highlight region mask and the second highlight region mask may include: taking the second highlight region mask as a reference, traversing the position coordinates of the highlight pixel points in the second highlight region mask; determining whether the pixel point at the position coordinate in the first highlight region mask is a highlight pixel point, and recording the total number of non-highlight pixel points; when the total number is less than a preset number threshold, determining that the first eye region feature and the second eye region feature match; otherwise, determining that the first eye region feature and the second eye region feature do not match.

[0097] In another example embodiment, whether the first eye region feature and the second eye region feature match can be determined based on the number of highlight pixels in the first highlight region mask and the second highlight region mask, respectively. Specifically, determining whether the first eye region feature and the second eye region feature match based on the first highlight region mask and the second highlight region mask can include: determining the first highlight pixel number of the first highlight region mask and the second highlight pixel number of the second highlight region mask; determining that the first eye region feature and the second eye region feature match when the first highlight pixel number is equal to the second highlight pixel number; determining that the first eye region feature and the second eye region feature do not match when the first highlight pixel number is less than the second highlight pixel number; wherein the first eye region feature is the first highlight pixel number, and the second eye region feature is the second highlight pixel number.

[0098] Specifically, determining the first number of highlighted pixels in the first highlight area mask and the second number of highlighted pixels in the second highlight area mask may include: determining the number of highlighted pixels in the second highlight area mask, obtaining the second number of highlighted pixels, and recording the position coordinates of each highlighted pixel; traversing the pixel points at the position coordinates in the first highlight area mask, determining the number of highlighted pixels in the traversed pixel points, and obtaining the first number of highlighted pixels.

[0099] That is, the pixels in the second highlight area mask can be traversed, the number of highlight pixels can be counted, the second highlight pixel number DarkCount can be obtained, and the coordinates of each highlight pixel can be recorded. Then, according to the coordinates of the highlight pixels in the second highlight area mask, the pixels at these coordinates in the first highlight area mask can be traversed, and the number of highlight pixels can be counted during the traversal process to obtain the first highlight pixel number BrightCount. Afterwards, the following expression (5) can be used to determine whether the first eye area feature matches the second eye area feature:

[0100]

[0101] The image fusion method provided by the embodiment of the present invention can collect two frames of images with different exposure parameters from the monitoring video stream to obtain a bright frame image and a dark frame image, and respectively determine the first signal light area in the bright frame image and the second signal light area in the dark frame image, and generate a first highlight area mask of the first signal light area and a second highlight area mask of the second signal light area based on the second signal light area in the dark frame image; then perform a brightness comparison on the first highlight area mask and the second highlight area mask, and if they match, directly perform double-frame fusion on the collected bright frame image and dark frame image; if they do not match, fill the first signal light area in equal parts, and then perform double-frame fusion. The problem of mis-fusion caused by time domain differences when using differentiated exposure parameters for image fusion is solved, and the restoration effect based on the dark frame light eye area is ensured. At the same time, the overall solution has low complexity and occupies little computing power, which can effectively release software and hardware resources and better meet the real-time requirements of video surveillance.

[0102] The image fusion device provided by the present invention is described below. The image fusion device described below and the image fusion method described above can be referred to each other.

[0103] Figure 3 The schematic diagram of the structure of the image fusion device provided by the embodiment of the present invention is exemplarily shown. Figure 3As shown, the image fusion device may include: an image acquisition module 310, used to acquire a bright frame image and a dark frame image from a monitoring video stream, the exposure intensity of the bright frame image being greater than the exposure intensity of the dark frame image; a feature matching module 320, used to determine whether a first light eye area feature of the bright frame image and a second light eye area feature of the dark frame image match; an image filling module 330, used to fill the light eye area in the target image with a target non-light eye area image when the first light eye area feature and the second light eye area feature do not match, so as to obtain a processed target bright frame image and a target dark frame image; wherein the target non-light eye area image is an image of a single-eye signal light without the light on, and the target image is a bright frame image or a dark frame image; an image fusion module 340, used to fuse the target bright frame image and the target dark frame image to obtain a signal light enhanced image.

[0104] In an exemplary embodiment, the image filling module 330 includes: an equal division unit, used to divide the light group area of ​​the light group where the light eye area in the target image is located equally according to the number of light eyes in the light group area, to obtain light eye areas and at least one non-light eye area; a first determination unit, used to determine one non-light eye area in the at least one non-light eye area as a target non-light eye area image; and a filling unit, used to fill the light eye area in the target image with the target non-light eye area image.

[0105] In an exemplary embodiment, the feature matching module 320 includes: a second determination unit, used to determine a first signal light area in a bright frame image and a second signal light area in a dark frame image; a generation unit, used to generate a first highlight area mask for the first signal light area and a second highlight area mask for the second signal light area; a third determination unit, used to determine whether the first light eye area feature and the second light eye area feature match based on the first highlight area mask and the second highlight area mask; wherein the first highlight area mask represents the first light eye area feature, and the second highlight area mask represents the second light eye area feature.

[0106] In an exemplary embodiment, the third determination unit includes: a first determination subunit, used to determine the first highlight pixel number of the first highlight area mask and the second highlight pixel number of the second highlight area mask; a second determination subunit, used to determine that the first eye area feature and the second eye area feature match when the first highlight pixel number is equal to the second highlight pixel number; and determine that the first eye area feature and the second eye area feature do not match when the first highlight pixel number is less than the second highlight pixel number; wherein the first eye area feature is the first highlight pixel number, and the second eye area feature is the second highlight pixel number.

[0107] In an exemplary embodiment, the first determination subunit is specifically used to: determine the number of highlighted pixels in the second highlight area mask, obtain the second number of highlighted pixels, and record the position coordinates of each highlighted pixel; traverse the pixel points at the position coordinates in the first highlight area mask, determine the number of highlighted pixels in the traversed pixel points, and obtain the first number of highlighted pixels.

[0108] In an exemplary embodiment, the generation unit includes: a third determination subunit, used to take the first signal light area and the second signal light area as target signal light areas, respectively, and determine the maximum color channel value of the pixel point for each pixel point in the target signal light area; a fourth determination subunit, when the maximum color channel value is greater than or equal to a preset brightness threshold, assigns the brightness value of the pixel point to the highest order brightness value, or when the maximum color channel value is less than the preset brightness threshold, assigns the brightness value of the pixel point to the lowest order brightness value; a generation subunit, used to generate a highlight area mask corresponding to the target signal light area based on each reassigned pixel point.

[0109] In an exemplary embodiment, the image fusion module 340 is further configured to fuse the bright frame image and the dark frame image to obtain a signal light enhanced image when the first light eye region feature matches the second light eye region feature.

[0110] Figure 4 An example of a structural diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the image fusion method provided by any of the above method embodiments.

[0111] Exemplarily, the processor 410 may include a central processing unit (CPU), a microprocessor, a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0112] Exemplarily, the communication bus 440 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc.

[0113] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image fusion method provided by any of the above method embodiments.

[0115] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image fusion method provided by any of the above method embodiments is implemented.

[0116] By way of example, computer readable storage media include non-transitory computer readable storage media.

[0117] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image fusion method, characterized in that: include: Acquire a bright frame image and a dark frame image from a surveillance video stream, wherein the exposure intensity of the bright frame image is greater than the exposure intensity of the dark frame image; Determine whether a first lamp eye region feature of the bright frame image matches a second lamp eye region feature of the dark frame image; In the case where the first eye region feature and the second eye region feature do not match, the eye region in the target image is filled with the target non-eye region image to obtain a processed target bright frame image and a target dark frame image; wherein the target non-eye region image is an image of a single-eye signal light without lighting, and the target image is the bright frame image or the dark frame image; The target bright frame image and the target dark frame image are fused to obtain a signal light enhanced image.

2. The image fusion method according to claim 1, characterized in that: The method of filling the lamp eye area in the target image with the target non-lamp eye area image comprises: The lamp group area of ​​the lamp group where the lamp eye area in the target image is located is equally divided according to the number of lamp eyes in the lamp group area to obtain the lamp eye area and at least one non-lamp eye area; Determine a non-lamp eye area in the at least one non-lamp eye area as the target non-lamp eye area image; The target non-lamp eye area image is used to fill the lamp eye area in the target image.

3. The image fusion method according to claim 1, characterized in that: The determining whether the first light eye region feature of the bright frame image matches the second light eye region feature of the dark frame image includes: Determining a first signal light region in the bright frame image and a second signal light region in the dark frame image; generating a first highlight area mask of the first signal light area and a second highlight area mask of the second signal light area; Determine whether the first eye region feature and the second eye region feature match based on the first highlight region mask and the second highlight region mask; wherein the first highlight region mask represents the first eye region feature, and the second highlight region mask represents the second eye region feature.

4. The image fusion method according to claim 3, characterized in that: The determining whether the first lamp eye region feature and the second lamp eye region feature match based on the first highlight region mask and the second highlight region mask comprises: Determine a first highlight pixel number of the first highlight area mask and a second highlight pixel number of the second highlight area mask; When the number of the first highlight pixels is equal to the number of the second highlight pixels, determining that the first eye region feature matches the second eye region feature; When the number of the first highlight pixels is less than the number of the second highlight pixels, determining that the first eye region feature and the second eye region feature do not match; Among them, the first light eye area feature is the number of the first highlight pixels, and the second light eye area feature is the number of the second highlight pixels.

5. The image fusion method according to claim 4, characterized in that: The determining of the first highlight pixel number of the first highlight area mask and the second highlight pixel number of the second highlight area mask includes: Determine the number of highlighted pixels in the second highlighted area mask, obtain the number of the second highlighted pixels, and record the position coordinates of each of the highlighted pixels; The pixel points at the position coordinates are traversed in the first highlight area mask, and the number of highlight pixel points among the traversed pixel points is determined to obtain the first number of highlight pixel points.

6. The image fusion method according to claim 3, characterized in that: The generating a first highlight area mask of the first signal light area and a second highlight area mask of the second signal light area includes: Taking the first signal light area and the second signal light area as target signal light areas respectively, and determining the maximum color channel value of each pixel in the target signal light area; When the maximum color channel value is greater than or equal to a preset brightness threshold, the brightness value of the pixel point is assigned to a highest-order brightness value; or when the maximum color channel value is less than the preset brightness threshold, the brightness value of the pixel point is assigned to a lowest-order brightness value; A highlight area mask corresponding to the target signal light area is generated based on the re-assigned pixel points.

7. The image fusion method according to any one of claims 1 to 6, characterized in that: Also includes: When the first eye-of-the-light region feature matches the second eye-of-the-light region feature, the bright frame image and the dark frame image are fused to obtain the signal light enhanced image.

8. An image fusion device, characterized in that: include: An image acquisition module, used to acquire a bright frame image and a dark frame image from a surveillance video stream, wherein the exposure intensity of the bright frame image is greater than the exposure intensity of the dark frame image; A feature matching module, used to determine whether a feature of a first lamp eye region of the bright frame image matches a feature of a second lamp eye region of the dark frame image; An image filling module, used for filling the eye area in the target image with the target non-eye area image when the first eye area feature and the second eye area feature do not match, to obtain a processed target bright frame image and a target dark frame image; wherein the target non-eye area image is an image of a single-eye signal light without lighting, and the target image is the bright frame image or the dark frame image; The image fusion module is used to fuse the target bright frame image and the target dark frame image to obtain a signal light enhanced image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image fusion method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image fusion method according to any one of claims 1 to 7 is implemented.

Images