Image fusion method, target detection method, monitoring method, medium and drone

Through image fusion method and drone monitoring, the problem of identifying illegal operations of dump trucks under low illumination at night is solved, and the fine-grained identification and off-site law enforcement of night targets are achieved, which reduces law enforcement costs and expands the scope of law enforcement.

CN118710519BActive Publication Date: 2024-11-26HANGZHOU GAODE DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411190594.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-26
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

Existing single sensor imaging equipment cannot accurately identify illegal operations of dump trucks at night, especially under low illumination conditions, which cannot achieve the identification of fine target scenes, resulting in the inability to monitor and collect evidence in real time, increasing law enforcement costs and limiting law enforcement. scope.

Method used

The image fusion method is adopted to achieve cross-modal feature fusion through visible light low-illumination image enhancement algorithm and infrared image feature encoding, and generate dual-light fusion images, and monitor them with drones to achieve night target recognition and off-site law enforcement.

Benefits of technology

It realizes fine-grained identification of targets in night scenes, reduces law enforcement costs, expands law enforcement areas, avoids the problem of vulnerability of fixed cameras, and provides a set of linkage technical paths from detection to identification to evidence collection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118710519B_ABST
    Figure CN118710519B_ABST
Patent Text Reader

Abstract

The present application discloses an image fusion method, including: based on a visible light low-illuminance image enhancement algorithm, imaging enhancement is performed on a visible light image acquired under conditions below a preset illumination to obtain an enhanced visible light image; image feature encoding is performed on an infrared image acquired under conditions below a preset illumination and an enhanced visible light image to obtain an explicit feature vector of infrared thermal imaging target temperature and an explicit feature vector of visible light imaging target detail texture; cross-modal feature fusion is performed on the above two feature vectors to achieve spatial alignment of feature vectors of two different modalities and attention position encoding of feature vectors; and a dual-light fusion image after fusion at a preset resolution is generated through an image reconstruction algorithm. The present application can fuse multi-dimensional information through the above dual-light image feature fusion algorithm, and fuses the texture detail features of visible light while enhancing foreground target imaging, and ultimately can realize the recognition of fine-grained features of targets in night scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of smart city governance technology, and more specifically, to an image fusion method, a target detection method, a monitoring method, a medium and a drone. Background Art

[0002] With the development of the economy, various construction projects are being carried out in full swing. Construction vehicles are speeding on the streets, especially dump trucks running red lights at night or performing other illegal operations, posing huge threats and safety hazards to people's lives and property.

[0003] Existing monitoring methods are generally based on fixed cameras on site or near the construction site to capture illegal operations, and then identify the scene through artificial intelligence algorithms. However, fixed cameras are easily damaged by various factors on site, resulting in the inability to capture illegal operations and identify scenes normally.

[0004] Illegal operations of dump trucks usually occur at night. However, a single sensor imaging device cannot meet the requirements of nighttime target scene recognition imaging and cannot accurately identify fine target scenes at night. For example, visible light imaging has weak imaging light at night, and infrared imaging is difficult to meet recognition requirements due to the lack of texture detail information.

[0005] It is impossible to monitor and collect evidence of illegal operations of dump trucks at night in real time under the conditions of non-site and contactless enforcement, which makes the manpower cost of on-site enforcement by relevant departments high, and the scope of enforcement area is also very limited. Summary of the invention

[0006] In response to at least one defect or improvement need in the prior art, the present application provides an image fusion method, a target detection method, a monitoring method, a medium and a drone, which are used to at least solve the technical problem that a single sensor imaging device cannot meet the requirements of night target scene recognition imaging and cannot accurately identify fine target scenes at night.

[0007] To achieve the above objectives, in a first aspect, the present application provides an image fusion method.

[0008] include:

[0009] Based on the visible light low-illuminance image enhancement algorithm, the visible light image acquired under the condition of illumination lower than the preset illumination is enhanced to obtain an enhanced visible light image;

[0010] Performing image feature encoding on the infrared image acquired under the condition lower than the preset illumination and the enhanced visible light image to obtain an explicit feature vector of the temperature of the infrared thermal imaging target and an explicit feature vector of the detail texture of the visible light imaging target;

[0011] Performing cross-modal feature fusion on the infrared thermal imaging target temperature explicit feature vector and the visible light imaging target detail texture explicit feature vector to achieve spatial alignment of feature vectors of two different modalities and attention position encoding of feature vectors;

[0012] Through the image reconstruction algorithm, based on the spatial alignment of the feature vectors of two different modalities and the attention position encoding of the feature vectors, a dual-light fusion image with a preset resolution is generated.

[0013] Furthermore, the visible light low illumination image enhancement algorithm includes:

[0014] Using unsupervised learning, construct one or more of an image space consistency function, an exposure control function, a chromaticity control function, and a brightness smoothing function;

[0015] By weighted control of one or more of the above functions, the objective function of the visible light low-illumination image enhancement algorithm is constructed.

[0016] Furthermore, the cross-modal feature fusion of the infrared thermal imaging target temperature explicit feature vector and the visible light imaging target detail texture explicit feature vector comprises:

[0017] Obtain the degree of association between each position of the infrared thermal imaging target temperature explicit feature vector in the query sentence state and all positions in the visible light imaging target detail texture explicit feature vector in the keyword state corresponding to the data item state, and obtain a similarity matrix;

[0018] Normalizing the similarity matrix to obtain a weight matrix;

[0019] The weight matrix is ​​multiplied by the explicit feature vector of the visible light imaging target detail texture of the data item state, and weighted summation is performed to obtain a new image feature vector after adding conditional attention.

[0020] Furthermore, the objective function of the visible light low illumination image enhancement algorithm is:

[0021] ;

[0022] in, F t ( I l ) represents the objective function of the visible light low illumination image enhancement algorithm, I l represents the input visible light low illumination image; F sc ( I l )and α1 represent the image space consistency function and its corresponding control weight respectively; F ec ( I l ) and α 2 respectively represent the exposure control function and its corresponding control weight; F cc ( I l ) and α 3 respectively represent the chromaticity control function and its corresponding control weight; F is ( I l ) and α 4 represent the brightness smoothing function and its corresponding control weight respectively;

[0023] ;

[0024] in, K Indicates the number of local areas; Indicates local area i Four adjacent local areas centered on the center; Y n Represents the local area of ​​the enhanced visible light low-light image n The average intensity of I n Represents the local area of ​​the input visible light low-light image n The average intensity of

[0025] ;

[0026] in, M Indicates size m × m The number of non-overlapping local regions;

[0027] ;

[0028] in,( p,q ) represents a pair of color channels; J n Indicates the enhanced visible light low illumination image n Average brightness of a color channel;

[0029] ;

[0030] in, N Indicates the number of iterations; represents the horizontal gradient; represents the vertical gradient; In c Represents the local area of ​​the input visible light low-light image n middle c The average intensity of the color channel.

[0031] In a second aspect, the present application provides a target detection method, comprising: based on a trained target detection and recognition model, detecting and identifying a target to be identified on a fused dual-light fusion image generated by any of the image fusion methods described above, to obtain a target recognition result.

[0032] In a third aspect, the present application provides a monitoring method, comprising:

[0033] Use drones to obtain visible light images and infrared images under conditions below a preset illumination;

[0034] Based on the visible light image and the infrared image, the aforementioned target detection method is used to monitor illegal operations of vehicles under conditions lower than the preset illumination.

[0035] Furthermore, the monitoring of illegal operation of a vehicle under the condition of illumination lower than the preset illumination includes:

[0036] Identify the types of materials loaded on the muck truck and obtain the material type identification results;

[0037] If illegal materials are found in the material type identification results, the license plate area of ​​the dump truck is photographed and the license plate is identified to achieve evidence collection for non-site law enforcement.

[0038] Furthermore, after the use of a drone to obtain a visible light image and an infrared image under a condition of illumination lower than a preset illumination, the method further includes:

[0039] Using a muck truck detection algorithm to detect and identify the muck truck in the visible light image and / or infrared image, and using a muck truck loading behavior detection algorithm to detect and identify the muck truck loading behavior in the visible light image and / or infrared image;

[0040] If the detection and identification result shows that a muck truck and / or a muck truck that is loading materials exists, the image fusion method described in any one of the above items is used to fuse the visible light image and the infrared image.

[0041] In a fourth aspect, the present application provides a storage medium storing a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device is enabled to execute the steps of the image fusion method described in any one of the preceding items, to execute the steps of the target detection method described in any one of the preceding items, and / or to execute the steps of the monitoring method described in any one of the preceding items.

[0042] In a fifth aspect, the present application provides a drone storing a set of instruction sets, which are executed by the drone to implement the steps of the image fusion method described in any one of the preceding items, the steps of the target detection method described in any one of the preceding items, and / or the steps of the monitoring method described in any one of the preceding items.

[0043] In general, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art:

[0044] (1) This application can fuse multi-dimensional information through a dual-light image feature fusion algorithm, which can enhance the foreground target imaging while fusing the texture detail features of visible light, and ultimately realize the recognition of fine-grained features of the target in night scenes.

[0045] (2) This application can realize image enhancement of low-light visible light imaging at night through a visible light low-light image enhancement algorithm. During the algorithm training stage, only a low-light image needs to be input to restore the image at normal light, which provides a technical basis for the subsequent dual-light image feature fusion.

[0046] (3) This application utilizes the flexible and maneuverable characteristics of drones to achieve non-site law enforcement monitoring and evidence collection, thereby avoiding as much as possible the situation where fixed cameras are easily damaged by various factors on site and cannot normally capture illegal operations and identify scenes, reducing the manpower cost of on-site law enforcement by relevant departments and expanding the scope of law enforcement areas. This application realizes a complete set of linkage technology implementation paths from detection to identification and then to evidence collection, providing a new technical path for application scenarios such as smart city governance, smart construction site governance, and illegal operation governance of muck trucks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of an image fusion method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of a monitoring method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0051] The terms "including" or "having" and any variations thereof in the specification, claims or drawings of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0052] As described in the background technology section of the specification, construction vehicles such as muck trucks generally choose to perform illegal operations at night. However, a single fixed sensor imaging device cannot meet the requirements of night target scene recognition imaging, and cannot accurately recognize fine target scenes at night. For example, visible light imaging has weak imaging light at night, and infrared imaging is difficult to meet recognition requirements due to the lack of texture detail information. In view of this, this application proposes an image fusion method to overcome the above technical defects.

[0053] refer to Figure 1 An embodiment of the present application proposes an image fusion method, which may mainly include the following steps 1 to 4.

[0054] Step 1: Based on the visible light low-light image enhancement algorithm, the visible light image acquired at night (below the preset illumination) (can be acquired through the deployed public security surveillance camera or the camera on the drone, and the image acquisition method is not limited) is enhanced, such as balancing the brightness of over-exposed and under-exposed areas, restoring dark light detail features, and removing noise, to obtain an enhanced visible light image.

[0055] In view of the technical problem that the illumination of visible light imaging at night is low and there are many noise points, which makes it difficult for subsequent algorithms to provide effective identifiable features, the present application specifically designs the visible light low-illumination image enhancement algorithm. At the same time, considering the difficulty of obtaining paired data (normal illumination and low-illumination image pairs) based on supervised learning, the visible light low-illumination image enhancement algorithm adopts an unsupervised learning method, that is, in the training stage of the algorithm, only the low-illumination image of visible light needs to be input to restore the corresponding normal illumination imaging image. The algorithm mainly realizes the modeling of image space consistency function, exposure control function, chromaticity control function and brightness smoothing function. At the same time, the objective function of the visible light low-illumination image enhancement algorithm is synthesized through weighted control of various functions. The modeling formula refers to the following formula (1).

[0056] (1)

[0057] in, F t ( I l ) represents the objective function of the visible light low illumination image enhancement algorithm, I l represents the input visible light low illumination image; F sc ( I l )and α 1 represents the image space consistency function and its corresponding control weight respectively; F ec ( I l ) and α 2 respectively represent the exposure control function and its corresponding control weight; F cc ( I l ) and α 3 represent the chromaticity control function and its corresponding control weight respectively; F is ( I l ) and α 4 represent the brightness smoothing function and its corresponding control weight respectively. Preferably, α 1: α 2: α 3: α 4=1:1:2:3. This parameter is a better experimental parameter and can be fine-tuned on this basis.

[0058] Image space consistency function F sc ( I l) It mainly promotes the spatial consistency of images by maintaining the gradient of the adjacent local areas of the input image (the abbreviation of the input visible light low-light image) and the enhanced image (the abbreviation of the enhanced visible light low-light image). First, the average value of the input image (I) and the enhanced image (Y) in the channel dimension is calculated to obtain two grayscale images, which are then decomposed into several 4×4 patches (patch, that is, patch). Finally, the center of the patch is calculated. i With adjacent j The difference of pixels and then the average value. Image space consistency function F sc ( I l ) is expressed in formula (2).

[0059] (2)

[0060] in, K Indicates the number of local areas; Indicates local area i Four adjacent local regions (upper, lower, left and right) centered at the center; Y n Represents the local area of ​​the enhanced visible light low-light image n The average intensity of I n Represents the local area of ​​the input visible light low-light image n The average intensity.

[0061] Exposure control function F ec ( I l ) Suppress underexposed or overexposed areas, control exposure levels, and perform exposure control for local areas. Exposure control function F ec ( I l ) is expressed in Formula (3).

[0062] (3)

[0063] in, M Indicates size m × m ( m The initial value of is 16. This parameter is a better experimental parameter and can be fine-tuned according to the specific scenario. Y n Represents the local area of ​​the enhanced visible light low-light image n The average intensity.

[0064] Chroma Control Function F cc ( I l ) is mainly to enhance the potential color cast in the visible light image and build the relationship between the three color channels. First, the visible light image is divided into three color channels: R, G, and B (two color channels are combined into ε ),against ε Any combination of ( p , q ) (Formula means: ) Calculate the average brightness of each color channel, then subtract the average brightness of different color channels from each other to find the average sum. Chroma control function F cc ( I l ) is expressed in Formula (4).

[0065] (4)

[0066] in,( p,q ) represents a pair of color channels; J n Indicates the enhanced visible light low illumination image n The average brightness of the color channel.

[0067] Brightness smoothing function F is ( I l ) To maintain the monotonic relationship between adjacent pixels, it plays a smooth transition role. Brightness smoothing function F is ( I l ) is expressed in Formula (5).

[0068] (5)

[0069] in, N Indicates the number of iterations; represents the horizontal gradient; represents the vertical gradient; I n c Represents the local area of ​​the input visible light low-light image n middle c The average intensity of the color channel.

[0070] The present application can realize imaging enhancement of low-illuminance visible light imaging at night through a visible light low-illuminance image enhancement algorithm. During the algorithm training stage, it only needs to input a low-illuminance image to restore an imaging image with normal illumination, providing a technical basis for subsequent dual-light image feature fusion.

[0071] Step 2: Encode the image features of the infrared image acquired at night (below the preset illumination) and the enhanced visible light image to obtain the explicit feature vector of the temperature of the infrared thermal imaging target and the explicit feature vector of the detail texture of the visible light imaging target.

[0072] That is, the acquired infrared image and the enhanced visible light image are encoded with image features. This step 2 mainly realizes the encoding of the temperature explicit features of the infrared thermal imaging target and the detail texture explicit features of the visible light imaging target, and realizes the vectorization of different image features.

[0073] Step 3: Use a dual-light image feature fusion algorithm to perform cross-modal feature fusion on the temperature explicit feature vector of the infrared thermal imaging target and the detail texture explicit feature vector of the visible light imaging target, so as to achieve spatial alignment of feature vectors of two different modalities and encoding of the attention position of feature vectors, and solve the technical problems of unified representation of cross-modal feature vectors and information association fusion. The dual-light image feature fusion algorithm mainly realizes the functions of information extraction, encoding and cross-modal feature fusion of two modalities, and solves the limitation problem of long-distance small target imaging of single-modal sensors in night scenes, that is, solves the technical difficulties that infrared imaging features are single and lack texture features, while visible light night imaging targets are not prominent.

[0074] More specifically, the key process steps of the bi-optical image feature fusion algorithm are as follows.

[0075] Step 31, two different modal coding features: infrared image feature Q, visible light image feature P (P = K = V). For each position of the infrared image feature Q, calculate its correlation with all positions in the visible light image feature K to obtain the similarity matrix W.

[0076] Step 32: normalize (Softmax) the similarity matrix W to obtain a weight matrix. The weight matrix represents the degree of association between each position of the infrared image feature Q and the visible light image feature K.

[0077] Step 33: Then multiply the weight matrix by the visible light image feature V, perform weighted summation, and obtain a new image feature representation after adding conditional attention.

[0078] The function modeling of the dual-light image feature fusion algorithm is shown in formula (6).

[0079] (6)

[0080] in, , and They represent infrared image feature Q, visible light image feature K and visible light image feature V respectively; represents the similarity matrix W; represents the weight matrix; represents the new image feature after adding conditional attention; d represents the dimension of the visible light image feature K.

[0081] Q is a set of query statements, and V is a database with several data items. For each query statement, it is expected to query a data item (after weighting) from the database. How to query? This requires considering both each q itself and each item in V. If K represents a set of keys, each key in this set corresponds to each item in V, representing a certain query feature of each item in V (so the number of K and V must be equal, and there is no strict limit on the dimension. When doing attention, the dimension is the same as q, just for the convenience of doing dot product, but there is also attention without dot product). Then, for each q in Q, sum the attention of each k as the weighting coefficient of the corresponding Value, and use it to weight each item in the database V, and you will get the query result expected by q. That is: Q is equivalent to the question, V is equivalent to a set of answers, K is equivalent to the keyword of the question corresponding to the answer, Q and K calculate the similarity, which is used to assign weights to V, and finally get a reasonable answer corresponding to the question Q.

[0082] Step 4: Generate a dual-light fusion image after fusion at a preset resolution through an image reconstruction algorithm based on the spatial alignment of feature vectors of two different modalities and the attention position encoding of feature vectors, and output the fused dual-light fusion image.

[0083] This application can fuse multi-dimensional information through a dual-light image feature fusion algorithm, which can enhance the foreground target imaging while fusing the texture detail features of visible light, and ultimately realize the recognition of fine-grained features of targets in night scenes.

[0084] An embodiment of the present application also provides a target detection method, which detects and identifies the target to be identified on the fused dual-light fusion image generated by the aforementioned image fusion method based on a trained target detection and recognition model, and obtains a target recognition result. The target detection method of the present application overcomes the technical difficulties that visible light imaging has weak imaging light at night and infrared imaging has difficulty meeting recognition requirements due to the lack of texture detail information, and can realize accurate recognition of fine-grained features of targets in night scenes.

[0085] An embodiment of the present application also provides a monitoring method, referring to Figure 2 The monitoring method as a whole can include the following steps 01 to 07.

[0086] Step 01: Use the dual-light sensor mounted on the drone to obtain visible light images and infrared images at night (below the preset illumination).

[0087] Step 02: Detect, identify and warn the muck truck in the infrared image using the muck truck detection algorithm; detect, identify and warn the muck truck loading behavior based on the imaging features of the muck truck loading behavior using the muck truck loading behavior detection algorithm (the muck truck loading behavior detection algorithm can use the target detection method of the aforementioned embodiment).

[0088] Step 03: Determine whether there is a muck truck and / or whether there is a muck truck loading materials based on the recognition results of the two algorithms in step 02.

[0089] Step 04: If the target in step 03 exists, the infrared and visible light image features of the target area (this step is performed based on the small-size image of the target area in order to save computing resources) are fused and enhanced. The specific fusion and enhancement steps refer to the aforementioned embodiments of the image fusion method and will not be repeated here.

[0090] Step 05: Use a material type recognition algorithm to identify the type of material loaded on the muck truck using the fused dual-light fusion image to obtain a material type recognition result. The types of materials include muck, soil, building materials, and construction waste. The material type recognition algorithm can use the target detection method in the aforementioned embodiment, which will not be described in detail here.

[0091] Step 06: Determine whether there is any illegal material loading in the material type identification results, that is, determine whether there is any illegal operation behavior.

[0092] Step 07: If there is any illegal operation described in step 06 (i.e., there is loading of illegal materials), then the evidence collection process is carried out. First, the drone is linked to adjust the zoom and shooting angle of the target area, then the license plate area is captured, and finally the license plate is recognized to complete the evidence collection process. The license plate recognition can also adopt the target detection method in the above embodiment, which will not be repeated here.

[0093] This application uses the flexible and maneuverable characteristics of drones to achieve non-site law enforcement monitoring and evidence collection, thereby avoiding as much as possible the situation where fixed cameras are easily damaged by various factors on site and cannot normally capture illegal operations and identify scenes, reducing the manpower cost of on-site law enforcement by relevant departments and expanding the scope of law enforcement areas. This application realizes a complete set of linkage technology implementation paths from detection to identification and then to evidence collection, providing a new technical path for application scenarios such as smart city governance, smart construction site governance, and illegal operation governance of muck trucks.

[0094] The embodiments of the present application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the steps of the method (including the image fusion method, the target detection method, and the monitoring method) according to the embodiments of the present application can be implemented.

[0095] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In an embodiment of the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0096] An embodiment of the present application also provides a drone, which stores a set of instruction sets, and the instruction sets are executed by the drone to implement the steps of any of the aforementioned image fusion methods, the steps of the aforementioned target detection methods, and / or the steps of any of the aforementioned monitoring methods.

[0097] The drone may include multiple subsystems, such as a drive system, a control system, a perception system, a communication system, and a computing system. The drone may also include more or fewer subsystems, and each subsystem may also include multiple components, which are not described in detail here.

[0098] The drive system includes components that provide power movement for the drone, such as engines, energy sources, transmissions, etc.

[0099] The control system includes components that provide control for the drone, such as the flight controller.

[0100] The perception system includes components that provide the drone with environmental awareness, such as the drone positioning system, laser sensors, voice sensors, ultrasonic sensors, and camera equipment.

[0101] The communication system includes components that provide communication connections for the drone, such as mobile communication networks (such as 3G, 4G and 5G networks, etc.), WiFi, Bluetooth and machine networking, etc.

[0102] The computing and processing system includes components that provide data computing and processing capabilities for the drone. The computing and processing system may include at least one processor and a memory. The processor may execute instructions stored in the memory. The processor may be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or a combination thereof. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a disk or an optical disk, etc.

[0103] In an embodiment of the present application, a set of instruction sets is stored in the memory, and the processor can execute the instruction set to implement all or part of the steps of the image fusion method described in any of the above exemplary embodiments, or implement all or part of the steps of the target detection method in the above exemplary embodiments, or implement all or part of the steps of the monitoring method described in any of the above exemplary embodiments.

[0104] It should be noted that the functional modules in the various embodiments of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product.

[0105] The flowchart and / or block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart and / or block diagram can represent a part of a module, program segment or code, and a part of the above-mentioned module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the technical features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope of the present application.

[0107] Although the present application has been shown and described with reference to specific exemplary embodiments of the present application, it should be understood by those skilled in the art that various changes in form and details may be made to the present application without departing from the spirit and scope of the present application as defined by the appended claims and their equivalents. Therefore, the scope of the present application should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. An image fusion method, characterized in that: include: Based on the visible light low illumination image enhancement algorithm, the visible light image acquired under the condition of illumination lower than the preset illumination is enhanced to obtain the enhanced visible light image; The objective function of the visible light low illumination image enhancement algorithm is: ; in, represents the objective function of the visible light low illumination image enhancement algorithm, represents the input visible light low illumination image; and α 1 represents the image space consistency function and its corresponding control weight respectively; and α 2 respectively represent the exposure control function and its corresponding control weight; and α 3 represent the chromaticity control function and its corresponding control weight respectively; and α 4 represent the brightness smoothing function and its corresponding control weight respectively; ; in, K Indicates the number of local areas; Indicates a local area i Four adjacent local areas centered on the Represents the local area of ​​the enhanced visible light low-light image n The average intensity of I n Represents the local area of ​​the input visible light low illumination image n The average intensity of ; in, M Indicates size m × m The number of non-overlapping local regions; ; in, represents a pair of color channels; Indicates the enhanced visible light low illumination image n Average brightness of the color channel; ; in, N Indicates the number of iterations; represents the horizontal gradient; represents the vertical gradient; I n c Represents the local area of ​​the input visible light low illumination image n middle c Average intensity of color channels; Performing image feature encoding on the infrared image acquired under conditions lower than the preset illumination and the enhanced visible light image to obtain an explicit feature vector of the temperature of the infrared thermal imaging target and an explicit feature vector of the detail texture of the visible light imaging target; Performing cross-modal feature fusion on the infrared thermal imaging target temperature explicit feature vector and the visible light imaging target detail texture explicit feature vector to achieve spatial alignment of feature vectors of two different modalities and attention position encoding of feature vectors; Through the image reconstruction algorithm, based on the spatial alignment of the feature vectors of two different modalities and the attention position encoding of the feature vectors, a dual-light fusion image with a preset resolution is generated.

2. The image fusion method according to claim 1, wherein: The visible light low illumination image enhancement algorithm includes: Using an unsupervised learning approach, construct one or more of an image space consistency function, an exposure control function, a chromaticity control function, and a brightness smoothing function; By weighted control of one or more of the above functions, the objective function of the visible light low illumination image enhancement algorithm is constructed.

3. The image fusion method according to claim 1, wherein: The cross-modal feature fusion of the infrared thermal imaging target temperature explicit feature vector and the visible light imaging target detail texture explicit feature vector comprises: Obtain the degree of correlation between each position of the infrared thermal imaging target temperature explicit feature vector in the query sentence state and all positions of the visible light imaging target detail texture explicit feature vector in the corresponding keyword state in the data item state, and obtain a similarity matrix; Normalizing the similarity matrix to obtain a weight matrix; The weight matrix is ​​multiplied by the visible light imaging target detail texture explicit feature vector of the data item state, and weighted summation is performed to obtain a new image feature vector after adding conditional attention.

4. A target detection method, characterized in that: include: Based on the trained target detection and recognition model, the target to be identified is detected and recognized on the fused dual-light fusion image generated by the image fusion method according to any one of claims 1 to 3 to obtain a target recognition result.

5. A monitoring method, characterized in that: include: Use drones to acquire visible light and infrared images under conditions below a preset illumination level; Based on the visible light image and the infrared image, the target detection method according to claim 4 is used to monitor illegal operations of vehicles under conditions lower than the preset illumination.

6. The monitoring method according to claim 5, wherein: The monitoring of illegal operation of a vehicle under the condition of illumination lower than the preset illumination includes: Identify the types of materials loaded on the muck truck and obtain the material type identification results; If illegal materials are found in the material type identification results, the license plate area of ​​the dump truck will be photographed and the license plate will be identified to achieve evidence collection for non-site law enforcement.

7. The monitoring method according to claim 6, wherein: After the drone is used to acquire the visible light image and the infrared image under the condition of illumination lower than the preset illumination, the method further includes: Using a muck truck detection algorithm to detect and identify the muck truck in the visible light image and / or infrared image, and using a muck truck loading behavior detection algorithm to detect and identify the muck truck loading behavior in the visible light image and / or infrared image; If the detection and identification results show that a muck truck and / or a muck truck that is loading materials exists, the image fusion method according to any one of claims 1 to 3 is used to fuse the visible light image and the infrared image.

8. A storage medium, characterized in that: It stores a computer program that can be executed by an access authentication device. When the computer program is run on the access authentication device, the access authentication device can execute the steps of the image fusion method described in any one of claims 1 to 3, the steps of the target detection method described in claim 4, and / or the steps of the monitoring method described in any one of claims 5 to 7.

9. A drone, characterized in that: A set of instruction sets is stored, and the instruction sets are executed by the drone to implement the steps of the image fusion method described in any one of claims 1 to 3, the steps of the target detection method described in claim 4, and / or the steps of the monitoring method described in any one of claims 5 to 7.