Image processing method, electronic device, vehicle, medium, and computer program product

Abstract

The embodiment of the invention provides an image processing method, an electronic device, a vehicle, a medium and a computer program product, and the method comprises the steps: obtaining a first image frame group and a second image frame group, the collection time of the first image frame group is the same as that of the second image frame group, the first image frame group is collected through a far infrared camera, and the second image frame group is collected through a far infrared camera; the second image frame group is acquired through a near-infrared camera; and carrying out fusion processing on the first image frame group and the second image frame group to obtain a target image frame group, namely, carrying out fusion processing on the far infrared image and the near infrared image, so that the accuracy of the obtained image information can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automobiles, and particularly relates to an image processing method, an electronic device, a vehicle, a medium and a computer program product. BACKGROUND

[0002] With the continuous development of intelligent technology, intelligent driving technology is becoming mature. In order to improve driving safety, it is very important to accurately perceive the surrounding environment through a camera during driving, especially in the scene of night and bad weather such as rain, snow and fog.

[0003] In the related art, the image information of the surrounding environment is obtained by fusing a far-infrared image and a visible light image.

[0004] However, the accuracy of the image information obtained by the method in the related art is not high. SUMMARY

[0005] The embodiments of the application provide an image processing method, an electronic device, a vehicle, a medium and a computer program product to solve the problem of low accuracy of the image information obtained in the prior art.

[0006] The first aspect of the embodiments of the application provides an image processing method, which comprises:

[0007] obtaining a first image frame group and a second image frame group, wherein the acquisition times of the first image frame group and the second image frame group are the same, the first image frame group is acquired by a far-infrared camera, and the second image frame group is acquired by a near-infrared camera;

[0008] performing fusion processing on the first image frame group and the second image frame group to obtain a target image frame group.

[0009] Optionally, the fusion processing on the first image frame group and the second image frame group to obtain a target image frame group comprises:

[0010] performing target detection on each first image in the first image frame group to obtain a first target detection result;

[0011] performing target detection on each second image in the second image frame group to obtain a second target detection result;

[0012] for each first image, performing fusion processing on the first image and the second image corresponding to the first image based on the first target detection result of the first image and the second target detection result of the second image corresponding to the first image to obtain a target detection result of a target image.

[0013] Optionally, the step of performing target detection on each frame of the first image in the first image frame group to obtain a first target detection result includes:

[0014] Based on each frame of the first image in the first image frame group, feature extraction is performed on the first image to obtain a multi-scale first feature map;

[0015] The first feature map of the multi-scale is aggregated in a dual-path manner, first from bottom to top and then from top to bottom, to obtain a fused first feature map of the multi-scale.

[0016] Target detection is performed on the fused multi-scale first feature map to obtain the first target detection result.

[0017] Optionally, before performing bottom-up and then top-down dual-path aggregation on the multi-scale first feature map to obtain the fused multi-scale first feature map, the method further includes:

[0018] The first feature map of the multi-scale is subjected to feature enhancement processing to obtain the first feature map of the multi-scale after feature enhancement processing.

[0019] Optionally, before performing target detection on each frame of the first image in the first image frame group to obtain the first target detection result, the method further includes:

[0020] The scale of each first image in the first image frame group is scaled to obtain a scaled first image, and the scaled first image has the same resolution as the second image.

[0021] Optionally, the step of performing target detection on each frame of the second image in the second image frame group to obtain a second target detection result includes:

[0022] Based on each frame of the second image in the second image frame group, feature extraction is performed on the second image to obtain a multi-scale second feature map;

[0023] The multi-scale second feature map is aggregated using a dual-path approach, first from bottom to top and then from top to bottom, to obtain a fused multi-scale second feature map.

[0024] Target detection is performed at each location of the fused multi-scale second feature map to obtain the second target detection result.

[0025] Optionally, before performing bottom-up and then top-down dual-path aggregation on the multi-scale second feature map to obtain the fused multi-scale second feature map, the method further includes:

[0026] The second feature map at multiple scales is subjected to feature enhancement processing to obtain the second feature map at multiple scales after feature enhancement processing.

[0027] Optional, also includes:

[0028] Based on the first target detection result of each frame of the first image in the first image frame group, target tracking is performed on the first image frame group to obtain the first target tracking result;

[0029] Based on the second target detection result of each frame of the second image in the first image frame group, target tracking is performed on the second image frame group to obtain the second target tracking result;

[0030] The first target tracking result and the second target tracking result are fused to obtain the target tracking result of the target image frame group.

[0031] Optionally, the step of performing target tracking on the first image frame group based on the first target detection result of each first image in the first image frame group to obtain a first target tracking result includes:

[0032] Target trajectory prediction is performed on the first image of the previous frame in the first image frame group to obtain the first target prediction box of the first image of the current frame.

[0033] The first target prediction bounding box of the first image in the current frame is matched with the first target detection result of the first image in the current frame to obtain the first target tracking result.

[0034] Optionally, the step of performing target tracking on the second image frame group based on the second target detection result of each frame of the second image in the first image frame group to obtain the second target tracking result includes:

[0035] Perform target trajectory prediction on the previous frame of the second image in the second image frame group to obtain the second target prediction box of the current frame of the second image;

[0036] The second target prediction bounding box of the second image in the current frame is matched with the second target detection result of the second image in the current frame to obtain the second target tracking result.

[0037] Optionally, before fusing the first target tracking result and the second target tracking result to obtain the target tracking result of the target image frame group, the method further includes:

[0038] Perform multi-frame verification on the first image frame group to obtain the first multi-frame verification result, and perform correction processing on the first image frame group based on the first multi-frame verification result.

[0039] Perform multi-frame verification on the second image frame group to obtain the second multi-frame verification result, and perform correction processing on the second image frame group based on the second multi-frame verification result.

[0040] A second aspect of this application provides an image processing apparatus, the apparatus comprising:

[0041] The acquisition module is used to acquire a first image frame group and a second image frame group, wherein the first image frame group and the second image frame group are acquired at the same time, the first image frame group is acquired by a far-infrared camera, and the second image frame group is acquired by a near-infrared camera.

[0042] The processing module is used to perform fusion processing on the first image frame group and the second image frame group to obtain the target image frame group.

[0043] Optionally, the processing module is specifically used to perform target detection on each frame of the first image in the first image frame group to obtain a first target detection result; to perform target detection on each frame of the second image in the second image frame group to obtain a second target detection result; and for each frame of the first image, based on the first target detection result of the first image and the second target detection result of the second image corresponding to the first image, to perform fusion processing on the first image and the second image corresponding to the first image to obtain the target detection result of the target image.

[0044] Optionally, the processing module is specifically used to extract features from the first image based on each frame of the first image in the first image frame group to obtain a multi-scale first feature map; to perform dual-path aggregation on the multi-scale first feature map, first from bottom to top and then from top to bottom, to obtain a fused multi-scale first feature map; and to perform target detection on the fused multi-scale first feature map to obtain the first target detection result.

[0045] Optionally, the processing module is further configured to perform feature enhancement processing on the multi-scale first feature map respectively to obtain the multi-scale first feature map after feature enhancement processing.

[0046] Optionally, the acquisition module is further configured to scale each frame of the first image in the first image frame group to obtain a scale-transformed first image, wherein the scale-transformed first image has the same resolution as the second image.

[0047] Optionally, the processing module is specifically used to extract features from the second image based on each frame of the second image in the second image frame group to obtain a multi-scale second feature map; to perform dual-path aggregation on the multi-scale second feature map from bottom to top and then from top to bottom to obtain a fused multi-scale second feature map; and to perform target detection at each position of the fused multi-scale second feature map to obtain the second target detection result.

[0048] Optionally, the processing module is further configured to perform feature enhancement processing on the multi-scale second feature map respectively to obtain the multi-scale second feature map after feature enhancement processing.

[0049] Optionally, the processing module is further configured to perform target tracking on the first image frame group based on the first target detection result of each first image in the first image frame group to obtain a first target tracking result; perform target tracking on the second image frame group based on the second target detection result of each second image in the first image frame group to obtain a second target tracking result; and perform fusion processing on the first target tracking result and the second target tracking result to obtain the target tracking result of the target image frame group.

[0050] Optionally, the processing module is specifically used to predict the target trajectory of the first image in the previous frame of the first image frame group to obtain the first target prediction box of the first image in the current frame; and to match the first target prediction box of the first image in the current frame with the first target detection result of the first image in the current frame to obtain the first target tracking result.

[0051] Optionally, the processing module is specifically used to predict the target trajectory of the previous frame of the second image in the second image frame group to obtain the second target prediction box of the current frame of the second image; and to match the second target prediction box of the current frame of the second image with the second target detection result of the current frame of the second image to obtain the second target tracking result.

[0052] Optionally, the processing module is further configured to perform multi-frame verification on the first image frame group to obtain a first multi-frame verification result, and perform correction processing on the first image frame group based on the first multi-frame verification result; perform multi-frame verification on the second image frame group to obtain a second multi-frame verification result, and perform correction processing on the second image frame group based on the second multi-frame verification result.

[0053] A third aspect of this application provides an electronic device, comprising: a processor, the processor being connected to a memory, the memory storing a computer program / instructions executable on the processor, the computer program / instructions being executed by the processor to implement the steps of the image processing method as described in any of the first aspects.

[0054] A fourth aspect of this application provides a vehicle including the electronic device described in the third aspect.

[0055] A fifth aspect of this application provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the image processing method as described in any of the first aspects.

[0056] A sixth aspect of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the image processing method as described in any of the first aspects.

[0057] The image processing method, electronic device, vehicle, medium, and computer program product provided in this application acquire a first image frame group and a second image frame group, wherein the first image frame group and the second image frame group are acquired at the same time, the first image frame group is acquired by a far-infrared camera, and the second image frame group is acquired by a near-infrared camera; the first image frame group and the second image frame group are fused to obtain a target image frame group, that is, the far-infrared image and the near-infrared image are fused, thereby improving the accuracy of the acquired image information. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application;

[0059] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0060] Figure 3 This is a schematic diagram of another image processing system provided in an embodiment of this application;

[0061] Figure 4 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0062] Figure 5A This is a schematic diagram of the structure of a target detection model provided in an embodiment of this application;

[0063] Figure 5B This is a schematic diagram of another target detection model provided in an embodiment of this application;

[0064] Figure 5C This is a schematic diagram of the structure of a C2f module provided in an embodiment of this application;

[0065] Figure 5D This is a schematic diagram of the structure of an SCDown module provided in an embodiment of this application;

[0066] Figure 5E A schematic diagram of the structure of a C2fCIB module provided in an embodiment of this application;

[0067] Figure 5F This is a schematic diagram of the structure of a CIB module provided in an embodiment of this application;

[0068] Figure 5G This is a schematic diagram of the structure of an SPPF module provided in an embodiment of this application;

[0069] Figure 5H This is a schematic diagram of the structure of a PSA module provided in an embodiment of this application;

[0070] Figure 5I This is a schematic diagram of the structure of a Mamba module provided in an embodiment of this application;

[0071] Figure 6 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0072] Figure 7 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0073] Figure 8 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0074] Figure 9 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0075] Figure 10 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0076] Figure 11 This is a schematic diagram of the structure of another image processing system provided in an embodiment of this application;

[0077] Figure 12 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0078] Figure 13 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0079] Figure 14 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0080] Figure 15 This is a schematic diagram of the structure of another image processing system provided in an embodiment of this application;

[0081] Figure 16 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0082] Figure 17 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0083] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0084] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0085] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc., in the instruction sent. An indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0086] In related technologies, target detection and tracking are achieved by fusing far-infrared and visible light images. However, under conditions such as nighttime or inclement weather, when there is no significant temperature difference between the target and the environment (e.g., the body temperature of a pedestrian wearing a raincoat or thick clothing is blocked), the visible light camera cannot clearly perceive the surrounding environment, and the far-infrared camera cannot capture a clear image, resulting in low accuracy of the acquired image information. To address these issues, this application employs a far-infrared and near-infrared image fusion method, combined with deep learning-based target detection and tracking algorithms. Target detection and tracking are performed on far-infrared and near-infrared images captured simultaneously. Then, multiple frames of far-infrared and near-infrared images are fused to obtain the target tracking trajectory of the fused image. This improves the accuracy of the acquired image information, thereby enhancing the accuracy of target detection and tracking, and simultaneously improving vehicle driving safety.

[0087] Figure 1 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application, such as...Figure 1 As shown, the system includes: a far-infrared camera, a near-infrared camera, and a fusion unit; the far-infrared camera is used to acquire a first image frame group; the near-infrared camera is used to acquire a second image frame group; the fusion unit is used to perform fusion processing on the first image frame group and the second image frame group to obtain a target image frame group.

[0088] The technical solution of the applied image processing method is described below using several specific embodiments as examples:

[0089] Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The method of this embodiment is applied to, for example... Figure 1 The image processing system shown, such as Figure 2 As shown, the flow of the method in this embodiment is as follows:

[0090] S21: Obtain the first image frame group and the second image frame group.

[0091] The first image frame group and the second image frame group are acquired at the same time. The first image frame group is acquired by a far-infrared camera, and the second image frame group is acquired by a near-infrared camera.

[0092] Specifically, the far-infrared camera is a thermal infrared camera, also known as a thermal imaging camera. Employing thermal imaging technology, it can measure the temperature of any scene by acquiring a temperature distribution map and convert the heat emitted by objects into a visual image. Specifically, all objects with temperatures above absolute zero (-273.15℃) emit infrared radiation. The far-infrared camera uses this emitted infrared radiation to create an image, converting different intensities of infrared radiation into brightness values ​​in the image. Using pseudo-color display as an example, different colors represent different temperature ranges; for instance, red and yellow might indicate higher temperature areas, while blue and purple might indicate lower temperature areas. The far-infrared camera primarily operates in the long-wave infrared region, with a wavelength range of approximately 8 to 14 micrometers. The intensity and wavelength distribution of this infrared radiation are related to the object's temperature and do not depend on an external light source, thus allowing it to operate in dark conditions.

[0093] For example, if a smart car is equipped with a forward-looking thermal infrared camera and is driving on a rural road at night, it generates a set of first image frames. When pedestrians or wild animals approach the road, even though the surrounding environment is dark, these warm-blooded creatures will be hotter than the surrounding environment and will therefore stand out in each first image generated by the thermal infrared camera. If the thermal infrared camera images in false color, the pedestrians or wild animals will show obvious thermal outlines, and the heat distribution of their bodies will be clearly visible. In particular, the face, neck and other parts that dissipate heat quickly will appear in a color close to red, while other parts may appear yellow, green or blue.

[0094] Near-infrared cameras, also known as night vision cameras, capture near-infrared radiation, enabling us to see in the dark. Specifically, near-infrared cameras are equipped with near-infrared light sources, such as light-emitting diodes (LEDs) infrared halos. These sources emit near-infrared light to illuminate the target object, and the camera generates an image based on the object's reflectivity to this light. The image is displayed as a grayscale image, where brightness reflects the object's ability to reflect near-infrared light, rather than its color. Near-infrared light has a wavelength range of 700 to 2500 nanometers, is highly sensitive to weak signals, and penetrates some substances, such as smoke, more effectively than visible light. Near-infrared imaging can also distinguish materials that appear similar under visible light. Therefore, near-infrared cameras can operate at night or in low-light environments and clearly display object details.

[0095] Based on the foregoing example, suppose the intelligent driving vehicle is equipped with a forward-looking near-infrared camera that generates a set of second image frames. The second image generated by the near-infrared camera is close to a black and white photograph, but rich in detail. When a pedestrian approaches, the outline and details of the pedestrian in the image will be relatively clear, and the texture of the skin and clothing can be well distinguished. The pedestrian is wearing a black jacket and white cotton pants. Since the jacket is made of synthetic fibers, it is more reflective in the near-infrared band than in visible light. Therefore, the black jacket appears brighter in the near-infrared image. Since the cotton pants are made of cotton, the reflectivity in the near-infrared band is relatively low. Therefore, the white cotton pants appear darker in the near-infrared image.

[0096] S22: Perform fusion processing on the first image frame group and the second image frame group to obtain the target image frame group.

[0097] In this system, the first image frame group is a far-infrared image frame group, and the second image frame group is a near-infrared image frame group. Each first image frame in the first image frame group and each second image frame in the second image frame group with the same timestamp are fused separately to obtain target images that form the target image frame group. Each target image in the target image frame group exhibits the advantages and characteristics of both far-infrared and near-infrared images, displaying not only the temperature difference between the target and the environment but also the reflectivity of the target. One possible fusion method is a weighted average fusion method; this method primarily achieves fusion by assigning different weights to different features and then calculating a weighted average.

[0098] Specifically, the pixels of each frame of the first image and each frame of the second image are normalized and aligned to ensure a one-to-one correspondence between the pixels of the two images; then, weights are assigned based on the characteristics of the far-infrared and near-infrared images; finally, a weighted average is calculated for each pair of corresponding pixels, and the pixel value of the resulting fused image can be represented as:

[0099] I fused (x,y)=ω1×I1(x,y)+ω2×I2(x,y)

[0100] Where (x,y) represents the pixel position, I fused Let I1 represent the far-infrared image, ω1 represent the weight assigned to the far-infrared image, I2 represent the near-infrared image, and ω2 represent the weight assigned to the near-infrared image.

[0101] In this embodiment, by acquiring a first image frame group and a second image frame group, wherein the first image frame group and the second image frame group are acquired at the same time, the first image frame group is acquired by a far-infrared camera and the second image frame group is acquired by a near-infrared camera; and by performing a fusion processing on the first image frame group and the second image frame group, a target image frame group is obtained, that is, by fusing the far-infrared image and the near-infrared image, thereby improving the accuracy of the acquired image information.

[0102] In the above embodiments, the far-infrared camera and the near-infrared camera capture images at a frequency of 30 frames per second. Each first image in the first image frame group and each second image in the second image frame group are three-channel YUV images, where Y represents luminance (Luma), i.e., grayscale value, and U and V represent chrominance (Chrominance), i.e., hue and saturation. The YUV format can efficiently process and compress image data while retaining key luminance information, which conforms to the visual characteristics of the human eye.

[0103] Figure 3 This is a schematic diagram of another image processing system provided in an embodiment of this application, as shown below. Figure 3As shown, the fusion unit includes: a far-infrared target detection module, a near-infrared target detection module, and a fusion module; the far-infrared target detection module is used to perform target detection on each frame of the first image frame group to obtain a first target detection result; the near-infrared target detection module is used to perform target detection on each frame of the second image in the second image frame group to obtain a second target detection result; the fusion module is used to perform fusion processing on the first image and the second image corresponding to the first image for each frame of the first image, based on the first target detection result of the first image and the second target detection result of the second image corresponding to the first image, to obtain the target detection result of the target image.

[0104] Figure 4 This is a schematic flowchart of another image processing method provided in an embodiment of this application. Figure 4 Is Figure 2 Based on the illustrated embodiment, this describes a possible implementation of S22. The method of this embodiment is applied to, for example, Figure 3 The image processing system shown, such as Figure 4 As shown, the flow of the method in this embodiment is as follows:

[0105] S221: Perform target detection on each frame of the first image in the first image frame group to obtain the first target detection result.

[0106] One possible implementation of object detection is to obtain object detection results through an object detection model, such as YOLOv10. The YOLOv10 model employs a consistent dual-assignments strategy, combining one-to-many label assignment with one-to-one label assignment. This preserves rich supervisory information while avoiding the need for non-maximum suppression (NMS). Specifically, during the training phase, YOLOv10 assigns multiple positive samples to each real target, enabling the model to learn relevant information from more detection boxes. These positive samples provide rich supervisory signals, helping the model better optimize its classification and localization capabilities. A preset overlap threshold is set; the multiple positive samples are predicted detection boxes, and their overlap with the real detection boxes must exceed this threshold. During the inference phase, YOLOv10 assigns a unique positive sample to each target, using a consistency matching metric to ensure the consistency and accuracy of label assignment. This metric combines confidence, class probability, and intersection-union ratio (IU). The overUnion (IoU) method enables the model to more accurately evaluate the quality of each detection box. This comprehensive evaluation method reduces the generation of redundant detection boxes, thereby avoiding the need for NMS.

[0107] Specifically, Figure 5A This is a schematic diagram of the structure of a target detection model provided in an embodiment of this application, as shown below. Figure 5A As shown, the YOLOv10 model consists of a backbone network, a feature enhancement module (Mamba), a neck network, and a head. The backbone network is used to extract multi-scale features from the image, the feature enhancement module is used to perform global modeling of the multi-scale features to increase the receptive field of the features, the neck network is used to fuse features at different scales, and the head is used to output detection boxes for targets of different sizes.

[0108] It is understandable that a YOLOv10 target detection model can be pre-trained using a pre-labeled far-infrared image dataset. By inputting the first image of each frame in the first image frame group into the model, the first target detection result can be obtained, which includes, but is not limited to, the detection box, category label, and confidence score of the target contained in the far-infrared image.

[0109] The detection box represents the position and size of the target, and the parameters include the center point coordinates, width and height; the category label means that each detected target is assigned a category label to indicate the category to which it belongs, such as "person", "vehicle" and "animal", and the confidence score represents the probability that the target belongs to a certain category.

[0110] S222: Perform target detection on each frame of the second image in the second image frame group to obtain the second target detection result.

[0111] It can be understood that a YOLOv10 target detection model can be pre-trained using a pre-labeled near-infrared image dataset, and each frame of the second image in the second image frame group can be input into the model to obtain the second target detection result, which includes, but is not limited to, the target detection box, category label and confidence score contained in the near-infrared image.

[0112] S223: For each frame of the first image, based on the first target detection result of the first image and the second target detection result of the second image corresponding to the first image, perform fusion processing on the first image and the second image corresponding to the first image to obtain the target detection result of the target image.

[0113] Wherein, the first image and the corresponding second image are far-infrared and near-infrared images of the same scene captured at the same time. The first target detection result of the first image is the target detection result of the far-infrared image, and the second target detection result of the corresponding second image is the target detection result of the near-infrared image. Based on the target detection results of the far-infrared and near-infrared images, the far-infrared and near-infrared images are fused to obtain a fused target image and its target detection result. The target detection result of the target image includes, but is not limited to, the detection box, category label, and confidence score of the fused target.

[0114] In this embodiment, target detection is performed on each frame of the first image in the first image frame group to obtain a first target detection result; target detection is performed on each frame of the second image in the second image frame group to obtain a second target detection result; for each frame of the first image, based on the first target detection result of the first image and the second target detection result of the corresponding second image, the first image and the corresponding second image are fused to obtain the target detection result of the target image. Thus, the target detection results of far-infrared images and near-infrared images are fused, improving the accuracy of target detection.

[0115] Figure 6 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 6 Is Figure 4 Based on the illustrated embodiment, this describes a possible implementation of S221. The method of this embodiment is applied to, for example, Figure 3 The image processing system shown, such as Figure 6 As shown, the flow of the method in this embodiment is as follows:

[0116] S2211: Based on each frame of the first image in the first image frame group, perform feature extraction on the first image to obtain a multi-scale first feature map.

[0117] In this process, the backbone network of the YOLOv10 model extracts features from each frame of the first image in the first image frame group, and can output first feature maps of multiple scales to the neck network. Figure 5B This is a schematic diagram of another target detection model provided in an embodiment of this application, as shown below. Figure 5B As shown, in Figure 5ABased on this, the backbone network consists of a CBS (Convolution Batch Normalization SiLU) convolution module, a fast implementation of CSP Bottleneck with 2 convolutions (C2f) module, a spatial-channel decoupled downsampling (SCDown) module, a C2f compact inverted block (C2fCIB) module, a spatial pyramid pooling-fast (SPPF) module, and a partial self-attention (PSA) module.

[0118] Furthermore, the CBS convolutional module is a common network building block, consisting of convolutional layers, batch normalization (BN) layers, and a sigmoid-weighted linear unit (SiLU) activation function. It is used to extract local features and enable the model to learn more complex feature mapping relationships. A schematic diagram of the C2f module is shown below. Figure 5C As shown, through feature transformation, branching processing, and splicing strategies, features and cross-channel fused features are efficiently represented; the structural diagram of the SCDown module is shown below. Figure 5D As shown, it consists of pointwise convolution (PW Conv) and depthwise convolution (DW Conv), used to effectively reduce the spatial and channel dimensions of the feature map while minimizing feature information loss. Specifically, the pointwise convolution in the SCDown module is used to adjust the channel dimensions, and the depthwise convolution in the SCDown module is used for spatial downsampling. The structural diagram of C2fCIB is shown below. Figure 5E As shown, the bottleneck in the original C2f was replaced with a Compact Inverted Block (CIB). A schematic diagram of the CIB structure is shown below. Figure 5F As shown, a combination of depthwise convolution and pointwise convolution is used to effectively enhance the expressive power of features while reducing computational cost; the structural diagram of the SPPF module is shown below. Figure 5GAs shown, by using multiple small pooling layers for fast pooling operations and then stitching the outputs together, the model can better capture the shape, size, and position information of the target; the structural diagram of the PSA module is shown below. Figure 5H As shown, the feature map is first divided into two parts evenly by convolution. Then, only one part enters the multi-head self-attention (MHSA) module. The two parts are connected and fused by convolution, which can highlight important features without increasing the computational cost too much.

[0119] Specifically, such as Figure 5B As shown, taking the YOLOv10 model's backbone network outputting a first feature map at three scales as an example, assuming the input image size is 640×640×3, where 640×640 represents the spatial dimensions of the input image (length and width), and 3 indicates three color channels; firstly, after passing through two CBS convolutional modules, a feature map of size 160×160×128 is output. Then, after passing through three C2f modules and one CBS module, an 80×80×256 feature map is obtained. The first feature map of size 80×80×256 is then output to the neck network through six C2f modules; finally, it passes through an SCDown module. The first feature map, with a size of 40×40×512, is output as a second 40×40×512 feature map to the neck network through six C2f modules. Then, it passes through an SCDown module to obtain a 20×20×1024 feature map, which is then input into the C2fCIB module. The compact architecture design reduces the complexity of redundant stages. The SPPF module enhances global perspective information and features of different scales, and the PSA module incorporates global representation learning capabilities at low computational cost. Finally, a third 20×20×1024 feature map is output to the neck network.

[0120] S2212: Perform dual-path aggregation on the first feature map of the multi-scale, first from bottom to top and then from top to bottom, to obtain a fused first feature map of the multi-scale.

[0121] In this process, the neck network of the YOLOv10 model performs a dual-path aggregation on the multi-scale first feature map output by the backbone network, first from bottom to top and then from top to bottom, to obtain a fused multi-scale first feature map. For example... Figure 5BAs shown, the neck network adopts the structure of Path Aggregation Network (PANet), which improves the model's ability to detect targets of different sizes through multi-scale feature fusion. Specifically, PANet combines bottom-up and top-down bidirectional paths and achieves the fusion of features at different scales through upsampling, convolution operations, and feature concatenation. The C2f module combines feature fusion and path aggregation, reducing memory consumption and improving the model's speed. The neck network further optimizes computational efficiency through the combination of depthwise convolution and pointwise convolution in C2fCIB.

[0122] Specifically, such as Figure 5B As shown, based on the aforementioned example, after extracting the first feature maps at three scales from the backbone network, the 20×20×1024 feature map is upsampled to 40×40×512 and fused with the 40×40×512 feature map, which is then output through the C2fCIB module. The fused 40×40×512 feature map is then upsampled to 80×80×256 and fused with the 80×80×256 feature map. The first fused feature map at a scale of 80×80×256 is then output to the detection head through the C2f module. Finally, the fused 80×80×256 feature map is... The feature map is downsampled to 40×40×512 through convolution operations and fused with a 40×40×512 feature map output from the C2fCIB module. The C2f module then outputs a second fused feature map of 40×40×512 size to the detection head. Next, the 40×40×512 feature map output from the C2f module is downsampled to 20×20×1024 size through convolution operations and fused with a 20×20×1024 feature map output from the backbone network. The C2fCIB module then outputs a third fused feature map of 20×20×1024 size to the detection head. The feature map processed by the neck network incorporates multi-scale information, enhancing the detection capability for targets of different sizes.

[0123] S2213: Perform target detection on the fused multi-scale first feature map to obtain the first target detection result.

[0124] The multi-scale first feature map processed by the neck network is input into multiple detection heads for target detection. The detection head adopts a lightweight architecture, namely depthwise separable convolution and 1×1 convolution, which can reduce computational complexity. It mainly extracts the target's location information and category information from the first feature map at each scale, and outputs the detection box coordinates, the probability of the category to which the target belongs, and the confidence that the target object is contained in the detection box for targets of different sizes, which is the first target detection result.

[0125] Specifically, such as Figure 5B As shown, for multi-object detection, during the training phase, a one-to-many head can generate multiple predictions for each object, while a one-to-one head is used for single-object detection during the inference phase, focusing on high-precision single-object localization. The detection head can output detection results for objects of different sizes based on feature maps of different scales. For example, an 80×80×256 feature map is used to detect large objects, a 40×40×512 feature map is used to detect medium-sized objects, and a 20×20×1024 feature map is used to detect small objects.

[0126] In this embodiment, feature extraction is performed on the first image based on each frame of the first image in the first image frame group to obtain a multi-scale first feature map; the multi-scale first feature map is then aggregated using a dual-path approach, first from bottom to top and then from top to bottom, to obtain a fused multi-scale first feature map; target detection is performed on the fused multi-scale first feature map to obtain the first target detection result, thereby realizing target detection in far-infrared images.

[0127] Figure 7 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 7 Is Figure 6 Based on the illustrated embodiment, prior to S2212, a description of a possible implementation of feature enhancement is included. The method of this embodiment is applied to, for example... Figure 3 The image processing system shown, such as Figure 7 As shown, the flow of the method in this embodiment is as follows:

[0128] S7001: Perform feature enhancement processing on the first feature map of the multi-scale to obtain the first feature map of the multi-scale after feature enhancement processing.

[0129] In the YOLOv10 model, feature enhancement modules can be used to enhance the multi-scale first feature maps output by the backbone network. One possible implementation of the feature enhancement module is the Mamba module, the structure of which is illustrated in the diagram below. Figure 5IAs shown, it consists of modules such as Layer Normalization, Linear, SiLU activation function, 2D-Selective-Scan (SS2D) module, and convolutional layers with different kernel sizes. It is a deep learning architecture module based on Selective State Space Model (SSM), mainly used for efficient long sequence modeling tasks, and its input and output feature maps have consistent scale.

[0130] Specifically, the SS2D module is the core part of the Mamba module. It can scan the first feature map along four symmetrical directions (from top to bottom, from bottom to top, from left to right, and from right to left). The scanning results in each direction form a sequence. This four-directional scanning strategy ensures that each element in the feature map integrates information from all other locations in different directions, thereby creating a global receptive field without increasing linear computational complexity, providing a rich multidimensional information foundation for subsequent feature extraction.

[0131] In this embodiment, feature enhancement processing is performed on the first feature map at multiple scales to obtain the first feature map at multiple scales after feature enhancement processing, thereby helping to improve the accuracy of target detection in far-infrared images.

[0132] Figure 8 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 8 Is Figure 4 Based on the illustrated embodiment, before S221, the method further includes scaling the first image in each frame of the first image frame group. This method is applied to applications such as... Figure 3 The image processing system shown, such as Figure 8 As shown, the flow of the method in this embodiment is as follows:

[0133] S801: Perform scale transformation on each frame of the first image in the first image frame group to obtain a scale-transformed first image. The scale-transformed first image has the same resolution as the second image.

[0134] Since the resolutions of the images captured by the far-infrared camera and the near-infrared camera are inconsistent, the resolution scale of each frame of the first image needs to be scaled to match the resolution of the second image. For example, the resolution of each frame of the first image captured by the far-infrared camera is 640×512, and the resolution of each frame of the second image captured by the near-infrared camera is 1920×1080. One possible way to achieve this scale change is to calculate a transformation matrix by combining the intrinsic parameters of the far-infrared camera and the near-infrared camera, and transform the resolution of each frame of the first image from 640×512 to 1920×1080. The camera intrinsic parameters include, but are not limited to, focal length (fx, fy), optical center (cx, cy), and distortion coefficients. The focal length refers to the distance from the optical center of the lens to the focal point on the imaging medium; the optical center refers to the position of the origin of the image coordinate system in the pixel coordinate system; and the distortion coefficients are a set of parameters used to describe the optical distortion of the camera lens.

[0135] Specifically, based on the focal lengths of the two cameras, horizontal and vertical scaling factors can be calculated, allowing the pixel coordinates of the first image to be mapped to the coordinate system of the second image, as follows:

[0136]

[0137] Among them, s x s represents the horizontal scaling factor. y f represents the vertical scaling factor. x,FIR f represents the normalized focal length of the far-infrared camera along the x-axis. x,NIR f represents the normalized focal length of the near-infrared camera along the x-axis. y,FIR f represents the normalized focal length of the far-infrared camera along the y-axis. y,NIR This represents the normalized focal length of the near-infrared camera on the y-axis. Based on the optical center positions of the two cameras and the scaling factor, the translation amount of the optical center position can be calculated, aligning the optical center of the first image to the optical center of the second image, expressed as:

[0138] t x =c x,NIR -s x ·c x,FIR

[0139] t y =c y,NIR -s y ·c y,FIR

[0140] Among them, t x c represents the horizontal optical center translation. x,FIR The x-axis coordinate representing the optical center position of the far-infrared camera, c x,NIR The x-axis coordinate of the near-infrared camera's optical center position, cy,FIR The y-axis coordinate of the far-infrared camera's optical center position, c y,NIR Let y be the y-coordinate of the optical center position of the near-infrared camera. Then, combining the scaling factor and the translation amount of the optical center position, we can construct an affine transformation matrix, expressed as:

[0141]

[0142] Here, H represents the affine transformation matrix, which is used to transform the pixel coordinates of the first image to the coordinate system of the near-infrared image. Furthermore, if distortion exists in the far-infrared and near-infrared cameras, distortion correction needs to be performed on the first and second images first.

[0143] In this embodiment, by scaling each frame of the first image in the first image frame group, a scale-transformed first image is obtained. The scale-transformed first image has the same resolution as the second image, which helps to fuse far-infrared and near-infrared images.

[0144] Figure 9 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 9 Is Figure 4 Based on the illustrated embodiment, this describes a possible implementation of S222. The method of this embodiment is applied to, for example... Figure 3 The image processing system shown, such as Figure 9 As shown, the flow of the method in this embodiment is as follows:

[0145] S2221: Based on each frame of the second image in the second image frame group, perform feature extraction on the second image to obtain a multi-scale second feature map.

[0146] In this process, the YOLOv10 backbone network extracts features from each frame of the second image in the second image frame group, and can output second feature maps at multiple scales to the neck network, such as... Figure 5B As shown, the detailed steps can be found in [reference needed]. Figure 6 The detailed description of step S2211 in the embodiment will not be repeated here.

[0147] S2222: Perform dual-path aggregation on the multi-scale second feature map, first from bottom to top and then from top to bottom, to obtain a fused multi-scale second feature map.

[0148] In this process, the neck network of the YOLOv10 model performs a dual-path aggregation on the multi-scale second feature maps output by the backbone network, first from bottom to top and then from top to bottom, to obtain a fused multi-scale second feature map, such as... Figure 5B As shown, the detailed steps can be found in [reference needed]. Figure 6The detailed description of step S2212 in the embodiment will not be repeated here.

[0149] S2223: Perform target detection at each location of the fused multi-scale second feature map to obtain the second target detection result.

[0150] The multi-scale second feature map, processed by the neck network, is then input into multiple detection heads for target detection. Detailed steps can be found in [link to documentation]. Figure 6 The detailed description of step S2213 in the embodiments will not be repeated here.

[0151] In this embodiment, feature extraction is performed on the second image based on each frame of the second image frame group to obtain a multi-scale second feature map; the multi-scale second feature map is then aggregated using a dual-path approach, first from bottom to top and then from top to bottom, to obtain a fused multi-scale second feature map; target detection is performed on each position of the fused multi-scale second feature map to obtain the second target detection result, thereby achieving target detection in near-infrared images.

[0152] Figure 10 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 10 Is Figure 9 Based on the illustrated embodiment, before S2222, a description of a possible implementation of feature enhancement is included. The method of this embodiment is applied to, for example... Figure 3 The image processing system shown, such as Figure 10 As shown, the flow of the method in this embodiment is as follows:

[0153] S1001: Perform feature enhancement processing on the multi-scale second feature map respectively to obtain the multi-scale second feature map after feature enhancement processing.

[0154] In the YOLOv10 model, a feature enhancement module can be used to perform feature enhancement processing on the multi-scale second feature maps output by the backbone network. One possible implementation of this feature enhancement module is the Mamba module, such as... Figure 5I As shown, the detailed steps can be found in [reference needed]. Figure 7 The detailed description of step S7001 in the embodiment will not be repeated here.

[0155] In this embodiment, feature enhancement processing is performed on the multi-scale second feature maps to obtain multi-scale second feature maps after feature enhancement processing, thereby helping to improve the accuracy of target detection in near-infrared images.

[0156] Figure 11This is a schematic diagram of the structure of another image processing system provided in the embodiments of this application, as shown below. Figure 11 As shown, the fusion unit further includes: a far-infrared target tracking module and a near-infrared target tracking module; the far-infrared target tracking module is used to perform target tracking on the first image frame group based on the first target detection result of each first image in the first image frame group to obtain a first target tracking result; the near-infrared target tracking module is used to perform target tracking on the second image frame group based on the second target detection result of each second image in the first image frame group to obtain a second target tracking result; the fusion module is also used to perform fusion processing on the first target tracking result and the second target tracking result to obtain the target tracking result of the target image frame group.

[0157] Figure 12 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 12 Is Figure 4 Based on the illustrated embodiment, it also includes target tracking. The method of this embodiment is applied to, for example, target tracking. Figure 11 The image processing system shown, such as Figure 12 As shown, the flow of the method in this embodiment is as follows:

[0158] S224: Based on the first target detection result of each frame of the first image in the first image frame group, target tracking is performed on the first image frame group to obtain the first target tracking result.

[0159] One possible way to perform target trajectory tracking is to obtain the target tracking result through a target tracking algorithm, such as the ByteTrack algorithm. The ByteTrack algorithm utilizes the similarity between the target detection box in the image and the target tracking trajectory. While retaining high-scoring detection results, it removes the background from low-scoring detection results and discovers occluded or blurred targets, thereby reducing missed detections and improving the continuity of the trajectory.

[0160] Specifically, the ByteTrack algorithm employs the Hungarian algorithm to associate the detection boxes from the YOLOv10 model's target detection results. More specifically, it uses Kalman filtering to predict the position of each target's bounding box in the current frame and calculates the Intersection over Union (IoU) between each detection box and its corresponding predicted box in the current frame as a similarity metric. A higher IoU value indicates a greater likelihood that the detection box and the predicted box belong to the same target. Based on this similarity, the Hungarian algorithm is used to match the detection boxes with the predicted boxes of the tracking trajectory, thus obtaining the tracking result for each target. Kalman filtering is a highly efficient self-recursive filter used to predict system states through mathematical models. In target tracking tasks, it can estimate the target's state, such as position and velocity, and predict its future position. The Hungarian algorithm is a combinatorial optimization algorithm that solves task assignment problems in polynomial time, used to solve the maximum matching problem and linear assignment problem in bipartite graphs. In target tracking tasks, it can optimally match detection boxes with tracking trajectories.

[0161] It is understood that by inputting the first target detection result of each frame of the first image in the first image frame group into the ByteTrack algorithm, the first target tracking result can be obtained, which includes, but is not limited to, the target's unique identifier, bounding box, category label, confidence level, trajectory and tracking status contained in the far-infrared image.

[0162] Among them, the unique identifier of the target is used to continuously identify the same target in the video sequence; the bounding box is the position information of each target in the current frame; the category label is used to distinguish different categories of targets when tracking multiple categories; the confidence score indicates the probability that the target is correctly tracked; the trajectory is the movement trajectory of each target generated in the associated consecutive frames; the tracking status includes whether the target is in an active state, a lost state, or a removed state, which is used to manage the life cycle of the target.

[0163] S225: Based on the second target detection result of each frame of the second image in the first image frame group, target tracking is performed on the second image frame group to obtain the second target tracking result.

[0164] Specifically, by inputting the second target detection result of each frame of the second image in the first image frame group into the ByteTrack algorithm, the second target tracking result can be obtained, which includes, but is not limited to, the target's unique identifier, bounding box, category label, confidence level, trajectory, and tracking status contained in the far-infrared image.

[0165] S226: The first target tracking result and the second target tracking result are fused to obtain the target tracking result of the target image frame group.

[0166] Wherein, the first target tracking result is the target tracking result of the far-infrared image, and the second target tracking result is the target tracking result of the near-infrared image; based on the target tracking results of the far-infrared image and the near-infrared image, the far-infrared image and the near-infrared image are fused to obtain the fused target image frame group and its target tracking result. The target tracking result of the target image includes, but is not limited to, the unique identifier, bounding box, category label, confidence level, trajectory and tracking status of the fused target.

[0167] In this embodiment, target tracking is performed on the first image frame group based on the first target detection result of each first image in the first image frame group to obtain a first target tracking result; target tracking is performed on the second image frame group based on the second target detection result of each second image in the first image frame group to obtain a second target tracking result; the first target tracking result and the second target tracking result are fused to obtain the target tracking result of the target image frame group, thereby realizing the fusion of target tracking results of far-infrared images and near-infrared images and improving the accuracy of target tracking.

[0168] Figure 13 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 13 Is Figure 12 Based on the illustrated embodiment, this describes a possible implementation of S224. The method of this embodiment is applied to, for example... Figure 11 The image processing system shown, such as Figure 13 As shown, the flow of the method in this embodiment is as follows:

[0169] S2241: Perform target trajectory prediction on the first image of the previous frame in the first image frame group to obtain the first target prediction box of the first image of the current frame.

[0170] Specifically, Kalman filtering is used to predict and track the target trajectory in the first image of the previous frame in the first image frame group, thereby obtaining the first target prediction box in the first image of the current frame in the first image frame group.

[0171] For example, suppose the first image frame group contains vehicles, people and animals in the previous and current frames. The first target detection results are shown in Table 1. The first target prediction box position information of the current frame is predicted based on the positions of the detection boxes A, B and C in the previous frame in Table 1, as shown in Table 2.

[0172] Table 1 shows the detection results of the first target contained in the first image of the previous frame and the current frame.

[0173]

[0174] Table 2 shows the location information of the first target prediction bounding box in the first image of the current frame.

[0175]

[0176] S2242: Match the first target prediction box of the first image in the current frame with the first target detection result of the first image in the current frame to obtain the first target tracking result.

[0177] The Hungarian algorithm, based on Interchange of Unit (IoU), matches the predicted bounding box of the first target in the first image of the current frame with the first target detection result in the first image of the current frame to obtain the first target tracking result. Specifically, firstly, two score thresholds, T_high and T_low, are preset. For the first target detection result, the detection bounding box in each frame of the first image group is divided into high-scoring and low-scoring boxes based on the confidence score. That is, if the confidence score is higher than T_high, the detection bounding box is classified as a high-scoring box; if the confidence score is lower than T_high but higher than T_low, the detection bounding box is classified as a low-scoring box. Then, an IoU threshold is preset. First, the high-scoring bounding box in the first image of the current frame is matched with the predicted bounding box of the tracking trajectory. Second, the low-scoring bounding box in the first image of the current frame is matched with the predicted bounding box of the tracking trajectory that did not match a high-scoring bounding box in the first match. For detection bounding boxes that do not match a predicted bounding box of the tracking trajectory in the current frame but have a sufficiently high score, a new tracking trajectory is predicted for them in the next frame using Kalman filtering. For tracking trajectories that do not match a detection bounding box, they are retained for 30 frames and matched again when the target reappears.

[0178] For example, as shown in Tables 1 and 2, A1, B1, and C1 in the previous frame are all high-resolution bounding boxes; A2 and E1 in the current frame are high-resolution bounding boxes, and D1 is a low-resolution bounding box; A3 and E2 in the next frame are high-resolution bounding boxes, and D2 is a low-resolution bounding box. Kalman filtering is used to predict and track the target contained in the detection boxes A1, B1, and C1. The tracking trajectory of A1 includes the corresponding predicted bounding box A' in the current frame, the tracking trajectory of B1 includes the corresponding predicted bounding box B' in the current frame, and the tracking trajectory of C1 includes the corresponding predicted bounding box C' in the current frame image. Furthermore, with a preset IoU threshold of 0.8, the first matching involves matching the high-scoring bounding box with the predicted bounding box of the tracking trajectory. That is, if the detection box A2 is a high-scoring bounding box, and its IoU with the predicted box A' is calculated to be 0.9, then the detection box A2 matches the predicted box A', and the position of the detection box A2 is updated in the vehicle's tracking trajectory in A1. Since there is no high-scoring bounding box with a close position to match the predicted box B' in the current frame, the low-scoring bounding box is matched with the tracking trajectory predicted box that did not match the high-scoring bounding box in the first match. That is, the confidence score of the detection box D1 may decrease because the animal in the detection box B1 is severely occluded. Assuming its IoU with the predicted box B' is 0.8, the detection box D1 matches the predicted box B', and the position of the detection box D1 is updated in the animal's tracking trajectory in B1.

[0179] Furthermore, as shown in Table 1, if there is no predicted bounding box matching the high-resolution bounding box E1 in the current frame, a new tracking trajectory can be predicted for the person in E1, which is the predicted bounding box E' in the next frame, and the high-resolution bounding box E2 in the next frame can be matched with it; as shown in Table 2, if there is no detection box matching the predicted bounding box C' in the current frame, the first image frame group will be reserved for 30 frames. When a detection box with a similar position appears again in the subsequent frame, it will be matched again.

[0180] In this embodiment, by predicting the target trajectory of the first image in the previous frame of the first image frame group, a first target prediction box of the first image in the current frame is obtained; the first target prediction box of the first image in the current frame is matched with the first target detection result of the first image in the current frame to obtain the first target tracking result, thereby realizing target tracking of far-infrared images.

[0181] Figure 14 A flowchart illustrating another image processing method provided in this application embodiment is shown. Figure 14 Is Figure 12 Based on the illustrated embodiment, this describes a possible implementation of S225. The method of this embodiment is applied to, for example... Figure 11 The image processing system shown, such as Figure 14 As shown, the flow of the method in this embodiment is as follows:

[0182] S2251: Perform target trajectory prediction on the previous frame of the second image in the second image frame group to obtain the second target prediction box of the current frame of the second image.

[0183] Specifically, Kalman filtering is used to predict and track the target trajectory in the previous frame of the second image in the second image frame group, thereby obtaining the second target prediction bounding box in the current frame of the second image in the second image frame group. Detailed steps can be found in [link to documentation]. Figure 13 The detailed description of step S2241 in the embodiment will not be repeated here.

[0184] S2252: Match the second target prediction box of the second image in the current frame with the second target detection result of the second image in the current frame to obtain the second target tracking result.

[0185] Specifically, the Hungarian algorithm is used to match the second target prediction bounding box of the second image in the current frame with the second target detection result of the second image in the current frame based on IoU, to obtain the second target tracking result. Detailed steps can be found in [link to documentation]. Figure 13 The detailed description of step S2242 in the embodiment will not be repeated here.

[0186] In this embodiment, by predicting the target trajectory of the previous frame of the second image in the second image frame group, a second target prediction box for the current frame of the second image is obtained; the second target prediction box of the current frame of the second image is matched with the second target detection result of the current frame of the second image to obtain the second target tracking result. Thus, target tracking of near-infrared images is realized.

[0187] Figure 15 This is a schematic diagram of the structure of another image processing system provided in the embodiments of this application, as shown below. Figure 15 As shown, the fusion unit further includes: a far-infrared multi-frame verification module and a near-infrared multi-frame verification module; the far-infrared multi-frame verification module is used to perform multi-frame verification on the first image frame group to obtain a first multi-frame verification result, and to perform correction processing on the first image frame group based on the first multi-frame verification result; the near-infrared multi-frame verification module is used to perform multi-frame verification on the second image frame group to obtain a second multi-frame verification result, and to perform correction processing on the second image frame group based on the second multi-frame verification result.

[0188] Figure 16 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. Figure 16 Is Figure 12 Based on the illustrated embodiment, before S226, multi-frame verification is further performed on the first image frame group and the second image frame group. The method of this embodiment is applied to, for example, Figure 15 The image processing system shown, such as Figure 16As shown, the flow of the method in this embodiment is as follows:

[0189] S1601: Perform multi-frame verification on the first image frame group to obtain the first multi-frame verification result, and perform correction processing on the first image frame group based on the first multi-frame verification result.

[0190] The multi-frame verification is to ensure the fusion effect and quality of far-infrared and near-infrared images. One possible implementation is to extract feature points from multiple frames and perform feature point matching to verify the similarity between image frames, thereby obtaining the first multi-frame verification result. This may include, but is not limited to, the following two methods:

[0191] One feature point extraction and matching method is the Scale Invariant Feature Transform (SIFT) algorithm, which is used to extract feature points in an image and calculate the descriptors of these feature points. Specifically, the scale space of the image is constructed by using a Gaussian filter to obtain a Difference of Gaussians (DoG) image, and local extrema are found in it, that is, those points whose pixel values ​​at their own scale and adjacent scales are higher or lower than those of the surrounding pixels. The position, scale and orientation information of these points are extracted to form a vector, thus obtaining the descriptor of the feature point.

[0192] Another feature point extraction and matching method is the Speeded-Up Robust Features (SURF) algorithm, which is also used to detect feature points and calculate descriptors in images. Specifically, it constructs the scale space of the image through box filters, uses the determinant of the Hessian matrix to detect extreme points and obtain the sub-pixel level positions of feature points, and finally uses integral images to accelerate the calculation of feature point descriptors.

[0193] Furthermore, both the SIFT and SURF algorithms possess scale invariance and rotation invariance, thus enabling accurate identification and matching of image features even when the image undergoes scaling or rotation, or when there are changes in illumination.

[0194] The first multi-frame verification result is the similarity between two consecutive first images. Based on the first multi-frame verification result, the first images in the first image frame group are either retained or deleted. For example, for the first image frame, assuming that the SIFT algorithm is used to extract feature points and their descriptors for each first image frame, a preset Euclidean distance threshold of size 'a' is used. Then, a matcher is used to perform feature point matching on any two consecutive first images by calculating the Euclidean distance. The smaller the Euclidean distance, the higher the feature point similarity. Assuming that the Euclidean distance d < a for a pair of feature points, they are a pair of matching points. When the Euclidean distance is greater than the preset threshold, it indicates that the current feature point pair is unreliable. After the feature point matching is completed, mismatched points are eliminated. Then, the similarity between two consecutive first images is calculated by calculating the number of matched feature points. That is, the more feature points matched between two images, the higher the similarity, and the verification is qualified, and the corresponding first image frame is retained. Suppose that in the first image frame group, the first image of the current frame and the first image of the previous frame have a very low similarity, so it is marked as an abnormal frame. Then, feature point matching and similarity calculation are performed on the first image of the previous frame and the first image of the next frame. If the result is highly similar, the first image of the current frame is determined to be an abnormal frame and is removed.

[0195] S1602: Perform multi-frame verification on the second image frame group to obtain the second multi-frame verification result, and perform correction processing on the second image frame group based on the second multi-frame verification result.

[0196] The second multi-frame verification result is the similarity between two consecutive frames of the second image. Based on the second multi-frame verification result, the second image in the second image frame group is retained or deleted. For detailed steps, please refer to the detailed description of step S1601, which will not be repeated here.

[0197] Figure 17 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application, as shown below. Figure 17 As shown, the device includes: an acquisition module 1701 and a processing module 1702, wherein,

[0198] The acquisition module 1701 is used to acquire a first image frame group and a second image frame group, wherein the first image frame group and the second image frame group are acquired at the same time, the first image frame group is acquired by a far-infrared camera, and the second image frame group is acquired by a near-infrared camera.

[0199] The processing module 1702 is used to perform fusion processing on the first image frame group and the second image frame group to obtain the target image frame group.

[0200] Optionally, the processing module 1702 is specifically used to perform target detection on each frame of the first image in the first image frame group to obtain a first target detection result; to perform target detection on each frame of the second image in the second image frame group to obtain a second target detection result; and for each frame of the first image, based on the first target detection result of the first image and the second target detection result of the second image corresponding to the first image, to perform fusion processing on the first image and the second image corresponding to the first image to obtain the target detection result of the target image.

[0201] Optionally, the processing module 1702 is specifically used to extract features from the first image based on each frame of the first image in the first image frame group to obtain a multi-scale first feature map; to perform dual-path aggregation on the multi-scale first feature map from bottom to top and then from top to bottom to obtain a fused multi-scale first feature map; and to perform target detection on the fused multi-scale first feature map to obtain the first target detection result.

[0202] Optionally, the processing module 1702 is further configured to perform feature enhancement processing on the multi-scale first feature map respectively to obtain the multi-scale first feature map after feature enhancement processing.

[0203] Optionally, the acquisition module 1701 is further configured to scale each frame of the first image in the first image frame group to obtain a scale-transformed first image, wherein the scale-transformed first image has the same resolution as the second image.

[0204] Optionally, the processing module 1702 is specifically used to extract features from the second image based on each frame of the second image in the second image frame group to obtain a multi-scale second feature map; to perform dual-path aggregation on the multi-scale second feature map from bottom to top and then from top to bottom to obtain a fused multi-scale second feature map; and to perform target detection on each position of the fused multi-scale second feature map to obtain the second target detection result.

[0205] Optionally, the processing module 1702 is further configured to perform feature enhancement processing on the multi-scale second feature map respectively to obtain the multi-scale second feature map after feature enhancement processing.

[0206] Optionally, the processing module 1702 is further configured to perform target tracking on the first image frame group based on the first target detection result of each first image in the first image frame group to obtain a first target tracking result; perform target tracking on the second image frame group based on the second target detection result of each second image in the first image frame group to obtain a second target tracking result; and perform fusion processing on the first target tracking result and the second target tracking result to obtain the target tracking result of the target image frame group.

[0207] Optionally, the processing module 1702 is specifically used to predict the target trajectory of the first image in the previous frame of the first image frame group to obtain the first target prediction box of the first image in the current frame; and to match the first target prediction box of the first image in the current frame with the first target detection result of the first image in the current frame to obtain the first target tracking result.

[0208] Optionally, the processing module 1702 is specifically used to predict the target trajectory of the previous frame of the second image in the second image frame group to obtain the second target prediction box of the current frame of the second image; and to match the second target prediction box of the current frame of the second image with the second target detection result of the current frame of the second image to obtain the second target tracking result.

[0209] Optionally, the processing module 1702 is further configured to perform multi-frame verification on the first image frame group to obtain a first multi-frame verification result, and perform correction processing on the first image frame group based on the first multi-frame verification result; perform multi-frame verification on the second image frame group to obtain a second multi-frame verification result, and perform correction processing on the second image frame group based on the second multi-frame verification result.

[0210] The apparatus in this embodiment can be used to execute the solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0211] This application also provides an electronic device, including: a processor, the processor being connected to a memory, the memory storing a computer program / instructions executable on the processor, the computer program / instructions being executed by the processor to implement the steps of the image processing method described above.

[0212] This application also provides a vehicle that includes the electronic device described above, or the network device described above, or the electronic and electrical architecture described above.

[0213] This application also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the image processing method described in any of the preceding claims.

[0214] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the image processing method as described in any of the preceding claims.

[0215] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0216] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0217] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Acquire a first image frame group and a second image frame group, wherein the first image frame group and the second image frame group are acquired at the same time, the first image frame group is acquired by a far-infrared camera, and the second image frame group is acquired by a near-infrared camera. The first image frame group and the second image frame group are fused to obtain the target image frame group.

2. The method according to claim 1, characterized in that, The step of fusing the first image frame group and the second image frame group to obtain the target image frame group includes: Target detection is performed on each frame of the first image in the first image frame group to obtain the first target detection result; Target detection is performed on each frame of the second image in the second image frame group to obtain the second target detection result; For each frame of the first image, based on the first target detection result of the first image and the second target detection result of the second image corresponding to the first image, the first image and the second image corresponding to the first image are fused to obtain the target detection result of the target image.

3. The method according to claim 2, characterized in that, The step of performing target detection on each frame of the first image in the first image frame group to obtain a first target detection result includes: Based on each frame of the first image in the first image frame group, feature extraction is performed on the first image to obtain a multi-scale first feature map; The first feature map of the multi-scale is aggregated in a dual-path manner, first from bottom to top and then from top to bottom, to obtain a fused first feature map of the multi-scale. Target detection is performed on the fused multi-scale first feature map to obtain the first target detection result.

4. The method according to claim 3, characterized in that, Before performing bottom-up and then top-down dual-path aggregation on the multi-scale first feature map to obtain the fused multi-scale first feature map, the method further includes: The first feature map of the multi-scale is subjected to feature enhancement processing to obtain the first feature map of the multi-scale after feature enhancement processing.

5. The method according to claim 2, characterized in that, Before performing target detection on each frame of the first image in the first image frame group to obtain the first target detection result, the method further includes: The scale of each first image in the first image frame group is scaled to obtain a scaled first image, and the scaled first image has the same resolution as the second image.

6. The method according to claim 2, characterized in that, The step of performing target detection on each frame of the second image in the second image frame group to obtain a second target detection result includes: Based on each frame of the second image in the second image frame group, feature extraction is performed on the second image to obtain a multi-scale second feature map; The multi-scale second feature map is aggregated using a dual-path approach, first from bottom to top and then from top to bottom, to obtain a fused multi-scale second feature map. Target detection is performed at each location of the fused multi-scale second feature map to obtain the second target detection result.

7. The method according to claim 6, characterized in that, Before performing bottom-up and then top-down dual-path aggregation on the multi-scale second feature map to obtain the fused multi-scale second feature map, the method further includes: The second feature map at multiple scales is subjected to feature enhancement processing to obtain the second feature map at multiple scales after feature enhancement processing.

8. The method according to claim 2, characterized in that, Also includes: Based on the first target detection result of each frame of the first image in the first image frame group, target tracking is performed on the first image frame group to obtain the first target tracking result; Based on the second target detection result of each frame of the second image in the first image frame group, target tracking is performed on the second image frame group to obtain the second target tracking result; The first target tracking result and the second target tracking result are fused to obtain the target tracking result of the target image frame group.

9. The method according to claim 8, characterized in that, The step of performing target tracking on the first image frame group based on the first target detection result of each first image in the first image frame group to obtain the first target tracking result includes: Target trajectory prediction is performed on the first image of the previous frame in the first image frame group to obtain the first target prediction box of the first image of the current frame. The first target prediction bounding box of the first image in the current frame is matched with the first target detection result of the first image in the current frame to obtain the first target tracking result.

10. The method according to claim 8, characterized in that, The second target detection result is obtained by performing target tracking on the second image frame group based on the second target detection result of each second image in the first image frame group, including: Perform target trajectory prediction on the previous frame of the second image in the second image frame group to obtain the second target prediction box of the current frame of the second image; The second target prediction bounding box of the second image in the current frame is matched with the second target detection result of the second image in the current frame to obtain the second target tracking result.

11. The method according to claim 8, characterized in that, Before fusing the first target tracking result and the second target tracking result to obtain the target tracking result of the target image frame group, the method further includes: Perform multi-frame verification on the first image frame group to obtain the first multi-frame verification result, and perform correction processing on the first image frame group based on the first multi-frame verification result. Perform multi-frame verification on the second image frame group to obtain the second multi-frame verification result, and perform correction processing on the second image frame group based on the second multi-frame verification result.

12. An electronic device, characterized in that, include: A processor for connection to a memory storing a computer program / instructions executable on the processor, the computer program / instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1 to 11.

13. A vehicle, characterized in that, Including the electronic device as described in claim 12.

14. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program / instruction that, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the image processing method as described in any one of claims 1 to 11.

Images