Data processing method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides a data processing method, device, electronic device and storage medium, the method comprising: obtaining an RGB video stream and an infrared video stream; performing time domain alignment processing on the RGB video stream and the infrared video stream to obtain multiple aligned image pairs; based on the infrared video stream, executing at least one preset task, for each of the multiple aligned image pairs, performing an image fusion operation on the aligned image pair, and obtaining all fused images associated with the aligned image pair. It can not only ensure that a more accurate task execution result is obtained, but also when the user needs to view a certain image or an image within a certain time, a fused image that takes into account both color features and infrared features within a certain time can be provided to the user, so as to meet the user's need to understand the specific situation by viewing the image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to data processing methods, devices, electronic devices and storage media. Background Art

[0002] At present, artificial intelligence technologies such as face recognition and pedestrian re-identification have been widely used in security systems, making security systems intelligent. In low-light scenes such as night scenes, compared with RGB images, the features of infrared (IR) images used to perform corresponding tasks are less affected by light. In low-light scenes, using infrared video streams including infrared images to perform corresponding tasks can obtain more accurate task execution results.

[0003] Users, such as staff using security systems, often need to view an image or images within a certain time period related to the task execution results that the user is concerned about in order to understand the specific situation. However, when using an infrared video stream including infrared images to perform corresponding tasks, only infrared images can be provided to users. Infrared images are black and white images that lack color features and are not suitable for user viewing. It is difficult to meet the user's needs to understand the specific situation by viewing the image. Summary of the invention

[0004] Embodiments of the present application provide a data processing method, device, electronic device, and storage medium.

[0005] The present application provides a data processing method, including:

[0006] Obtain an RGB video stream and an infrared video stream, wherein the RGB video stream is collected by an RGB sensor in a binocular camera, and the infrared video stream is collected by an infrared sensor in a binocular camera;

[0007] Performing time domain alignment processing on the RGB video stream and the infrared video stream to obtain a plurality of aligned image pairs, wherein the aligned image pairs include: an RGB image and an infrared image, and the acquisition time of the RGB image in the aligned image pair is the same as the acquisition time of the infrared image;

[0008] Based on the infrared video stream, at least one preset task is executed, and for each of a plurality of aligned image pairs, an image fusion operation is performed on the aligned image pair to obtain all fused images associated with the aligned image pair.

[0009] In some embodiments, the image fusion operation for the aligned image pair includes:

[0010] fusing the RGB image in the aligned image pair and the infrared image in the aligned image pair to obtain a fused image corresponding to the infrared image in the aligned image pair, and using the fused image corresponding to the infrared image in the aligned image pair as the fused image associated with the aligned image pair;

[0011] Determine an infrared image to be colored in the infrared image set associated with the aligned image pair, wherein a collection time of the infrared image to be colored is located after a collection time of the infrared image in the aligned image pair and before a collection time of a next RGB image of the RGB images in the aligned image pair;

[0012] For each infrared image to be colored, based on the infrared image in the aligned image pair and the infrared image to be colored, the optical flow information corresponding to the infrared image to be colored is calculated, and based on the optical flow information, the RGB image in the aligned image pair and the infrared image to be colored are fused to obtain a fused image corresponding to the infrared image to be colored, and the fused image corresponding to the infrared image to be colored is used as the fused image associated with the aligned image pair.

[0013] In some embodiments, determining an infrared image to be colored in the set of infrared images associated with the aligned image pair comprises:

[0014] Each infrared image in the set of infrared images associated with the aligned image pair is determined as an infrared image to be rendered.

[0015] In some embodiments, the frame rate of the RGB video stream is lower than the frame rate of the infrared video stream.

[0016] In some embodiments, each preset task is one of the following: face recognition, key point recognition, and pedestrian re-identification.

[0017] In some embodiments, the image fusion operation for the aligned image pair is performed by a fused image generation model; and before acquiring the RGB video stream and the infrared video stream, the method further includes:

[0018] Taking training samples and optical flow information associated with the training samples as inputs of a fusion image generation model to obtain a predicted fusion image output by the fusion image generation model, wherein the training samples include: an RGB image for training, an infrared image for training that is acquired at the same time as the RGB image for training, and the optical flow information associated with the training samples is obtained by performing optical flow calculation on the infrared image for training and a reference infrared image that is acquired after the time when the infrared image for training is acquired;

[0019] Calculating a loss between the predicted fused image and a target output, wherein the target output is obtained by fusing the RGB image for training and the reference infrared image using a preset image fusion algorithm;

[0020] Based on the loss, parameter values ​​of parameters of the fused image generation model are updated.

[0021] The present application provides a data processing device, including:

[0022] An acquisition unit is configured to acquire an RGB video stream and an infrared video stream, wherein the RGB video stream is acquired by an RGB sensor in the binocular camera, and the infrared video stream is acquired by an infrared sensor in the binocular camera;

[0023] An alignment unit is configured to perform time domain alignment processing on the RGB video stream and the infrared video stream to obtain a plurality of aligned image pairs, wherein the aligned image pairs include: an RGB image and an infrared image, and the acquisition time of the RGB image in the aligned image pair is the same as the acquisition time of the infrared image;

[0024] The processing unit is configured to execute at least one preset task based on the infrared video stream, and to perform an image fusion operation on each of a plurality of aligned image pairs to obtain all fused images associated with the aligned image pair.

[0025] In some embodiments, the processing unit includes:

[0026] an image fusion subunit, configured to fuse the RGB image in the aligned image pair and the infrared image in the aligned image pair to obtain a fused image corresponding to the infrared image in the aligned image pair, and use the fused image corresponding to the infrared image in the aligned image pair as a fused image associated with the aligned image pair;

[0027] Determine an infrared image to be colored in the infrared image set associated with the aligned image pair, wherein a collection time of the infrared image to be colored is located after a collection time of the infrared image in the aligned image pair and before a collection time of a next RGB image of the RGB images in the aligned image pair;

[0028] For each infrared image to be colored, the optical flow information corresponding to the infrared image to be colored is calculated based on the infrared image in the aligned image pair and the infrared image to be colored; based on the optical flow information, the RGB image in the aligned image pair and the infrared image to be colored are fused to obtain a fused image corresponding to the infrared image to be colored, and the fused image corresponding to the infrared image to be colored is used as the fused image associated with the aligned image pair.

[0029] In some embodiments, the image fusion subunit is further configured to determine each infrared image in the set of infrared images associated with the aligned image pair as an infrared image to be rendered.

[0030] In some embodiments, the frame rate of the RGB video stream is lower than the frame rate of the infrared video stream.

[0031] In some embodiments, each preset task is one of the following: face recognition, key point recognition, and pedestrian re-identification.

[0032] In some embodiments, the image fusion operation for the aligned image pair is performed by a fused image generation model; and the data processing device further comprises:

[0033] A training unit is configured to use training samples and optical flow information associated with the training samples as inputs of a fusion image generation model before acquiring an RGB video stream and an infrared video stream, so as to obtain a predicted fusion image output by the fusion image generation model, wherein the training samples include: an RGB image for training, an infrared image for training whose acquisition time is the same as that of the RGB image for training, and the optical flow information associated with the training sample is obtained by performing optical flow calculation on the infrared image for training and a reference infrared image whose acquisition time is later than that of the infrared image for training; calculating the loss between the predicted fusion image and a target output, wherein the target output is obtained by fusing the RGB image for training and the reference infrared image using a preset image fusion algorithm; and updating the parameter values ​​of the parameters of the fusion image generation model based on the loss.

[0034] The data processing method and device provided in the embodiment of the present application realizes the execution of at least one preset task based on the infrared video stream, and for each of the multiple aligned image pairs, performs an image fusion operation on the aligned image pair to obtain all fused images associated with the aligned image pair. It can ensure that a more accurate task execution result is obtained, and when the user needs to view a certain image or an image within a certain time, a fused image that takes into account both color features and infrared features of a certain image or an image within a certain time can be provided to the user. The fused image is suitable for user viewing, meeting the user's need to understand the specific situation by viewing the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] Figure 1 A flow chart showing a data processing method provided by an embodiment of the present application is shown;

[0037] Figure 2 A structural block diagram of a data processing device provided in an embodiment of the present application is shown;

[0038] Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] Figure 1 A flow chart of a data processing method provided in an embodiment of the present application is shown, and the method includes:

[0042] Step 101, obtaining an RGB video stream and an infrared video stream.

[0043] In the present application, the RGB video stream is collected by the RGB sensor in the binocular camera, and the infrared video stream is collected by the infrared sensor in the binocular camera.

[0044] The frame rate of a video stream may refer to the time interval between images captured at adjacent times in the video stream. The corresponding sensor continuously captures images at the frame rate of the corresponding video stream to form the corresponding video stream.

[0045] In some embodiments, the frame rate of the RGB video stream is lower than the frame rate of the infrared video stream.

[0046] The frame rate of the RGB video stream is lower than the frame rate of the infrared video stream so as to control the number of aligned image pairs that need to be processed by the processor used to execute the data processing method provided in the embodiment of the present application, so as to avoid the situation where the frame rate of the RGB video stream is the same as the frame rate of the infrared video stream, the number of aligned image pairs is too large, and the processor may be overloaded, thereby affecting the execution of related tasks.

[0047] For example, the frame rate of the infrared video stream is a relatively high frame rate, which can meet the needs of some tasks that require the analysis of a large number of infrared images. The frame rate of the RGB video stream is a relatively low frame rate, which can control the number of aligned image pairs that need to be processed by the processor for executing the data processing method provided in the embodiment of the present application, and avoid the situation where the frame rate of the RGB video stream and the frame rate of the infrared video stream are both relatively high frame rates, and there are too many aligned image pairs, which may cause the processor to be overloaded and affect the execution of related tasks.

[0048] Step 102: perform time domain alignment processing on the RGB video stream and the infrared video stream to obtain a plurality of aligned image pairs.

[0049] In the present application, each aligned image pair includes an RGB image from an RGB video stream and an infrared image from an infrared video stream.

[0050] For each aligned image pair, the acquisition time of the RGB image in the aligned image pair is the same as the acquisition time of the infrared image in the aligned image pair.

[0051] Step 103: execute at least one preset task based on the infrared video stream, and perform an image fusion operation on each of the multiple aligned image pairs to obtain all fused images associated with the aligned image pair.

[0052] In the present application, one or more preset tasks such as face recognition, key point recognition, pedestrian re-identification, etc. can be performed based on the infrared video stream.

[0053] In the present application, for each aligned image pair, the image fusion operation for the aligned image pair may include: fusing the RGB image in the aligned image pair and the infrared image in the aligned image pair.

[0054] For each aligned image pair, when performing an image fusion operation on the aligned image pair, the RGB image in the aligned image pair and the infrared image in the aligned image pair may be fused.

[0055] Correspondingly, all fused images associated with the aligned image pair include: a fused image obtained by fusing the RGB image in the aligned image pair and the infrared image in the aligned image pair. The fused image takes into account both color features and infrared features, and the fused image can also be called a full-color image.

[0056] The fusion of the RGB image and the infrared image can also be called coloring the infrared image using the RGB image. In the process of fusing the RGB image and the infrared image, all matching pixel pairs associated with the RGB image and the infrared image can be determined, thereby determining the correspondence between the RGB pixels in the RGB image and the infrared pixels in the infrared image.

[0057] The matching pixel pair associated with the RGB image and the infrared image includes: an RGB pixel in the RGB image and an infrared pixel corresponding to the RGB pixel in the infrared image. The two pixels in the matching pixel pair associated with the RGB image and the infrared image may correspond to the same point in the three-dimensional space.

[0058] For each infrared pixel in the infrared image, a fusion value of the infrared pixel can be calculated based on the pixel value of the infrared pixel and the pixel value of the RGB pixel corresponding to the infrared pixel in the RGB image. Fusion of the RGB image and the infrared image can be equivalent to setting the pixel value of the pixel in the infrared image to the calculated fusion value, so that the pixel value of the pixel in the infrared image is converted from the original pixel value to the fusion value of the pixel, thereby obtaining a fused image. In the fused image, the pixel value of each pixel is a fusion value.

[0059] In some embodiments, the image fusion operation for the aligned image pair includes: fusing the RGB image in the aligned image pair and the infrared image in the aligned image pair to obtain a fused image corresponding to the infrared image in the aligned image pair, and using the fused image corresponding to the infrared image in the aligned image pair as the fused image associated with the aligned image pair; determining an infrared image to be colored in the infrared image set associated with the aligned image pair, wherein the acquisition time of the infrared image to be colored is after the acquisition time of the infrared image in the aligned image pair and before the acquisition time of the next RGB image of the RGB image in the aligned image pair; for each infrared image to be colored, based on the infrared image in the aligned image pair and the infrared image to be colored, calculating the optical flow information corresponding to the infrared image to be colored, based on the optical flow information corresponding to the infrared image to be colored, fusing the RGB image in the aligned image pair and the infrared image to be colored to obtain the fused image corresponding to the infrared image to be colored, and using the fused image corresponding to the infrared image to be colored as the fused image associated with the aligned image pair.

[0060] In the present application, the infrared image set associated with the aligned image pair consists of all infrared images whose acquisition time is after the acquisition time of the infrared image in the aligned image pair and before the acquisition time of the next RGB image of the RGB image in the aligned image pair.

[0061] For example, the first RGB image in the RGB video stream and the first infrared image in the infrared video stream form the first aligned image pair, the second RGB image in the RGB video stream and the sixth infrared image in the infrared video stream form the second aligned image pair, and the third RGB image in the RGB video stream and the eleventh infrared image in the infrared video stream form the third aligned image pair. Except for the first aligned image pair, for each other aligned image pair, between the acquisition moments of the two images in the other aligned image pair, four infrared images, i.e., four infrared images, are acquired, and so on.

[0062] For the first aligned image pair, all infrared images located in the aligned image pair, i.e., after the acquisition time of the first infrared image in the infrared video stream, and before the acquisition time of the next RGB image of the RGB image in the aligned image pair, i.e., the second RGB image in the RGB video stream, include: the second infrared image, the third infrared image, the fourth infrared image, and the fifth infrared image. Therefore, for the first aligned image pair, the infrared image set associated with the first aligned image pair consists of the second infrared image, the third infrared image, the fourth infrared image, and the fifth infrared image.

[0063] For the second aligned image pair, the infrared image set associated with the second aligned image pair consists of the 7th infrared image, the 8th infrared image, the 9th infrared image, and the 10th infrared image.

[0064] In the present application, for an aligned image pair, when determining the infrared images to be colored in the infrared image set associated with the aligned image pair, a preset number of infrared images can be extracted from the infrared image set associated with the aligned image pair, and the extracted infrared images can be determined as the infrared images to be colored in the infrared image set associated with the aligned image pair. The extraction method can be random extraction, or starting from the infrared images in the aligned image pair, one infrared image can be extracted every certain number of infrared images.

[0065] In some embodiments, for each aligned image pair, determining the infrared image to be colorized in the set of infrared images associated with the aligned image pair includes: determining each infrared image in the set of infrared images associated with the aligned image pair as the infrared image to be colorized.

[0066] In the present application, for each infrared image to be colored in an infrared image set associated with an aligned image pair, optical flow information corresponding to the infrared image to be colored can be calculated based on the infrared images in the aligned image pair and the infrared image to be colored.

[0067] When calculating the optical flow information corresponding to the infrared image to be colored based on the infrared image in the aligned image pair and the infrared image to be colored, an existing optical flow algorithm can be used to perform optical flow calculation on the infrared image in the aligned image pair and the infrared image to be colored to obtain the optical flow information corresponding to the infrared image to be colored.

[0068] For example, except for the first aligned image pair, for each other aligned image pair, four infrared images, i.e., four frames of infrared images, are collected between the collection moments of two images in the other aligned image pair. It is assumed that the third frame of infrared image and the fifth frame of infrared image in the infrared image set associated with the first aligned image pair are each used as an infrared image to be colored.

[0069] For the third infrared image in the infrared image set associated with the first aligned image pair, an existing optical flow algorithm can be used to perform optical flow calculation on the infrared images in the first aligned image pair, i.e., the first infrared image and the third infrared image in the infrared video stream, to obtain the optical flow information corresponding to the third infrared image.

[0070] For the fifth infrared image in the infrared image set associated with the first aligned image pair, an existing optical flow algorithm can be used to perform optical flow calculation on the infrared images in the first aligned image pair and the fifth infrared image to obtain optical flow information corresponding to the fifth infrared image.

[0071] For each infrared image to be colored in a set of infrared images associated with an aligned image pair, the optical flow information corresponding to the infrared image to be colored can describe the displacement of pixels in the infrared image in the aligned image pair and the infrared image to be colored from the acquisition moment of the infrared image in the aligned image pair to the acquisition moment of the infrared image to be colored.

[0072] The optical flow information corresponding to the infrared image to be colored may include the optical flow of each matching pixel pair associated with the infrared image in the aligned image pair and the infrared image to be colored. The matching pixel pair associated with the infrared image in the aligned image pair and the infrared image to be colored includes: an infrared pixel in the infrared image in the aligned image pair and an infrared pixel corresponding to the infrared pixel in the infrared image to be colored. An infrared pixel in the infrared image in the aligned image pair and an infrared pixel corresponding to the infrared pixel in the infrared image to be colored may correspond to the same point in three-dimensional space.

[0073] The optical flow of the matching pixel pairs associated with the infrared image in the aligned image pair and the infrared image to be colored may be: the pixel distance between the infrared pixels in the matching pixel pairs from the infrared image in the aligned image pair and the infrared pixels in the matching pixel pairs from the infrared image to be colored.

[0074] For each infrared image to be colored associated with an aligned image pair, the RGB image in the aligned image pair and the infrared image to be colored can be fused based on the optical flow information corresponding to the infrared image to be colored to obtain a fused image corresponding to the infrared image to be colored, and the fused image corresponding to the infrared image to be colored can be used as the fused image associated with the aligned image pair.

[0075] Except for the first aligned image pair, for each other aligned image pair, four infrared images, i.e., four frames of infrared images, are collected between the collection moments of the two images in the other aligned image pair. It is assumed that the second frame of infrared image, the third frame of infrared image, the fifth frame of infrared image, and the fourth frame of infrared image in the infrared image set associated with the first aligned image pair are all infrared images to be colored.

[0076] The optical flow information corresponding to the second infrared image, the optical flow information corresponding to the third infrared image, the optical flow information corresponding to the fourth infrared image, and the optical flow information corresponding to the fifth infrared image can be calculated. The optical flow information corresponding to the second infrared image includes the optical flow of each matching pixel pair associated with the first aligned image pair, i.e., the first infrared image and the second infrared image. And so on.

[0077] Based on the optical flow information corresponding to the second infrared image, the RGB image in the first aligned image pair and the second infrared image can be fused to obtain a fused image corresponding to the second infrared image. Based on the optical flow information corresponding to the third infrared image, the RGB image in the first aligned image pair and the third infrared image can be fused to obtain a fused image corresponding to the third infrared image, and so on.

[0078] In the present application, for each infrared image to be colored associated with an aligned image pair, the position of the infrared pixel corresponding to the infrared pixel in the infrared image in the aligned image pair in the infrared image to be colored can be accurately determined through the optical flow information corresponding to the infrared image to be colored, so that the corresponding relationship between the infrared pixel in the infrared image in the aligned image pair and the infrared pixel in the infrared image to be colored can be determined. Then, the corresponding relationship between the RGB pixel in the RGB image in the aligned image pair and the infrared pixel in the infrared image to be colored can be further determined based on the corresponding relationship between the infrared pixel in the infrared image in the aligned image pair and the infrared pixel in the infrared image to be colored, and the corresponding relationship between the RGB pixel in the RGB image in the aligned image pair and the infrared pixel in the infrared image in the aligned image pair.

[0079] Thus, for each infrared pixel in the infrared image to be colored, the fusion value of the infrared pixel can be calculated based on the pixel value of the infrared pixel and the pixel value of the RGB pixel corresponding to the infrared pixel in the RGB image in the aligned image pair. Then, the pixel value of each infrared pixel in the infrared image to be colored can be set to the corresponding fusion value to obtain the fusion image corresponding to the infrared image to be colored. Thus, the infrared image to be colored is colored using the RGB image.

[0080] In some embodiments, for each aligned image pair, the image fusion operation for the aligned image pair is performed by a pre-trained fusion image generation model. Before acquiring the RGB video stream and the infrared video stream, it also includes: before acquiring the RGB video stream and the infrared video stream, taking the training sample and the optical flow information associated with the training sample as the input of the fusion image generation model to obtain the predicted fusion image output by the fusion image generation model, wherein the training sample: an RGB image for training, an infrared image for training with the same acquisition time as the RGB image for training, and the optical flow information associated with the training sample is obtained by performing optical flow calculation on the infrared image for training and a reference infrared image with an acquisition time after the acquisition time of the infrared image for training; calculating the loss between the predicted fusion image and the target output, wherein the target output is obtained by fusing the RGB image for training and the reference infrared image using a preset image fusion algorithm; based on the loss, updating the parameter values ​​of the parameters of the fusion image generation model.

[0081] The fused image generation model may be a neural network, such as a convolutional neural network. Before acquiring the RGB video stream and the infrared video stream for the first time, the fused image generation model is pre-trained using a plurality of training samples.

[0082] Each time, a training sample is used to train the fusion image generation model. The training sample includes: an RGB image for training, and an infrared image for training that is acquired at the same time as the RGB image for training. For each training sample used in training, the training sample has at least one image that other training samples do not have.

[0083] Each time the fusion image generation model is trained, the input of the fusion image generation model includes: a training sample and optical flow information associated with the training sample, and the optical flow information associated with the training sample is obtained by performing optical flow calculation on the infrared image used for training in the training sample and a reference infrared image whose acquisition time is after the acquisition time of the infrared image used for training. The optical flow information associated with the training sample can be calculated by referring to the above calculation method of performing optical flow calculation on the infrared image in the aligned image pair and the infrared image to be colored to obtain the optical flow information corresponding to the infrared image to be colored.

[0084] Each time the fusion image generation model is trained, the target output of the fusion image generation model is obtained by fusing the RGB image used in this training with the reference infrared image used in this training using a preset image fusion algorithm.

[0085] Each time the fusion image generation model is trained, the fusion image generation model outputs a predicted fusion image based on the input, calculates the loss between the predicted fusion image and the target output, and updates the parameter values ​​of the parameters in the fusion image generation model based on the loss between the predicted fusion image and the target output.

[0086] Please refer to Figure 2 , which shows a structural block diagram of a data processing device provided in an embodiment of the present application. The device comprises: an acquisition unit 201, an alignment unit 202, and a processing unit 203203.

[0087] The acquisition unit 201 is configured to acquire an RGB video stream and an infrared video stream, wherein the RGB video stream is acquired by an RGB sensor in the binocular camera, and the infrared video stream is acquired by an infrared sensor in the binocular camera;

[0088] The alignment unit 202 is configured to perform time domain alignment processing on the RGB video stream and the infrared video stream to obtain a plurality of aligned image pairs, wherein the aligned image pairs include: an RGB image and an infrared image, and the acquisition time of the RGB image in the aligned image pair is the same as the acquisition time of the infrared image;

[0089] The processing unit 203 is configured to execute at least one preset task based on the infrared video stream, and to perform an image fusion operation on each of a plurality of aligned image pairs to obtain all fused images associated with the aligned image pair.

[0090] In some embodiments, the processing unit 203 includes:

[0091] an image fusion subunit, configured to fuse the RGB image in the aligned image pair and the infrared image in the aligned image pair to obtain a fused image corresponding to the infrared image in the aligned image pair, and use the fused image corresponding to the infrared image in the aligned image pair as a fused image associated with the aligned image pair;

[0092] Determine an infrared image to be colored in the infrared image set associated with the aligned image pair, wherein a collection time of the infrared image to be colored is located after a collection time of the infrared image in the aligned image pair and before a collection time of a next RGB image of the RGB images in the aligned image pair;

[0093] For each infrared image to be colored, based on the infrared image in the aligned image pair and the infrared image to be colored, the optical flow information corresponding to the infrared image to be colored is calculated, and based on the optical flow information, the RGB image in the aligned image pair and the infrared image to be colored are fused to obtain a fused image corresponding to the infrared image to be colored, and the fused image corresponding to the infrared image to be colored is used as the fused image associated with the aligned image pair.

[0094] In some embodiments, the image fusion subunit is further configured to determine each infrared image in the set of infrared images associated with the aligned image pair as an infrared image to be rendered.

[0095] In some embodiments, the frame rate of the RGB video stream is lower than the frame rate of the infrared video stream.

[0096] In some embodiments, each preset task is one of the following: face recognition, key point recognition, and pedestrian re-identification.

[0097] In some embodiments, the image fusion operation for the aligned image pair is performed by a fused image generation model; and the data processing device further comprises:

[0098] A training unit is configured to use training samples and optical flow information associated with the training samples as inputs of a fusion image generation model before acquiring an RGB video stream and an infrared video stream, so as to obtain a predicted fusion image output by the fusion image generation model, wherein the training samples include: an RGB image for training, an infrared image for training whose acquisition time is the same as that of the RGB image for training, and the optical flow information associated with the training sample is obtained by performing optical flow calculation on the infrared image for training and a reference infrared image whose acquisition time is later than that of the infrared image for training; calculating the loss between the predicted fusion image and a target output, wherein the target output is obtained by fusing the RGB image for training and the reference infrared image using a preset image fusion algorithm; and updating the parameter values ​​of the parameters of the fusion image generation model based on the loss.

[0099] Figure 3 3 is a block diagram of a structure of an electronic device provided by this embodiment. The electronic device includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332, which is used to store instructions executable by the processing component 322, such as an application. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute instructions to perform the above method.

[0100] The electronic device may also include a power supply component 326 configured to perform power management of the electronic device, a wired or wireless network interface 350 configured to connect the electronic device to a network, and an input / output (I / O) interface 358. The electronic device may operate based on an operating system stored in the memory 332, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

[0101] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0102] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0103] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain an RGB video stream and an infrared video stream, wherein the RGB video stream is collected by an RGB sensor in a binocular camera, and the infrared video stream is collected by an infrared sensor in a binocular camera; Performing time domain alignment processing on the RGB video stream and the infrared video stream to obtain a plurality of aligned image pairs, wherein the aligned image pairs include: an RGB image and an infrared image, and the acquisition time of the RGB image in the aligned image pair is the same as the acquisition time of the infrared image; Based on the infrared video stream, at least one preset task is executed, and for each of a plurality of aligned image pairs, an image fusion operation is performed on the aligned image pair to obtain all fused images associated with the aligned image pair; All fused images associated with the aligned image pair include: a fused image corresponding to the infrared image in the aligned image pair, and a fused image corresponding to the infrared image to be colored in the set of infrared images associated with the aligned image pair; The acquisition time of the infrared image to be colored is after the acquisition time of the infrared image in the aligned image pair and before the acquisition time of the next RGB image of the RGB image in the aligned image pair; The fused image corresponding to the infrared image to be colored is obtained according to the RGB image in the aligned image pair, the infrared image to be colored, and the pixel displacement between the infrared image in the aligned image pair and the infrared image to be colored.

2. The method according to claim 1, characterized in that: The image fusion operation for the aligned image pair includes: fusing the RGB image in the aligned image pair and the infrared image in the aligned image pair to obtain a fused image corresponding to the infrared image in the aligned image pair, and using the fused image corresponding to the infrared image in the aligned image pair as the fused image associated with the aligned image pair; determining an infrared image to be colored in the set of infrared images associated with the aligned image pair; For each of the infrared images to be colored, the optical flow information corresponding to the infrared image to be colored is calculated based on the infrared image in the aligned image pair and the infrared image to be colored; based on the optical flow information, the RGB image in the aligned image pair and the infrared image to be colored are fused to obtain a fused image corresponding to the infrared image to be colored, and the fused image corresponding to the infrared image to be colored is used as the fused image associated with the aligned image pair.

3. The method according to claim 2, characterized in that Determining an infrared image to be colored in the infrared image set associated with the aligned image pair comprises: Each infrared image in the set of infrared images associated with the aligned image pair is determined as an infrared image to be rendered.

4. According to claim 1 - The method according to any one of 3, characterized in that The frame rate of the RGB video stream is lower than that of the infrared video stream.

5. The method according to any one of claims 1 to 3, characterized in that: Each preset task is one of the following: face recognition, key point recognition, and pedestrian re-identification.

6. The method according to claim 1, characterized in that The image fusion operation for the aligned image pair is performed by a fused image generation model; And before acquiring the RGB video stream and the infrared video stream, the method further includes: Taking a training sample and optical flow information associated with the training sample as inputs of a fusion image generation model to obtain a predicted fusion image output by the fusion image generation model, wherein the training sample includes: an RGB image for training, an infrared image for training that is acquired at the same time as the RGB image for training, and the optical flow information associated with the training sample is obtained by performing optical flow calculation on the infrared image for training and a reference infrared image that is acquired after the time when the infrared image for training is acquired; Calculating a loss between the predicted fused image and a target output, wherein the target output is obtained by fusing the RGB image for training and the reference infrared image using a preset image fusion algorithm; Based on the loss, parameter values ​​of parameters of the fused image generation model are updated.

7. A data processing device, characterized in that: The device comprises: An acquisition unit is configured to acquire an RGB video stream and an infrared video stream, wherein the RGB video stream is acquired by an RGB sensor in the binocular camera, and the infrared video stream is acquired by an infrared sensor in the binocular camera; An alignment unit is configured to perform time domain alignment processing on the RGB video stream and the infrared video stream to obtain a plurality of aligned image pairs, wherein the aligned image pairs include: an RGB image and an infrared image, and the acquisition time of the RGB image in the aligned image pair is the same as the acquisition time of the infrared image; a processing unit configured to execute at least one preset task based on the infrared video stream, and to perform an image fusion operation on each of a plurality of aligned image pairs to obtain all fused images associated with the aligned image pair; All fused images associated with the aligned image pair include: a fused image corresponding to the infrared image in the aligned image pair, and a fused image corresponding to the infrared image to be colored in the set of infrared images associated with the aligned image pair; The acquisition time of the infrared image to be colored is after the acquisition time of the infrared image in the aligned image pair and before the acquisition time of the next RGB image of the RGB image in the aligned image pair; The fused image corresponding to the infrared image to be colored is obtained according to the RGB image in the aligned image pair, the infrared image to be colored, and the pixel displacement between the infrared image in the aligned image pair and the infrared image to be colored.

8. The device according to claim 7, characterized in that The processing unit includes: an image fusion subunit, configured to fuse the RGB image in the aligned image pair and the infrared image in the aligned image pair to obtain a fused image corresponding to the infrared image in the aligned image pair, and use the fused image corresponding to the infrared image in the aligned image pair as a fused image associated with the aligned image pair; determining an infrared image to be colored in the set of infrared images associated with the aligned image pair; For each infrared image to be colored, based on the infrared image in the aligned image pair and the infrared image to be colored, the optical flow information corresponding to the infrared image to be colored is calculated, and based on the optical flow information, the RGB image in the aligned image pair and the infrared image to be colored are fused to obtain a fused image corresponding to the infrared image to be colored, and the fused image corresponding to the infrared image to be colored is used as the fused image associated with the aligned image pair.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6. 10 . A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to claim 1 .

Images