Full-color image processing method and apparatus based on low-light night vision, electronic device, and readable medium

Abstract

Embodiments of the present application disclose a full-color image processing method and apparatus based on low-light night vision, an electronic device, and a readable medium. A specific implementation of the method comprises: collecting a target image information group; on the basis of target color image information, generating visual color image information; performing feature extraction on infrared image information to obtain infrared image feature information; performing brightness channel extraction on the visual color image information to obtain a brightness channel; performing feature fusion on the infrared image feature information and the brightness channel to obtain fused feature information; inputting the fused feature information into a pre-trained brightness enhancement processing model to obtain target brightness information; changing the visual color image information; and converting the changed visual color image information. The implementation can generate color image information and obtain more color details, thereby improving the quality of a generated color image, and further enhancing the effect of image processing.

Description

Black light night vision full-color image processing method and device, electronic equipment and readable medium

[0001] The present application claims priority to the invention patent application with the application number CN202411065007.1, the invention name of "Black light night vision full-color image processing method and device, electronic equipment and readable medium", which was filed on August 5, 2024, at the China Patent Bureau, and the entire content of the application is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of computer technology, specifically to a black light night vision full-color image processing method and device, electronic equipment and readable medium. BACKGROUND

[0003] Processing images taken in low-illumination scenes (scenes with light illumination levels less than 100 feet) can improve image quality and obtain relatively clear color images, making them more suitable for visual observation and subsequent image processing tasks. Currently, when processing images taken in low-illumination scenes, the commonly used methods include: using infrared imaging to obtain object outlines to provide relatively clear images taken in low-illumination scenes; using low-illumination cameras to take color images in low-illumination scenes; using thermal imagers to detect infrared radiation emitted by objects to image to improve images taken in low-illumination scenes; using multi-frame processing methods to enhance night imaging effects by synthesizing multiple frames of images to improve signal-to-noise ratio.

[0004] However, in practice, when the above methods are used to process images taken in low-illumination scenes, the following technical problems often occur:

[0005] First, using infrared imaging can only obtain object outlines in dark environments, resulting in the generation of only grayscale images and the inability to generate color image information; when using low-illumination cameras in extremely low-light conditions, the performance of low-illumination cameras decreases significantly, resulting in poor quality of the obtained color images; when using thermal imagers, only temperature distribution in the scene can be displayed, resulting in fewer color details in the generated actual scene; when the scene is a dynamic scene, using multi-frame processing methods results in blurred and artifact images, leading to poor image processing results.

[0006] Second, directly processing the taken images as color images without considering denoising the taken images, when the images contain large noise, enhancing the images containing large noise results in low accuracy of the extracted image features, leading to poor results of processing the images based on the low-accuracy image features.

[0007] The above information disclosed in this Background section is only for enhancing the understanding of the background of the present inventive concept, and therefore, it can contain information that does not form the prior art that is already known in this country to those of ordinary skill in the art. Technical problem

[0008] The summary section of the present application is used to introduce the concepts in a simple form, which will be described in detail in the specific embodiments section. The summary section of the present application is not intended to identify key or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

[0009] Some embodiments of the present application propose a black light night vision full-color image processing method, device, electronic equipment and computer readable medium to solve one or more of the technical problems mentioned in the background section. Technical solution

[0010] In a first aspect, some embodiments of the present application provide a black light night vision full-color image processing method, which comprises: in response to receiving image shooting request information, collecting a target image information group, wherein the target image information group comprises infrared image information and target color image information shot in a target lighting scene, and the target lighting scene is a scene with a light illuminance level less than a target foot; generating intuitive color image information corresponding to the target color image information according to the target color image information; performing feature extraction processing on the infrared image information to obtain infrared image feature information; performing brightness channel extraction processing on the intuitive color image information to obtain a brightness channel; performing feature fusion processing on the infrared image feature information and the brightness channel to obtain fusion feature information; inputting the fusion feature information into a pre-trained brightness enhancement processing model to obtain brightness information after brightness enhancement processing as target brightness information; performing change processing on the intuitive color image information according to the target brightness information to obtain intuitive color image information after change processing; performing conversion processing on the intuitive color image information after change processing to obtain target full-color image information as target full-color image information after conversion processing.

[0011] In a second aspect, some embodiments of the present application provide a black light night vision full-color image processing apparatus, the apparatus comprising: an acquisition unit configured to acquire a target image information set in response to receiving an image shooting request information, wherein the target image information set comprises an infrared image information and a target color image information shot under a target lighting scene, and the target lighting scene is a scene with a light illuminance level less than a target foot candle; a generation unit configured to generate a direct color image information corresponding to the target color image information according to the target color image information; a first extraction unit configured to perform feature extraction processing on the infrared image information to obtain infrared image feature information; a second extraction unit configured to perform luminance channel extraction processing on the direct color image information to obtain a luminance channel; a fusion unit configured to perform feature fusion processing on the infrared image feature information and the luminance channel to obtain fusion feature information; an enhancement unit configured to input the fusion feature information into a pre-trained luminance enhancement processing model to obtain luminance information processed by luminance enhancement as target luminance information; a changing unit configured to perform changing processing on the direct color image information according to the target luminance information to obtain changed direct color image information; and a conversion unit configured to perform conversion processing on the changed direct color image information to obtain converted changed direct color image information as a target full-color image.

[0012] In a third aspect, some embodiments of the present application provide an electronic device, comprising: one or more processors; and a storage device having one or more programs stored thereon, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation manner of the first aspect.

[0013] In a fourth aspect, some embodiments of the present application provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any implementation manner of the first aspect. Advantages

[0014] The above-mentioned various embodiments of the present application have the following beneficial effects: through the black light night vision full-color image processing method of some embodiments of the present application, color image information can be generated and more color details can be obtained, so that the quality of the generated color image can be improved, and the effect of image processing can be further improved. Specifically, the reason for being unable to generate color image information, the poor quality of the generated color image, obtaining less color details, and the poor effect of image enhancement processing is that: using infrared imaging, only the outline of the object can be obtained in a dark environment, resulting in only a grayscale image can be generated, leading to the inability to generate color image information; when in extremely low light conditions, using a low-illumination camera, the performance of the low-illumination camera decreases greatly, resulting in the quality of the obtained color image being poor; when using a thermal imager, only the temperature distribution in the scene can be displayed, resulting in less color details in the generated actual scene; when the scene is a dynamic scene, using a multi-frame processing method, resulting in the occurrence of blurred pictures and artifacts, leading to poor image processing effect.

[0015] Based on this, the black light night vision full-color image processing method of some embodiments of the present application first, in response to receiving the image shooting request information, the target image information group is collected, wherein the target image information group includes the infrared image information and the target color image information shot under the target lighting scene. The target lighting scene is a scene with a light intensity level less than the target foot. In this way, the image shot in the low light scene can be obtained. Then, according to the target color image information, intuitive color image information corresponding to the target color image information is generated. In this way, the intuitive color image information corresponding to the target color image information can be obtained, which can be used for image enhancement processing. Then, the feature extraction processing is performed on the infrared image information to obtain the infrared image feature information. In this way, the image feature of the infrared image information can be obtained. Then, the brightness channel extraction processing is performed on the intuitive color image information to obtain the brightness channel. In this way, the brightness channel of the intuitive color image information can be obtained, which can be used for feature fusion with the infrared image feature. Secondly, the infrared image feature information and the brightness channel are subjected to feature fusion processing to obtain fusion feature information. In this way, the fusion feature of the infrared image feature information and the brightness channel can be obtained. Then, the fusion feature information is input into the pre-trained brightness enhancement processing model to obtain the brightness information after brightness enhancement processing as the target brightness information. In this way, the enhanced brightness information can be obtained. Secondly, according to the target brightness information, the intuitive color image information is subjected to change processing to obtain the intuitive color image information after change processing. In this way, the intuitive color image information after brightness enhancement can be obtained. Finally, the intuitive color image information after change processing is subjected to conversion processing to obtain the intuitive color image information after conversion processing as the target full-color image. In this way, the intuitive color image information after change processing can be converted into an RGB image. Because the color image information is not generated only by the infrared image or by the thermal imager, but by the infrared image and the target color image information (RGB image) shot, the color image information containing color details can be generated. Because the target color image information is enhanced by the feature of the infrared image, the quality of the target color image information is improved. Because the processing is not performed on multiple frames of images in a dynamic scene, but on one frame of image, the case of image blur and artifacts is less, and the effect of image processing is improved. In this way, color image information can be generated and more color details can be obtained, so that the quality of the generated color image can be improved, and the effect of image processing can be further improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or exemplary description. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative effort based on these drawings.

[0017] FIG. 1 is a flowchart of the black light night vision full-color image processing method according to some embodiments of the present application;

[0018] FIG. 2 is a structural schematic diagram of the black light night vision full-color image processing apparatus according to some embodiments of the present application;

[0019] FIG. 3 is a structural schematic diagram of an electronic device suitable for implementing some embodiments of the present application. Embodiments of the Invention

[0020] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0021] It should be understood that the term "comprises" as used in the specification and the appended claims indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] It should also be understood that the term "and / or" as used herein refers to any one of the associated listed items, combinations of one or more of the associated listed items, and all possible combinations thereof.

[0023] As used in the description of the application and the appended claims, the term "if' can be interpreted as meaning "when" or "once" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once it is determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]", depending on the context.

[0024] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0025] Reference within the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within the specification are not necessarily all referring to the same embodiment, although it can. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner on one or more embodiments.

[0026] To illustrate the technical solutions provided in the present application, the following will be described in detail in combination with specific drawings and embodiments.

[0027] FIG. 1 is a flow 100 of some embodiments of the black light night vision full-color image processing method of some embodiments of the present application. The black light night vision full-color image processing method includes the following steps:

[0028] Step 101, in response to receiving image shooting request information, collecting a target image information set.

[0029] In some embodiments, in response to receiving image shooting request information, the execution subject (such as a computing device) of the black light night vision full-color image processing method can collect a target image information set. Wherein, the above-mentioned target image information set can include infrared image information and target color image information shot under a target lighting scene. The above-mentioned target lighting scene can be a scene with a light illuminance level less than a target foot. The above-mentioned target foot can be 100 feet. The above-mentioned image shooting request information can represent a request to shoot infrared image information and target color image information. The above-mentioned infrared image information can represent a shot infrared image. The above-mentioned target color image information can represent a shot RGB image. In practice, the above-mentioned execution subject can control the associated camera to shoot infrared image information and target color image information as a target image information set. The above-mentioned associated camera can be a single-lens double-sensor mode or a double-lens double-sensor mode. For example, the above-mentioned execution subject can be a process that can call the operating system to take a photo. For example, the target lighting scene can be a park at night or a dimly lit warehouse. The above-mentioned associated camera can be a starlight level camera or a starlight level sensor camera.

[0030] Step 102, generating intuitive color image information corresponding to the target color image information according to the target color image information.

[0031] In some embodiments, the execution subject can generate intuitive color image information corresponding to the target color image information according to the target color image information.

[0032] In some optional implementations of some embodiments, the execution subject can generate intuitive color image information corresponding to the target color image information according to the target color image information:

[0033] Firstly, the execution subject can perform feature extraction processing on the target color image information to obtain a feature map group. The feature map group can include feature maps of different scales. In practice, the execution subject can perform feature extraction processing on the target color image information through a plurality of convolutional layers to obtain a feature map group. Then, the scale of each feature map in the feature map group can be determined through a deep learning visualization tool. For example, the deep learning visualization tool can be TensorBoard.

[0034] Secondly, the execution subject can perform shallow feature extraction processing on the feature map group to obtain a shallow feature map group. The shallow feature maps in the shallow feature map group can one-to-one correspond to the feature maps in the feature map group. The shallow feature maps in the shallow feature map group can represent the shallow features of the feature maps in the feature map group. In practice, the execution subject can perform shallow feature extraction processing on the feature map group through a hybrid attention method.

[0035] Thirdly, the execution subject can perform sampling operations on each shallow feature map in the shallow feature map group to obtain a sampled feature map group. In practice, in response to determining that a shallow feature map is a first scale feature map, the execution subject can perform down-sampling processing twice on the shallow feature map to obtain a second scale feature map and a third scale feature map corresponding to the shallow feature map. In response to determining that a shallow feature map is a second scale feature map, the execution subject can perform down-sampling processing on the shallow feature map to obtain a third scale feature map corresponding to the shallow feature map, and perform up-sampling processing on the shallow feature map to obtain a first scale feature map corresponding to the shallow feature map. In response to determining that a shallow feature map is a third scale feature map, the execution subject can perform up-sampling processing twice on the shallow feature map to obtain a second scale feature map and a first scale feature map. The first scale can be greater than the second scale. The second scale can be greater than the third scale.

[0036] Fourthly, the sampled feature map groups are combined to obtain each interaction feature map group. Each interaction feature map group in the above-mentioned each interaction feature map group can represent each feature map with the same scale in the sampled feature map group. In practice, the execution subject can determine each feature map with the same scale in the above-mentioned sampled feature map group as an interaction feature map group.

[0037] Fifthly, each interaction feature map group is weighted to obtain each weighted interaction feature map group.

[0038] Sixthly, each weighted interaction feature map group is summed to obtain a fusion feature map group. Each fusion feature map in the above-mentioned fusion feature map group can correspond to a weighted interaction feature map in each weighted interaction feature map group. Each fusion feature map in the above-mentioned fusion feature map group can be a feature map obtained by summing each weighted interaction feature map group.

[0039] Seventhly, the fusion feature map group is subjected to deep feature extraction processing to obtain a deep feature map group. Each fusion feature map in the above-mentioned fusion feature map group can correspond to a deep feature map in the above-mentioned deep feature map group. Each deep feature map in the above-mentioned deep feature map group can represent a deep feature of each fusion feature map in the above-mentioned fusion feature map group. In practice, the execution subject can perform deep feature extraction processing on the above-mentioned fusion feature map group by a hybrid attention method.

[0040] Eighthly, the deep feature map group is fused to obtain a fused deep feature map. The above-mentioned fused deep feature map can represent the deep features of the above-mentioned deep feature map group. In practice, the execution subject can fuse each deep feature map in the above-mentioned deep feature map group by weighted summation.

[0041] Ninthly, the fused deep feature map is decoded to obtain a decoded fused deep feature map as the denoised target color image information. The above-mentioned decoded fused deep feature map can represent a restored deep feature map. In practice, the execution subject can decode the above-mentioned fused deep feature map by a decoder composed of multiple convolutional layers.

[0042] Tenthly, the color components of the denoised target color image information are normalized to obtain normalized color components. The above-mentioned normalized color components can represent color components normalized to [0, 1].

[0043] A tenth step is to perform a searching process on the normalized color component to obtain maximum color component information and minimum color component information. The maximum color component information can represent the maximum color component of the target color image information after the denoising process. The minimum color component information can represent the minimum color component of the target color image information after the denoising process. In practice, the execution subject can perform a searching process on the normalized color component by a direct traversal searching method to obtain the maximum color component information and the minimum color component information.

[0044] A twelfth step is to perform a conversion process on the maximum color component information and the minimum color component information to obtain hue information. The hue information can represent the hue value of the image. In practice, the execution subject can perform a conversion process on the maximum color component information and the minimum color component information by an HSV hue calculation formula to obtain the hue information.

[0045] A thirteenth step is to perform a modification process on the hue information to obtain target hue information. The target hue information can represent the hue information with an expanded range. In practice, the execution subject can perform an expansion process on the range of the hue information by a color scale adjustment method to obtain the target hue information.

[0046] A fourteenth step is to perform a first conversion process on the maximum color component information and the minimum color component information to obtain target saturation information. The target saturation information can represent the saturation of HSV. In practice, first, the execution subject can determine the difference between the maximum color component information and the minimum color component information as a target difference value. Then, the execution subject can perform a process on the target difference value and the maximum color component information by a normalized saturation method to obtain the target saturation information.

[0047] A fifteenth step is to perform a second conversion process on the maximum color component information to obtain target lightness information. The target lightness information can represent the lightness of HSV. In practice, first, the execution subject can normalize the maximum color component information to the range of [0, 1] as the target lightness information.

[0048] A sixteenth step is to determine the target hue information, the target saturation information, and the target lightness information as the intuitive color image information.

[0049] The technical scheme above is an inventive point of an embodiment of the present application, and solves the second technical problem: directly processing the photographed image as a color image without considering denoising the photographed image, which results in low accuracy of extracted image features and poor processing effect of the image according to the image features with low accuracy. Factors that result in low accuracy of extracted image features and poor processing effect of the image according to the image features with low accuracy are usually as follows: directly processing the photographed image as a color image without considering denoising the photographed image. If the above factors are solved, the accuracy of extracted image features can be improved, and the image enhancement effect according to the image features can be improved. To achieve this effect, the present application performs denoising processing on the image by feature extraction, feature interaction, hybrid attention, and gated fusion when processing the black light night vision full-color image. Then, the image enhancement processing is performed according to the denoised image. Thus, the noise in the image can be reduced, the accuracy of extracted image features can be further improved, and the image enhancement effect according to the extracted image features can be improved.

[0050] In step 103, feature extraction processing is performed on the infrared image information to obtain infrared image feature information.

[0051] In some embodiments, the execution subject can perform feature extraction processing on the infrared image information to obtain infrared image feature information. The infrared image feature information can represent the features of the extracted infrared image. For example, the features of the infrared image can represent the texture, lines, or contours of the infrared image.

[0052] In some optional implementations of some embodiments, the execution subject can perform feature extraction processing on the infrared image information to obtain infrared image feature information by the following steps:

[0053] First, denoising processing is performed on the infrared image information to obtain denoised infrared image information. In practice, the execution subject can perform denoising processing on the infrared image information by a median filter algorithm.

[0054] Second, edge fidelity processing is performed on the denoised infrared image information to obtain changed infrared image information. The changed infrared image information can represent the infrared image after edge fidelity processing. In practice, the execution subject can perform edge fidelity processing on the infrared image by a filter. For example, the filter can be an edge preserving filter.

[0055] Thirdly, the changed infrared image information is processed by multi-layer convolution to obtain a feature map information group. The feature map information group can represent feature maps of different scales.

[0056] Fourthly, the feature map information group is processed by feature alignment to obtain a feature map information group processed by feature alignment. In practice, the execution subject can process the feature map information group by a feature alignment algorithm. For example, the alignment algorithm can be a global alignment algorithm based on geometric transformation.

[0057] Fifthly, the feature map information group processed by feature alignment is processed by fusion to obtain infrared image features. The infrared image features can represent infrared image features obtained by fusing features of each feature map in the feature map information group.

[0058] In some optional implementations of some embodiments, the execution subject can process the changed infrared image information by multi-layer convolution to obtain a feature map information group by the following steps:

[0059] Firstly, the changed infrared image information is processed by convolution to obtain one convolution feature map information. The one convolution feature map information can represent a feature map of the infrared image information. In practice, the execution subject can process the infrared image information by a convolution layer.

[0060] Secondly, the one convolution feature map information is processed by convolution to obtain two convolution feature map information. The two convolution feature map information can represent a feature map of the one convolution feature map information. In practice, the execution subject can process the one convolution feature map information by a convolution layer.

[0061] Thirdly, the two convolution feature map information is processed by convolution to obtain three convolution feature map information. The three convolution feature map information can represent a feature map of the two convolution feature map information. In practice, the execution subject can process the two convolution feature map information by a convolution layer.

[0062] Fourthly, the one convolution feature map information, the two convolution feature map information and the three convolution feature map information are determined as a feature map information group.

[0063] It should be noted that the convolution kernels of the convolution layers used for processing the changed infrared image information, processing the one convolution feature map information and processing the two convolution feature map information are different.

[0064] In some optional implementations of some embodiments, the execution subject can perform the fusion processing on the feature map information set after the feature alignment by the following steps to obtain the infrared image feature information:

[0065] Firstly, the execution subject performs the weighting processing on the feature map information set after the feature alignment to obtain the weighted feature map information set.

[0066] Secondly, the execution subject performs the summation processing on the weighted feature map information set to obtain the infrared image feature information.

[0067] In step 104, the execution subject performs the luminance channel extraction processing on the intuitive color image information to obtain the luminance channel.

[0068] In some embodiments, the execution subject can perform the luminance channel extraction processing on the intuitive color image information to obtain the luminance channel. The luminance channel can represent the V channel luminance information in the intuitive color image information. In practice, the execution subject can perform the luminance channel extraction processing on the intuitive color image information by an extraction function. For example, the extraction function can be the cv2.split() function of OpenCV.

[0069] In step 105, the execution subject performs the feature fusion processing on the infrared image feature information and the luminance channel to obtain the fusion feature information.

[0070] In some embodiments, the execution subject can perform the feature fusion processing on the infrared image feature information and the luminance channel to obtain the fusion feature information. The fusion feature information can represent the fusion feature of the infrared image feature information and the luminance channel.

[0071] In some optional implementations of some embodiments, the execution subject can perform the feature fusion processing on the infrared image feature information and the luminance channel by the following steps to obtain the fusion feature information:

[0072] Firstly, the execution subject generates an adaptive threshold according to the infrared image feature information and the luminance channel. The adaptive threshold can represent the threshold dynamically changing according to the local feature of the image. In practice, the execution subject can generate the adaptive threshold by the Otsu method.

[0073] Secondly, the execution subject determines the target fusion ratio corresponding to the adaptive threshold according to the adaptive threshold. The target fusion ratio can be the fusion ratio of the infrared image feature information and the luminance channel. In practice, the execution subject can determine the target fusion ratio corresponding to the adaptive threshold by the threshold-based linear fusion algorithm or the threshold-based nonlinear fusion algorithm.

[0074] In the third step, the infrared image feature information and the luminance channel are fused according to the target fusion ratio to obtain fused image information.

[0075] In the fourth step, the fused image information is post-processed to obtain fused feature information. The fused feature information can represent features of the fused image information after sharpening and noise reduction processing. In practice, the execution subject can post-process the fused image information by a high-pass filtering method.

[0076] In some optional implementations of some embodiments, the execution subject can fuse the infrared image feature information and the luminance channel according to the target fusion ratio to obtain fused image information by the following steps:

[0077] In the first step, the infrared image feature information and the luminance channel are spatially aligned to obtain aligned infrared image feature information and luminance channel. The aligned infrared image feature information and luminance channel can be geometrically corrected and image-registered infrared image feature information and luminance channel. In practice, first, the execution subject can geometrically correct the infrared image feature information and the luminance channel by affine transformation or perspective transformation. Then, the infrared image feature information and the luminance channel are image-registered by feature detection, matching algorithm and transformation matrix.

[0078] In the second step, the aligned infrared image feature information and the luminance channel are weighted and summed according to the target fusion ratio to obtain fused image information.

[0079] In step 106, the fused feature information is input into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information.

[0080] In some embodiments, the execution subject can input the fused feature information into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information. The luminance enhancement processing model can be a neural network model for enhancing luminance information. The target luminance information can be enhanced luminance information obtained by mapping. The luminance enhancement processing model can be a model that takes fused feature information as input and outputs luminance information after luminance enhancement processing. For example, the luminance enhancement processing model can be a convolutional neural network (CNN).

[0081] In step 107, the intuitive color image information is changed according to the target luminance information to obtain changed intuitive color image information.

[0082] In some embodiments, the execution subject can perform a modification process on the intuitive color image information according to the target luminance information, to obtain modified intuitive color image information. The modified intuitive color image information can be luminance-enhanced intuitive color image information. In practice, the execution subject can replace the luminance information in the intuitive color image information with the target luminance information.

[0083] At step 108, the modified intuitive color image information is converted to obtain converted intuitive color image information as target full-color image information.

[0084] In some embodiments, the execution subject can perform a modification process on the intuitive color image information according to the target luminance information, to obtain modified intuitive color image information. The modified intuitive color image information can be luminance-enhanced intuitive color image information. In practice, the execution subject can replace the luminance information in the intuitive color image information with the target luminance information.

[0085] The above various embodiments of the present application have the following beneficial effects: through the black light night vision full-color image processing method of some embodiments of the present application, color image information can be generated and more color details can be obtained, thereby the quality of the generated color image can be improved to further improve the effect of image processing. Specifically, the reason for being unable to generate color image information, the poor quality of the generated color image, obtaining less color details, and poor effect of image enhancement processing is that: using infrared imaging, only the outline of the object in the dark environment can be obtained, resulting in only grayscale images can be generated, leading to the inability to generate color image information; when in extremely low light conditions, using a low-illumination camera, the performance of the low-illumination camera decreases greatly, resulting in the quality of the obtained color image being poor; when using a thermal imager, only the temperature distribution in the scene can be displayed, resulting in fewer color details in the generated actual scene; when the scene is a dynamic scene, using a multi-frame processing method, resulting in the occurrence of blurred and false images, leading to poor image processing effect. Based on this, the black light night vision full-color image processing method of some embodiments of the present application first, in response to receiving image shooting request information, target image information groups are collected, wherein the target image information groups include infrared image information and target color image information shot in a target light scene, and the target light scene is a scene with light intensity level less than a target foot. Thus, the image shot in the low light scene can be obtained. Then, according to the target color image information, intuitive color image information corresponding to the target color image information is generated. Thus, the intuitive color image information corresponding to the target color image information can be obtained, which can be used for image enhancement processing. Then, the infrared image information is subjected to feature extraction processing to obtain infrared image feature information. Thus, the image features of the infrared image information can be obtained. Then, the intuitive color image information is subjected to luminance channel extraction processing to obtain a luminance channel. Thus, the luminance channel of the intuitive color image information can be obtained, which can be used for feature fusion with the infrared image feature. Secondly, the infrared image feature information and the luminance channel are subjected to feature fusion processing to obtain fusion feature information. Thus, the fusion feature of the infrared image feature information and the luminance channel can be obtained. Then, the fusion feature information is input into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information. Thus, the enhanced luminance information can be obtained. Secondly, according to the target luminance information, the intuitive color image information is subjected to change processing to obtain changed intuitive color image information. Thus, the luminance-enhanced intuitive color image information can be obtained. Finally, the changed intuitive color image information is subjected to conversion processing to obtain converted intuitive color image information as a target full-color image. Thus, the changed intuitive color image information can be converted into an RGB image.Because the color image information is not generated only by the infrared image or by the thermal imager, but is generated by the infrared image and the color image information (RGB image) of the photographed target, the color image information containing color details can be generated. Because the color image information of the photographed target is enhanced by the characteristics of the infrared image, the quality of the color image information of the photographed target is improved. Because the processing is performed on one frame of the photographed image instead of multiple frames of the photographed image in a dynamic scene, the case of image blur and artifacts is less, and the effect of image processing is improved. Thus, the color image information can be generated, more color details can be obtained, and the quality of the generated color image can be improved, and the effect of image processing can be further improved.

[0086] With further reference to FIG. 2, as an implementation of the methods shown in the above figures, the present application provides some embodiments of a black light night vision full-color image processing method, which corresponds to the method embodiments shown in FIG. 1. The device embodiments can be applied to various electronic devices.

[0087] As shown in FIG. 2, the black light night vision full-color image processing device 200 of some embodiments includes an acquisition unit 201, a generation unit 202, a first extraction unit 203, a second extraction unit 204, a fusion unit 205, an enhancement unit 206, a change unit 207, and a conversion unit 208. The acquisition unit 201 is configured to, in response to receiving image shooting request information, acquire a target image information set, wherein the target image information set includes infrared image information and color image information of a target photographed in a target lighting scene, and the target lighting scene is a scene in which the light intensity level is less than a target foot; the generation unit 202 is configured to generate intuitive color image information corresponding to the color image information of the target according to the color image information of the target; the first extraction unit 203 is configured to perform feature extraction processing on the infrared image information to obtain infrared image feature information; the second extraction unit 204 is configured to perform luminance channel extraction processing on the intuitive color image information to obtain a luminance channel; the fusion unit 205 is configured to perform feature fusion processing on the infrared image feature information and the luminance channel to obtain fusion feature information; the enhancement unit 206 is configured to input the fusion feature information into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information; the change unit 207 is configured to perform change processing on the intuitive color image information according to the target luminance information to obtain intuitive color image information after change processing; and the conversion unit 208 is configured to perform conversion processing on the intuitive color image information after change processing to obtain target full-color image information after conversion processing as target full-color image information.

[0088] It is appreciated that the units described in the apparatus 200 correspond to the steps in the method described with reference to Figure 1. Thus, the operations, features and advantages described above in relation to the method also apply to the apparatus 200 and the units contained therein, which will not be described again.

[0089] Reference is made below to Figure 3, which illustrates a structural schematic diagram of an electronic device 300 (e.g. a computing device) suitable for implementing some embodiments of the present application. The electronic device shown in Figure 3 is merely an example and should not impose any limitation on the functionality and scope of use of embodiments of the present application.

[0090] As shown in Figure 3, the electronic device 300 can include a processing apparatus 301 (e.g. a central processor, a graphics processor, etc.) which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or loaded from a storage apparatus 308 into a random access memory (RAM) 303. Various programs and data required for operation of the electronic device 300 are also stored in the RAM 303. The processing apparatus 301, the ROM 302 and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0091] Generally, the following apparatuses can be connected to the I / O interface 305: input apparatuses 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage apparatus 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 309. The communication apparatus 309 can allow the electronic device 300 to communicate wirelessly or by wire with other devices to exchange data. Although Figure 3 shows the electronic device 300 with various apparatuses, it should be understood that it is not required to implement or have all the apparatuses shown. More or fewer apparatuses can be implemented or provided instead. Each block shown in Figure 3 can represent one apparatus or, as appropriate, multiple apparatuses.

[0092] In particular, according to some embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In some such embodiments, the computer program can be downloaded and installed from a network via the communication apparatus 309, or installed from the storage apparatus 308, or installed from the ROM 302. When the computer program is executed by the processing apparatus 301, the above-mentioned functions defined in the methods of some embodiments of the present application are performed.

[0093] Note that the computer-readable medium in some embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus, or device to function or operate. In some embodiments of the present application, the computer-readable signal medium can include a computer-readable program code in a baseband or propagated as carrier waves in a carrier portion of a carrier wave. Such a propagated computer-readable signal medium can take many forms, including but not limited to, electro-magnetic, optical, or any suitable combination thereof. Computer-readable signal medium can be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained in a computer-readable medium can be transmitted or transmitted using any suitable medium, including but not limited to, wire, cable, RF, etc., or any suitable combination of the foregoing.

[0094] In some embodiments, the client, server, or both can communicate using any known or future developed network protocols, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current or future developed network.

[0095] The computer readable medium can be included in the electronic device, or exist separately from the electronic device. The computer readable medium carries one or more programs, which when executed by the electronic device, cause the electronic device to: in response to receiving image shooting request information, collect a target image information set, wherein the target image information set includes an infrared image and a target color image, the target color image being shot in a target light scene, the target light scene being a scene in which the light intensity level is less than a target foot candle; generate intuitive color image information corresponding to the target color image information according to the target color image information; perform feature extraction processing on the infrared image to obtain infrared image feature information; perform brightness channel extraction processing on the intuitive color image information to obtain a brightness channel; perform feature fusion processing on the infrared image feature information and the brightness channel to obtain fusion feature information; input the fusion feature information into a pre-trained brightness enhancement processing model to obtain brightness information after brightness enhancement processing as target brightness information; perform change processing on the intuitive color image information according to the target brightness information to obtain changed intuitive color image information; and perform conversion processing on the changed intuitive color image information to obtain converted intuitive color image information as a target full-color image.

[0096] Computer program code for carrying out operations of some embodiments of the application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0097] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. When the program runs on a computer, the process executed by the computer enables a machine to perform specified functions in accordance with the computer program of the present application.

[0098] The units described in some embodiments of the present application can be implemented by means of software and / or hardware, and these units can be located in a processor. For example, the units can be described as: an acquisition unit, a generation unit, a first extraction unit, a second extraction unit, a fusion unit, an enhancement unit, a change unit, and a conversion unit. In some cases, the names of these units do not limit the units themselves. For example, the acquisition unit can also be described as: “in response to receiving image shooting request information, acquiring a target image information set, wherein the target image information set includes an infrared image information and a target color image information shot in a target light scene, and the target light scene is a scene with a light intensity level less than a target foot”.

[0099] The functions described above can be performed at least in part by one or more hardware logic components. For example, non-limiting examples of exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), and so forth.

[0100] The above-described embodiments are merely intended for describing the technical solutions of the present application, but not to limit the present application; even though the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A black light night vision full-color image processing method, comprising: in response to receiving image shooting request information, collecting a target image information set, wherein the target image information set comprises infrared image information and target color image information shot in a target lighting scene, and the target lighting scene is a scene with a light illuminance level less than a target foot; generating intuitive color image information corresponding to the target color image information according to the target color image information; performing feature extraction processing on the infrared image information to obtain infrared image feature information; performing luminance channel extraction processing on the intuitive color image information to obtain a luminance channel; performing feature fusion processing on the infrared image feature information and the luminance channel to obtain fusion feature information; inputting the fusion feature information into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information; performing change processing on the intuitive color image information according to the target luminance information to obtain changed intuitive color image information; performing conversion processing on the changed intuitive color image information to obtain converted intuitive color image information as a target full-color image.

2. The method of claim 1, wherein, The feature extraction processing on the infrared image information to obtain infrared image feature information comprises: performing denoising processing on the infrared image information to obtain denoised infrared image information; performing edge fidelity processing on the denoised infrared image information to obtain changed infrared image information; performing multi-layer convolution processing on the changed infrared image information to obtain a feature map information set; performing feature alignment processing on the feature map information set to obtain a feature map information set after feature alignment processing; performing fusion processing on the feature map information set after feature alignment processing to obtain infrared image feature information.

3. The method of claim 2, wherein, The multi-layer convolution processing on the changed infrared image information to obtain a feature map information set comprises: performing convolution processing on the changed infrared image information to obtain one-time convolution processing feature map information; performing convolution processing on the one-time convolution processing feature map information to obtain two-time convolution processing feature map information; performing convolution processing on the two-time convolution processing feature map information to obtain three-time convolution processing feature map information; determining the one-time convolution processing feature map information, the two-time convolution processing feature map information, and the three-time convolution processing feature map information as the feature map information set.

4. The method of claim 2, wherein, The fusion processing on the feature map information set after feature alignment processing to obtain infrared image feature information comprises: performing weighting processing on the feature map information set after feature alignment processing to obtain a weighted feature map information set; performing summation processing on the weighted feature map information set to obtain infrared image feature information.

5. The method of claim 1, wherein, The feature fusion processing on the infrared image feature information and the luminance channel to obtain fusion feature information comprises: generating an adaptive threshold according to the infrared image feature information and the luminance channel; determining a target fusion ratio corresponding to the adaptive threshold according to the adaptive threshold; According to the target fusion ratio, the infrared image feature information and the luminance channel are fused to obtain fused image information; The fused image information is post-processed to obtain fused feature information.

6. The method of claim 5, wherein, According to the target fusion ratio, the infrared image feature information and the luminance channel are fused to obtain fused image information, including: The infrared image feature information and the luminance channel are spatially aligned to obtain aligned infrared image feature information and luminance channel; According to the target fusion ratio, the aligned infrared image feature information and luminance channel are weighted and summed to obtain fused image information.

7. A method of processing black light night vision full color images, characterized in that, Including: In response to receiving image shooting request information, a target image information group is collected, the target image information group including infrared image information and target color image information shot under a target lighting scene, the target lighting scene being a scene with a light intensity level less than 100 feet; According to the target color image information, intuitive color image information corresponding to the target color image information is generated, the target color image information is converted to HSV color space after denoising processing, and the intuitive color image information is generated; The infrared image information is sequentially subjected to denoising processing, edge fidelity processing, multi-layer convolution processing, fusion processing, and convolution processing to obtain infrared image feature information, the infrared image feature information including texture, lines, and contours of the infrared image; According to the transformed infrared image feature information, the intuitive color image information is transformed to serve as a target full-color image.

8. The method of processing black light night vision full-color images according to claim 7, wherein, According to the transformed infrared image feature information, the intuitive color image information is transformed to serve as a target full-color image, including: The intuitive color image information is subjected to luminance channel extraction processing to obtain a luminance channel; The infrared image feature information and the luminance channel are subjected to feature fusion processing to obtain fused feature information; The fused feature information is input into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information; According to the target luminance information, the intuitive color image information is changed to obtain changed intuitive color image information; The changed intuitive color image information is converted to obtain converted intuitive color image information as a target full-color image.

9. The method of claim 7, wherein the black light night vision full-color image processing method is characterized by, According to the target color image information, intuitive color image information corresponding to the target color image information is generated, including: The hue information is changed to obtain target hue information, the hue information being obtained by converting maximum color component information and minimum color component information through an HSV hue calculation model; The maximum color component information and the minimum color component information are subjected to first conversion processing to obtain target saturation information; The maximum color component information is subjected to second conversion processing to obtain target luminance information; The target hue information, the target saturation information, and the target luminance information are determined as intuitive color image information.

10. The method of processing full-color images for night vision according to claim 9, wherein, The maximum color component information and the minimum color component information are subjected to first conversion processing to obtain target saturation information, including: The difference between the maximum color component information and the minimum color component information is determined as a target difference value; The target difference value and the maximum color component information are processed by a normalization saturation method to obtain target saturation information.

11. The method of processing full-color images for night vision by black light according to claim 7, wherein, In the step of generating intuitive color image information corresponding to the target color image information according to the target color image information, the maximum color component information and the minimum color component information are obtained, including: The target color image information is subjected to feature extraction processing to obtain a feature map group; The feature map group is subjected to shallow feature extraction processing to obtain a shallow feature map group; Each shallow feature map in the shallow feature map group is subjected to a sampling operation to obtain a sampled feature map group; The sampled feature map group is subjected to a combination operation to obtain each interaction feature map group; Each interaction feature map group is subjected to weighting processing to obtain each weighted interaction feature map group; The weighted interaction feature map groups are subjected to summation processing to obtain a fusion feature map group; The fusion feature map group is subjected to deep feature extraction processing to obtain a deep feature map group; The deep feature map group is subjected to fusion processing to obtain a fusion deep feature map; The fusion deep feature map is subjected to decoding processing to obtain a decoded fusion deep feature map as the denoised target color image information; The color component of the denoised target color image information is subjected to normalization processing to obtain a normalized color component; The normalized color component is subjected to lookup processing to obtain the maximum color component information and the minimum color component information.

12. The method of processing full-color images for night vision according to claim 11, wherein, The sampling operation is performed on each shallow feature map in the shallow feature map group to obtain a sampled feature map group, including: In response to determining that the shallow feature map is a first scale feature map, the shallow feature map is subjected to twice downsampling processing to obtain a second scale feature map and a third scale feature map corresponding to the shallow feature map, wherein the first scale can be greater than the second scale, and the second scale can be greater than the third scale.

13. The method of full color image processing for black light night vision of claim 8, wherein, The infrared image feature information and the luminance channel are subjected to feature fusion processing to obtain fusion feature information, including: An adaptive threshold value is generated according to the infrared image feature information and the luminance channel; A target fusion ratio corresponding to the adaptive threshold value is determined according to the adaptive threshold value; The infrared image feature information and the luminance channel are subjected to fusion processing according to the target fusion ratio to obtain fused image information; The fused image information is subjected to post-processing to obtain fusion feature information.

14. The method of processing a black light night vision full color image according to claim 13, wherein, The infrared image feature information and the luminance channel are subjected to spatial alignment processing to obtain aligned infrared image feature information and luminance channel; ​ According to the target fusion ratio, the aligned infrared image feature information and the luminance channel are weighted and summed to obtain fused image information.

15. The method of processing full-color images for night vision according to claim 14, wherein, The infrared image feature information and the luminance channel are subjected to spatial alignment processing to obtain aligned infrared image feature information and a luminance channel, including: The infrared image feature information and the luminance channel are subjected to geometric correction through affine transformation or perspective transformation. The infrared image feature information and the luminance channel are subjected to image registration through feature detection, matching algorithm and transformation matrix.

16. The method of full color image processing for black light night vision of claim 13, wherein, The adaptive threshold is generated through the following steps: The local variance of the infrared image feature information and the luminance channel is calculated. The local variance is subjected to threshold segmentation based on the Otsu algorithm to obtain an initial threshold. The adaptive threshold is dynamically adjusted according to the initial threshold and image gradient information.

17. The method of full color image processing for black light night vision of claim 8, wherein, The luminance channel of the intuitive color image information is extracted to obtain a luminance channel, including: The luminance channel of the intuitive color image information is extracted through an extraction function, wherein the luminance channel represents the V channel luminance information in the intuitive color image information.

18. The method of full color image processing for black light night vision of claim 8, wherein, The pre-trained luminance enhancement processing model is a training data set constructed by taking the fusion feature information as input and luminance information after luminance enhancement processing as output. Or The luminance enhancement processing model is an end-to-end model based on a convolutional neural network.

19. A black light night vision full-color image processing device, comprising: An acquisition unit configured to acquire a target image information set in response to receiving image shooting request information, wherein the target image information set includes infrared image information and target color image information shot under a target lighting scene, and the target lighting scene is a scene with a light intensity level less than a target foot; A generation unit configured to generate intuitive color image information corresponding to the target color image information according to the target color image information; A first extraction unit configured to perform feature extraction processing on the infrared image information to obtain infrared image feature information; A second extraction unit configured to perform luminance channel extraction processing on the intuitive color image information to obtain a luminance channel; A fusion unit configured to perform feature fusion processing on the infrared image feature information and the luminance channel to obtain fusion feature information; An enhancement unit configured to input the fusion feature information into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information; A change unit configured to perform change processing on the intuitive color image information according to the target luminance information to obtain changed intuitive color image information; A conversion unit configured to perform conversion processing on the changed intuitive color image information to obtain converted intuitive color image information as a target full-color image.

20. A black light night vision full-color image processing device, comprising: The acquisition unit is configured to acquire a target image information set in response to receiving the image shooting request information, wherein the target image information set comprises infrared image information and target color image information shot in a target lighting scene, and the target lighting scene is a scene with a light intensity level less than 100 feet; The generation unit is configured to generate intuitive color image information corresponding to the target color image information by converting the target color image information into the HSV color space after denoising processing. The first extraction unit is configured to sequentially perform denoising processing, edge fidelity processing, multi-layer convolution processing, fusion processing and convolution processing on the infrared image information to obtain infrared image feature information, wherein the infrared image feature information comprises texture, lines and contours of the infrared image. The conversion unit is configured to perform conversion processing on the intuitive color image information according to the infrared image feature information after the conversion processing, and the changed intuitive color image is used as a target full-color image.

20. The black light night vision full color image processing device of claim 19, wherein, The conversion unit comprises: The second extraction unit is configured to perform luminance channel extraction processing on the intuitive color image information to obtain a luminance channel. The fusion unit is configured to perform feature fusion processing on the infrared image feature information and the luminance channel to obtain fusion feature information. The enhancement unit is configured to input the fusion feature information into a pre-trained luminance enhancement processing model to obtain luminance information after luminance enhancement processing as target luminance information. The change unit is configured to perform change processing on the intuitive color image information according to the target luminance information to obtain changed intuitive color image information. The sub-conversion unit is configured to perform conversion processing on the changed intuitive color image information to obtain changed intuitive color image information after conversion processing as a target full-color image.

21. The black light night vision full color image processing apparatus of claim 19, wherein, The generation unit comprises: The first sub-generation unit is configured to perform change processing on hue information to obtain target hue information, wherein the hue information is obtained by converting maximum color component information and minimum color component information through an HSV hue calculation model. The second sub-generation unit is configured to perform first conversion processing on the maximum color component information and the minimum color component information to obtain target saturation information. The third sub-generation unit is configured to perform second conversion processing on the maximum color component information to obtain target luminance information. The determination unit is configured to determine the target hue information, the target saturation information and the target luminance information as intuitive color image information.

22. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-18.

23. A computer readable medium having stored thereon a computer program, wherein, The computer program is executed by the processor to implement the method of any one of claims 1-18.

Images