Image data processing system, glasses and mobile terminal

Abstract

The embodiment of the invention discloses an image data processing system, glasses and a mobile terminal. Wherein the glasses comprise a camera sensor, a camera processor and a first communication unit, the camera sensor is used for acquiring an underexposure image data set and an overexposure image data set, and the camera processor is used for carrying out original data high-frequency noise reduction processing on the overexposure image data set to obtain preliminary noise reduction image data; the first communication unit is used for sending preliminary noise reduction image data and an underexposure image data set, the mobile terminal comprises a mobile processor and a second communication unit, and the second communication unit is used for receiving the preliminary noise reduction image data and the underexposure image data set. And the mobile processor is used for performing color space low-frequency noise reduction processing on the received data to obtain a target image. Therefore, the glasses of the embodiment carry out original data high-frequency noise reduction on the original image data, and the mobile terminal further carries out color space low-frequency noise reduction, so that the image quality is remarkably improved on the premise of ensuring low power consumption of equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to an image data processing system, glasses and a mobile terminal. BACKGROUND

[0002] At present, smart glasses gradually show great potential in consumer electronics products due to their small and portable size, and become the key entry to connect the virtual world and the real world. However, due to the compact body design, the size of the sensor integrated inside is greatly compressed. The small size sensor is inherently insufficient in light capture, color restoration and detail recording capability, resulting in poor image quality, which is difficult to meet the user's expectation of clear and real visual experience. Especially in complex lighting or high-speed motion scenes, the picture quality degradation problem is particularly prominent, which seriously restricts the application depth of smart glasses in remote collaboration, immersive entertainment, precise navigation and other scenes. SUMMARY

[0003] Therefore, the embodiments of the present application provide an image data processing system, glasses and a mobile terminal, wherein the glasses can directly perform original data high-frequency noise reduction processing on the original image data, and further perform color space low-frequency noise reduction processing on the data processed by the glasses through the mobile terminal, thereby significantly improving the image quality under the premise of ensuring low power consumption of the device.

[0004] In a first aspect, the embodiments of the present application provide an image data processing system, which comprises:

[0005] The glasses comprise a camera sensor, a camera processor and a first communication unit, the camera sensor is configured to acquire original image data, the original image data comprises an underexposed image data set and an overexposed image data set, the camera processor is configured to perform original data high-frequency noise reduction processing on the overexposed image data set to obtain corresponding preliminary noise reduction image data, and the first communication unit is configured to send the preliminary noise reduction image data and the underexposed image data set.

[0006] The mobile terminal comprises a mobile processor and a second communication unit, the second communication unit is configured to receive the preliminary noise reduction image data and the underexposed image data set, and the mobile processor is configured to perform color space low-frequency noise reduction processing on the preliminary noise reduction image data and the underexposed image data set to obtain corresponding target image.

[0007] Further, the underexposed image data set comprises underexposed image data with different degrees of exposure reduction, and the overexposed image data set comprises standard exposure image data and overexposed image data with different degrees of exposure increase.

[0008] Further, the camera processor is further configured to perform multi-frame noise reduction processing on the overexposed image data set.

[0009] Further, the camera sensor is further configured to acquire image regulation parameters, and the camera processor is further configured to determine to-be-acquired image parameters and frame numbers according to the image regulation parameters, and control the camera sensor to acquire corresponding raw image data according to the to-be-acquired image parameters and frame numbers, the image regulation parameters including exposure parameters, brightness distribution histograms, and / or active disturbance rejection control parameters.

[0010] Further, the camera processor is further configured to respectively send the preliminary noise reduction image data and the underexposed image data set to respective corresponding processing nodes in an image signal processing flow for image signal processing, to obtain processed preliminary noise reduction image data and underexposed image data set.

[0011] Further, the processing nodes include image preprocessing nodes, image processing engine nodes, and / or image encoding nodes, and the processing node corresponding to the underexposed image data set is the image preprocessing node.

[0012] Further, the mobile processor is further configured to perform high dynamic range fusion on the preliminary noise reduction image data and the underexposed image data set to obtain corresponding high dynamic image data, and perform color space low-frequency noise reduction processing on the high dynamic image data to obtain a target image.

[0013] Further, the mobile processor is further configured to perform tone mapping on the high dynamic image data before performing color space low-frequency noise reduction processing.

[0014] In a second aspect, an embodiment of the present application provides a pair of glasses, the pair of glasses comprising:

[0015] a camera sensor configured to acquire raw image data, the raw image data including an underexposed image data set and an overexposed image data set;

[0016] a camera processor configured to perform raw data high-frequency noise reduction processing on the overexposed image data set to obtain corresponding preliminary noise reduction image data;

[0017] a first communication unit configured to send the preliminary noise reduction image data and the underexposed image data set to a mobile terminal, so that the mobile terminal performs color space low-frequency noise reduction processing on the preliminary noise reduction image data and the underexposed image data set to obtain a target image.

[0018] In a third aspect, an embodiment of the present application provides a mobile terminal, the mobile terminal comprising:

[0019] a second communication unit configured to receive the preliminary denoised image data and the underexposed image data set from the glasses;

[0020] a mobile processor configured to perform color space low-frequency denoising processing on the preliminary denoised image data and the underexposed image data set to obtain a corresponding target image.

[0021] In the embodiment of the present application, the glasses include a camera sensor, a camera processor and a first communication unit, the camera sensor is configured to acquire raw image data, the raw image data includes an underexposed image data set and an overexposed image data set, the camera processor is configured to perform raw data high-frequency denoising processing on the overexposed image data set to obtain corresponding preliminary denoised image data, and the first communication unit is configured to send the preliminary denoised image data and the underexposed image data set. The mobile terminal includes a mobile processor and a second communication unit, the second communication unit is configured to receive the preliminary denoised image data and the underexposed image data set, and the mobile processor is configured to perform color space low-frequency denoising processing on the preliminary denoised image data and the underexposed image data set to obtain a corresponding target image. Thus, the glasses of the embodiment can perform raw data high-frequency denoising processing on the raw image data, and the mobile terminal can further perform color space low-frequency denoising processing on the data processed by the glasses, which significantly improves the image quality on the premise of ensuring low power consumption of the equipment. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 a schematic diagram of an image data processing system of an embodiment of the present application;

[0024] Figure 2 a flowchart of a raw image data acquisition method of an embodiment of the present application;

[0025] Figure 3 a schematic diagram of an image signal processing flow of an embodiment of the present application;

[0026] Figure 4 a flowchart of an image data processing method of an embodiment of the present application;

[0027] Figure 5 a data flow diagram of an image data processing system of an embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application is described in detail below based on examples, but the present application is not limited to only these examples. In the following detailed description of the present application, some specific details are described in detail. The present application can also be fully understood without the description of these details by those skilled in the art. In order to avoid confusion of the essence of the present application, well-known methods, processes, procedures, elements and circuits are not described in detail.

[0029] In addition, those skilled in the art should understand that the drawings provided herein are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0030] Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to".

[0031] In the description of the present application, it should be understood that the terms "first", "second", etc. are only configured for the purpose of description, and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "multiple" is two or more.

[0032] The solutions described in the specification and examples, such as those involving personal information processing, will be processed on the premise of legal basis (for example, with the consent of the subject of personal information, or necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users who refuse to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions.

[0033] Photographing and video recording are common use scenarios of smart glasses, in which the smart glasses can usually be used in cooperation with a mobile terminal or a cloud to overcome the storage shortage and computing power limitation caused by the lightweight hardware design of the smart glasses. The smart glasses can perform data communication with the mobile terminal or the cloud through Bluetooth, WIFI or other communication manners. In related comparative examples, the image data obtained by the smart glasses can be enhanced in image quality through a single RAW domain image quality algorithm or a single YUV domain image quality algorithm, and the execution subject of the algorithm can be the smart glasses or the mobile terminal. If the algorithm is executed by the smart glasses alone, the smart glasses need to complete a large amount of algorithm operation, which leads to a substantial increase in power consumption. At the same time, due to the lightweight design of the smart glasses, the computing power of the smart glasses has a great restriction on the degree of image quality improvement, and the effect of image quality improvement is limited. If the algorithm is executed by the mobile terminal or the cloud alone, a large amount of image data needs to be transmitted from the smart glasses, which will cause obvious transmission delay. Therefore, the embodiment of the present application provides an image data processing system, glasses and a mobile terminal, wherein the smart glasses perform original data high-frequency noise reduction processing on original image data, and the mobile terminal further performs color space low-frequency noise reduction processing on the data processed by the glasses, and the two devices are cooperatively processed to significantly improve the image quality under the premise of low power consumption of the devices.

[0034] Figure 1 FIG. 1 is a schematic diagram of an image data processing system according to an embodiment of the present application. As shown in FIG. 1, the image data processing system according to the embodiment of the present application includes glasses 10 and a mobile terminal 20. The mobile terminal 20 can be a mobile phone, a tablet computer, a smart watch or other mobile terminal, and the embodiment of the present application does not limit the device type of the mobile terminal 20. Figure 1

[0035] The glasses 10 include a camera sensor 11, a camera processor 12 and a first communication unit 13.

[0036] The camera sensor 11 is configured to obtain original image data.

[0037] In a possible implementation, the original image data can be image data corresponding to an image (such as a photo) or image data corresponding to a video frame in a video. The camera sensor 11 can be controlled by a corresponding photographing instruction to take a photograph to obtain the original image data, or be controlled by a corresponding video recording instruction to record a video to obtain the original image data.

[0038] ​In an alternative implementation, the shooting instruction can be triggered by a gesture operation (e.g. clicking or multiple consecutive clicking within a predetermined range of the glasses 10, sliding a predetermined gesture within a predetermined range of the glasses 10, etc.), by voice, or by the mobile terminal 20 in communication connection with the glasses 10. It should be understood that the present embodiment does not limit how the camera sensor 11 in the glasses 10 is triggered to perform a shooting action, which can be set based on actual needs.

[0039] In the present embodiment, the raw image data obtained by the camera sensor 11 is RAW domain data, which refers to raw data obtained directly from the camera sensor without image processing or compression, and is different from formats such as JPEG which have undergone certain image processing and compression. The RAW file saves all the information captured by the camera sensor, providing maximum flexibility for subsequent image processing. Correspondingly, the camera sensor 11 at least includes an image sensor capable of capturing raw light signals, such as a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD) image sensor. And the above image sensor needs to support uncompressed raw data output, which is usually realized by a Bayer filter to record a single pixel with a single color component.

[0040] In a possible implementation, the raw image data includes an underexposed image data set and an overexposed image data set. The underexposed image data set and the overexposed image data set are two groups of image data obtained by grouping the RAW domain data according to the image exposure degree. Optionally, the underexposed image data set includes underexposed image data with different degrees of exposure reduction, and the overexposed image data set includes standard exposure image data and overexposed image data with different degrees of exposure increase.

[0041] Specifically, an exposure value (EV) can be used to quantify the exposure degree of an image, and the EV is related to the influence of different exposure parameters on the amount of light in powers of 2. EV0 (standard exposure) is the reference value, and on the basis of the reference value, the exposure adjustment amount relative to EV0 can be represented by symbols +, - or specific numerical values. Taking symbols + and - as an example, the EV corresponding to different exposure degrees is illustrated, and the symbols + and - represent the relative adjustment amount based on EV0. + represents full exposure compensation, and - represents underexposure compensation. The underexposure image data corresponding to different degrees of exposure reduction can include EV-, EV--, EV---, and the like. One - symbol indicates that the exposure amount is halved once, the exposure amount of EV- is equal to one half of EV0, the exposure amount of EV-- is equal to one fourth of EV0, and the exposure amount of EV--- is equal to one eighth of EV0. The overexposure image data corresponding to different degrees of exposure increase can include EV+, EV++, EV+++ and the like. One + symbol indicates that the exposure amount is doubled, the exposure amount of EV+ is equal to twice of EV0, the exposure amount of EV++ is equal to four times of EV0, and the exposure amount of EV+++ is equal to eight times of EV0.

[0042] Further, the camera processor 12 is configured to perform RAW domain noise reduction (RAW NR) on the overexposure image data set to obtain corresponding preliminary noise-reduced image data.

[0043] The RAW NR is a technology of directly reducing noise on the original light signal captured by the sensor before the RAW data is processed in the machine (such as demosaicing, white balance, color space conversion), which can retain image details while suppressing sensor noise (such as shot noise, readout noise, fixed pattern noise, etc.), and provide cleaner original data for subsequent processing (such as HDR synthesis, color restoration, etc.).

[0044] In a possible implementation, to enable RAW NR to reduce noise for high-frequency noise, a noise reduction algorithm for high-frequency noise can be used. For example, the noise reduction algorithm of RAW NR can use an artificial intelligence-based noise reduction algorithm (AINR), a wavelet transform-based noise reduction algorithm (Wavelet NR), and / or a discrete cosine transform-based noise reduction algorithm (DCT NR), etc. The AI NR uses a deep learning model (such as a convolutional neural network CNN, a generative adversarial network GAN) to learn the noise distribution and achieve end-to-end noise reduction. It is suitable for complex noise scenarios and can adapt to different lighting conditions and sensor characteristics. The Wavelet NR decomposes the image into multi-scale subbands through wavelet transform, suppresses noise in the high-frequency subband, and preserves the details of the low-frequency subband. It is suitable for local noise suppression and can balance noise reduction and detail preservation. The DCT NR converts the image from the spatial domain to the frequency domain through DCT, suppresses noise by thresholding high-frequency components. It is computationally efficient but may blur details, and is often used for preprocessing before JPEG compression. The above noise reduction algorithms can effectively suppress noise while preserving image details.

[0045] In a possible implementation, to enable RAW NR to reduce noise for high-frequency noise, the RAW NR corresponding module can also be trained for high-frequency noise reduction before the glasses 10 are put into use, to improve the high-frequency noise reduction capability of the RAW NR corresponding module. For example, a deep learning model suitable for RAW domain noise reduction is selected as the basis, such as a convolutional neural network (CNN), U-Net, or a generative adversarial network (GAN). The model structure is customized according to the requirements of high-frequency noise reduction. For example, convolutional layers or attention mechanisms for high-frequency features can be added to enhance the model's sensitivity to high-frequency noise. While designing the model, a RAW image dataset containing high-frequency noise is collected and standardized, and the high-frequency noise region and clean region are distinguished. Then the labeled dataset is used to train the model to obtain a trained high-frequency noise reduction model. The trained high-frequency noise reduction model can be used to complete the high-frequency noise reduction processing of the original data.

[0046] The high-frequency noise reduction processing of the original data removes high-frequency noise within a single frame, and the preliminary noise reduction image data can be obtained after the high-frequency noise reduction processing of the original data.

[0047] In a possible implementation, the camera processor 12 is also configured to perform multi-frame noise reduction (MFNR) on the overexposed image data set. MFNR is a technology that reduces image noise by fusing multiple frames of image information. MFNR plays an important role in improving signal-to-noise ratio, preserving image details, adapting to complex lighting conditions, reducing motion artifacts, and improving computational photography effects by fusing multiple frames of image information.

[0048] Specifically, by fusing multiple frames of images, MFNR can increase the number of effective signals, while noise, due to randomness, will be averaged out during the multi-frame fusion process, thereby significantly improving the signal-to-noise ratio of the image. Compared with single-frame noise reduction, MFNR can better preserve image details and textures while reducing noise. This is because multi-frame fusion can provide more information to distinguish between signal and noise, thereby avoiding the loss of details caused by excessive noise reduction. MFNR is particularly suitable for image noise reduction under complex lighting conditions, such as low-light environments or high dynamic range scenes. In these cases, a single frame of image often has difficulty in capturing details in both bright and dark areas, while MFNR can cover a wider range of brightness by fusing multiple frames of images with different exposures, thereby improving the overall image quality. When shooting moving objects, a single frame of image may be blurred or have artifacts due to motion. MFNR can reduce the impact of motion on image quality by quickly capturing multiple frames of images and fusing them, generating clearer images. In the field of computational photography, MFNR is one of the key technologies for improving image quality. It can be combined with other technologies such as HDR and super-resolution to further improve the dynamic range, resolution, and detail performance of images.

[0049] Optionally, the camera processor 12 first performs MFNR and then performs RAW NR. Such a process design has significant advantages in terms of the level of noise suppression, computational efficiency and resource optimization, image quality improvement, and adaptation to different scene requirements. This process can fully utilize the complementarity of multi-frame data while preserving more details and generating higher quality images.

[0050] It should be noted that, considering the signal-to-noise ratio, dynamic range requirement, calculation efficiency and post-processing flexibility and other factors, MFNR and RAW NR are only processed for overexposed image data sets. Specifically, from the perspective of signal-to-noise ratio, positive exposure compensation increases the number of photons received by the sensor by increasing the exposure time or increasing the ISO. The increase in the number of photons directly improves the signal-to-noise ratio, so the RAW data of the positive exposure compensation has a lower noise level, and is more suitable for MFNR and RAW NR to further suppress the remaining noise. Negative exposure compensation reduces the number of photons by reducing the exposure time or reducing the ISO, resulting in a decrease in signal-to-noise ratio. At this time, the noise in the RAW data (especially the shot noise) increases significantly, and even if the noise reduction process is performed, it may lose details or introduce artifacts due to the weak signal. From the perspective of dynamic range coverage requirements, in the HDR (High Dynamic Range) scene, EV0 and EV+ are usually used to capture the medium-high brightness area of the scene (such as the sky and bright objects). The signal strength of these areas is high, and more details can be retained after noise reduction, and the contribution to the overall dynamic range is greater. EV- and EV-- are mainly used to capture extremely dark areas (such as shadows and dark details). The signal of these areas is weak, and if multi-frame noise reduction is forced, it may not be able to effectively recover the details due to the dominance of noise. In addition, dark details are usually presented through post-processing brightening, rather than relying on RAW domain noise reduction. From the perspective of calculation efficiency, MFNR requires operations such as alignment and weighted fusion of multiple RAW data, which requires a large amount of calculation. If this operation is performed on all exposure levels (including EV-, EV-- and EV---), it will significantly increase the processing delay and power consumption. Limited system resources, preferential processing of EV0 and EV+ frames with high signal-to-noise ratio and great impact on the final image quality, can more efficiently improve the overall imaging effect. From the perspective of post-processing flexibility, in the HDR synthesis or tone mapping stage, dark areas will be enhanced in visibility through brightness adjustment. At this time, if the RAW domain has been excessively denoised, it may cause the details to be blurred after brightening, and retaining some noise can restore some details through the algorithm.

[0051] In one possible implementation, before the camera sensor 11 acquires the original image data, the camera processor 12 can cooperate with the camera sensor 11 to first determine the image regulation parameters, and then acquire the corresponding original image data according to the image regulation parameters. This is an active and intelligent exposure control strategy, and the core purpose is to improve the final image quality through algorithm optimization, which is particularly effective in complex lighting or high dynamic range (HDR) scenes.

[0052] Figure 2 The flowchart of the original image data acquisition method of the embodiment of the application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the original image data acquisition method comprises the following steps:

[0053] Step S21: Obtain image regulation parameters.

[0054] In this process, the camera sensor 11 is configured to obtain image regulation parameters and send the image regulation parameters to the camera processor 12. The image regulation parameters include exposure parameters, brightness distribution histograms, and / or active disturbance rejection control parameters. Among them, the exposure parameters can include exposure time, aperture size, ISO sensitivity, etc. To obtain the exposure parameters, the camera sensor 11 includes a light sensor. The brightness distribution histogram is generated by analyzing the number of pixels at different brightness levels in the image. To obtain the brightness distribution histogram, the camera sensor 11 includes an image sensor. The active disturbance rejection control (ADRC) parameters include disturbance estimation, frequency analysis. To obtain the ADRC parameters, the camera sensor 11 includes sensors such as gyroscopes and accelerometers. The above-mentioned camera sensors collectively provide accurate ambient light measurement, image data analysis, and motion state monitoring for the camera, thereby ensuring that the camera can generate high-quality images.

[0055] In one possible implementation, the camera sensor 11 can be controlled to obtain corresponding parameters according to the parameter acquisition instruction. The parameter acquisition instruction can be triggered by gestures, buttons, voice, etc. on the glasses 10. In addition, the parameter acquisition instruction can also be a camera start-up instruction or a predetermined shooting instruction accompanying instruction, that is, the parameter acquisition instruction is automatically triggered after the camera is started up, or the parameter acquisition instruction is automatically triggered after receiving the predetermined shooting instruction, etc. It should be understood that the present embodiment does not limit how to trigger the camera sensor 11 in the glasses 10 to perform the parameter acquisition action, which can be set based on actual needs.

[0056] Step S22: Send the image regulation parameters.

[0057] In the present embodiment, the camera sensor 11 sends the image regulation parameters to the camera processor 12.

[0058] Step S23: Determine the image parameters and frame number to be obtained according to the image regulation parameters.

[0059] In this process, the camera processor 12 is configured to determine the image parameters and frame number to be obtained according to the image regulation parameters. For example, n frames of EV0 / EV+, one frame of EV-, and one frame of EV--. It should be understood that the present embodiment does not limit the specific image parameters and frame number, which can be set based on actual needs.

[0060] Specifically, to realize high dynamic range (HDR) or multi-frame noise reduction (MFNR), the camera needs to capture multiple frames of images with different exposures. The base exposure time can be fine-tuned according to the brightness distribution histogram and exposure parameters to generate a series of image parameters with different exposures, i.e. EV0, EV0+, EV0-, and so on. The active disturbance control parameters can help the camera adjust the exposure parameters in real time during shooting to cope with changes in ambient light or camera motion. For example, if the camera detects that the ambient light suddenly becomes darker, it can automatically increase the exposure time to maintain image brightness; if it detects that the camera is shaking, it can adjust the exposure parameters to reduce motion blur. The number of frames is usually determined according to the needs of the scene and the performance of the camera. For example, in HDR shooting, 3 to 5 frames of images with different exposures may be needed to synthesize a high dynamic range image, and in MFNR, the number of frames may be higher to provide sufficient noise suppression effect. The number of frames can also be dynamically adjusted according to the brightness distribution histogram and the active disturbance control parameters. For example, if the histogram shows that the image brightness distribution is very uneven, the number of frames may need to be increased to better capture the details of the bright and dark parts. If the active disturbance control parameters show that the ambient light changes greatly or the camera moves violently, the camera may also increase the number of frames to improve the stability and quality of the image. In addition, the performance resources of the glasses 10 can also be considered to affect the number of frames. It should be understood that the image parameters and the number of frames are not limited in this embodiment and can be set based on actual needs.

[0061] Step S24, sending an image capture instruction carrying the to-be-acquired image parameters and the number of frames.

[0062] In this embodiment, the camera processor 12 sends an image capture instruction carrying the to-be-acquired image parameters and the number of frames to the camera sensor 11.

[0063] Step S25, shooting according to the acquired image parameters and the number of frames to acquire corresponding raw image data.

[0064] In this embodiment, the camera sensor 11 is configured to shoot according to the acquired image parameters and the number of frames to acquire corresponding raw image data.

[0065] In one possible implementation, the camera processor 12 is further configured to send the preliminary noise reduction image data and the underexposed image data set to respective processing nodes in an image signal processing flow for image signal processing to obtain processed preliminary noise reduction image data and underexposed image data.

[0066] The camera processor 12 of this embodiment can use an existing ISP chip configured for the camera sensor, and use the image processing and compression functions in the ISP chip to complete the image signal processing flow.

[0067] Figure 3 Fig. 1 is a schematic diagram of an image signal processing flow according to an embodiment of the present application. As shown, the image signal processing flow comprises image pre-processing node 32, image processing engine node 34, and image encoding node 36, etc. Figure 3

[0068] The image pre-processing node 32 is responsible for basic processing of the image data, including black level compensation, pixel calibration, demosaicing, etc., to correct basic deviations in the image data and improve the initial quality of the image, providing better input for subsequent processing.

[0069] The image processing engine node 34 is responsible for higher-level image processing, including color correction, contrast enhancement, noise reduction, sharpening, etc., to further optimize image quality and make it more consistent with the aesthetic standards of human perception or specific application requirements.

[0070] The image encoding node 36 is responsible for compressing the processed image data into JPEG, YUV or other compressed image formats. This involves discrete cosine transform, quantization and entropy encoding of the image data, etc. This step is to compress the image data to reduce storage space and transmission bandwidth while maintaining acceptable image quality. Through these processing steps, the image data is gradually transformed from the original sensor output into a high-quality, compressed image file suitable for storage and transmission.

[0071] In one possible implementation, each of the image pre-processing node 32, the image processing engine node 34, and the image encoding node 36 is preceded by a corresponding data storage node (Double Data Rate SDRAM, DDR), respectively, the first storage node 31, the second storage node 33, and the third storage node 35, which functions to provide data buffering, synchronization, and improve processing efficiency between data processing nodes, and dynamically adjust the data processing flow as needed, such as increasing processing steps when higher image quality is needed, or reducing processing steps when fast response is needed, etc.

[0072] By way of example, the underexposed image data set and the underexposed image data set each correspond to the image pre-processing node. It should be understood that the processing nodes corresponding to the underexposed image data set and the underexposed image data set can be flexibly set and changed according to actual needs, and the present embodiment does not limit them.

[0073] Further, the first communication unit 13 is configured to send the preliminary noise-reduced image data and the underexposed image data set to the mobile terminal 20.

[0074] ​In an alternative implementation, the first communication unit 13 can be a Bluetooth device, a wifi device, or other communication type device, which can be based on the specific configuration of the glasses 10. Further alternatively, the first communication unit 13 can include both a Bluetooth device and a wifi device. When the glasses 10 are connected to the mobile terminal 20 via Bluetooth, data transmission can be performed via the Bluetooth device. When the glasses 10 are not connected to the mobile terminal 20 via Bluetooth, but are connected to a corresponding routing device via the wifi device, or when the amount of data to be transmitted is large and the glasses 10 are connected to a corresponding routing device via the wifi device, the glasses 10 can perform data transmission via the wifi communication network.

[0075] Further, the mobile terminal 20 includes a second communication unit 21 and a terminal processor 22.

[0076] The second communication unit 21 is configured to receive the preliminary noise-reduced image data and the underexposed image data set from the glasses 10. The mobile terminal 20 is communicatively connected to the glasses 10 via the second communication unit 21 and the first communication unit 13. Alternatively, the second communication unit 21 can be a Bluetooth device, a wifi device, or other communication type device, which can be based on the specific configuration of the mobile terminal 20. Further alternatively, the second communication unit 21 can include both a Bluetooth device and a wifi device. When the glasses 10 are connected to the mobile terminal 20 via Bluetooth, data transmission can be performed via the Bluetooth device. When the glasses 10 are not connected to the mobile terminal 20 via Bluetooth, but are connected to a corresponding routing device via the wifi device, or when the amount of data to be transmitted is large and the glasses 10 are connected to a corresponding routing device via the wifi device, the mobile terminal 20 can perform data transmission with the glasses 10 via the wifi communication network. It should be understood that the present embodiment does not limit the manner of data transmission between the glasses 10 and the mobile terminal 20.

[0077] The terminal processor 22 is configured to perform color space low-frequency noise reduction processing on the preliminary noise-reduced image data and the underexposed image data set to obtain a corresponding target image.

[0078] The color space low-frequency noise reduction processing refers to color space noise reduction (YUV Domain Noise Reduction, YUV NR) processing for low-frequency noise. The main role of YUV NR is to suppress noise in the YUV color space, including brightness noise (Y component) and chroma noise (U and V components). Through noise reduction processing, the image can be clearer and more delicate. Unlike RAW domain noise reduction, YUV NR is performed in the YUV color space and requires more delicate processing of image details to avoid losing important information during noise reduction. YUV NR can also dynamically adjust the noise reduction strength according to different shooting scenes and lighting conditions to balance between noise suppression and detail preservation. YUV NR can be implemented using various algorithms such as spatial domain filtering, frequency domain filtering, wavelet transform, etc., which can be selected according to specific application scenarios.

[0079] The processing object of YUV NR is the preliminary noise reduction image data and the underexposed image data set. For the overexposed image data set, YUV NR and RAW NR are cooperatively processed, RAW NR is mainly responsible for high-frequency noise removal and MFNR residual non-uniform noise removal, and YUV NR is mainly responsible for removing low-frequency noise and color noise. The glasses 10 usually have limited hardware resources, including processor performance and memory capacity. In the glasses 10, RAW NR can utilize limited resources to reduce noise as early as possible, avoid transmitting a large amount of noise data to the mobile terminal 20, reduce data transmission volume, and speed up the overall processing speed of the original image data. The mobile terminal 20 usually has stronger processing capability and can better handle complex algorithms such as YUV NR to further improve image quality. The algorithm layout of the embodiment is based on considerations of data volume and processing efficiency, hierarchy of noise suppression, hardware resources and processing capability, system flexibility and scalability, and transmission efficiency and stability. This hierarchical processing method can more effectively improve image quality while optimizing system performance.

[0080] In a possible implementation, to enable YUV NR to perform noise reduction on low-frequency noise, the YUV NR corresponding module can also be subjected to low-frequency noise reduction training before the mobile terminal 20 is put into use, to improve the low-frequency noise reduction capability of the YUV NR corresponding module. The specific implementation is similar to the high-frequency noise reduction training method of RAW NR, which will not be described here.

[0081] In a possible implementation, the mobile processor is further configured to perform high dynamic range fusion on the preliminary noise reduction image data and the underexposed image data set to obtain corresponding high dynamic image data, and perform color space low-frequency noise reduction processing on the high dynamic image data to obtain corresponding target images.

[0082] The main role of High Dynamic Range Fusion (HDR Fusion) is to combine multiple images with different exposures into one high dynamic range image. During shooting, due to changes in lighting conditions, it is often difficult for a single image to capture both bright and dark details. Through HDR Fusion, the information of multiple images can be integrated to generate an image with a wider brightness range and more detailed information. HDR Fusion can significantly improve the quality of the image, especially in high-contrast scenes. It can reduce overexposure and underexposure, making the image more natural and realistic.

[0083] Optionally, before performing HDR Fusion, the multiple images with different exposures need to be aligned to ensure spatial consistency between them.

[0084] Optionally, according to the brightness and detail information of the image, different weights are assigned to each frame of image. Generally, the weight of the bright area is assigned to the image with darker exposure, and the weight of the dark area is assigned to the image with brighter exposure.

[0085] In the image processing flow, HDR Fusion is usually placed before YUV NR. This is because HDR Fusion needs to integrate the information of multiple images, while YUV NR is a noise reduction process for a single image. By performing HDR Fusion first, a higher quality image can be generated, providing better input for subsequent YUV NR. HDR Fusion and YUV NR complement each other in image processing. HDR Fusion mainly improves the dynamic range and detail performance of the image, while YUV NR mainly suppresses the noise in the image. The combination of the two can generate a clearer, more delicate and natural image.

[0086] In one possible implementation, the mobile processor is also configured to perform Tone Mapping on the high dynamic image data before performing color space low frequency noise reduction processing. Tone Mapping is responsible for compressing the dynamic range of the HDR image, enhancing contrast and details, and correcting colors. Its correct implementation and parameter adjustment can obtain high-quality images that are visually appealing and suitable for display on standard devices.

[0087] Figure 4 The flowchart of the image data processing method of the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the image data processing method comprises the following steps: Figure 4

[0088] Step S41, the glasses obtain the original image data.

[0089] ​Step S42, the glasses perform original data high-frequency noise reduction processing on the overexposed image data set to obtain corresponding preliminary noise reduction image data.

[0090] Step S43, the glasses send the preliminary noise reduction image data and the underexposed image data set to the mobile terminal.

[0091] Step S44, the mobile terminal performs color space low-frequency noise reduction processing on the preliminary noise reduction image data and the underexposed image data set to obtain corresponding target images.

[0092] In the embodiment of the application, the glasses include a camera sensor, a camera processor and a first communication unit, the camera sensor is configured to acquire original image data, the original image data includes an underexposed image data set and an overexposed image data set, the camera processor is configured to perform original data high-frequency noise reduction processing on the overexposed image data set to obtain corresponding preliminary noise reduction image data, and the first communication unit is configured to send the preliminary noise reduction image data and the underexposed image data set, the mobile terminal includes a mobile processor and a second communication unit, the second communication unit is configured to receive the preliminary noise reduction image data and the underexposed image data set, and the mobile processor is configured to perform color space low-frequency noise reduction processing on the preliminary noise reduction image data and the underexposed image data set to obtain corresponding target images. Thus, the glasses of the embodiment can perform original data high-frequency noise reduction processing on the original image data, and the mobile terminal can further perform color space low-frequency noise reduction processing on the data processed by the glasses, which significantly improves the image quality on the premise of ensuring low power consumption of the equipment.

[0093] Figure 5 The data flow chart of the image data processing system of the embodiment of the application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the data flow of the image data processing system is as follows:

[0094] Step S501, preview self-tuning, to obtain image control parameters.

[0095] Specifically, the preview self-tuning refers to Preview 3A, which is a real-time automatic adjustment function of the camera assembly in the preview mode, used for adjusting 3A. 3A represents automatic exposure (AE), automatic focusing (AF) and automatic white balance (AWB). The image control parameters include exposure parameters, brightness distribution histogram and / or self-disturbance control parameters, that is, DRC stats (dynamic range control statistics) and AE stats (automatic exposure statistics).

[0096] Step S502, determining the image parameters and frame number to be acquired according to the image control parameters.

[0097] Specifically, the HDR AE (High Dynamic Range Auto Exposure) module calculates the parameters and frame number (i.e. AE param, FrameNum) of the multiple frames of images according to the image control parameters obtained in step S501, which are usually n frames of EV0 / EV+, one frame of EV-, and one frame of EV--. The camera sensor is informed to capture the frames according to the requirements.

[0098] Optionally, the HDR AE module transmits the calculated exposure parameters (such as exposure time, gain, etc.) and frame number information to the driving module of the sensor, so as to drive the sensor to capture multiple frames of images according to the specified exposure settings according to the received parameters. These images will contain different exposure versions, providing data for subsequent HDR synthesis.

[0099] Step S503, acquiring corresponding raw image data according to the image parameters and frame number to be acquired.

[0100] The raw image data includes an underexposed image data group and an overexposed image data group.

[0101] Optionally, the underexposed image data group includes underexposed image data with different degrees of exposure reduction, and the overexposed image data group includes standard exposure image data and overexposed image data with different degrees of exposure increase.

[0102] Step S504, receiving, decoding, converting, buffering, synchronizing and correcting data from the image sensor.

[0103] The camera serial interface decoder (CSID) is responsible for receiving, decoding, converting, buffering, synchronizing and correcting data from the image sensor, so as to ensure that the data can be correctly and efficiently processed by the subsequent processing modules.

[0104] The RAW image frame grabbing position can be adjusted according to actual conditions, and the embodiment does not limit the frame grabbing position.

[0105] Step S505, performing multi-frame noise reduction processing on the overexposed image data group.

[0106] Specifically, after the CSID, multiple frames of images with different exposure can be obtained, and according to actual requirements and processing capacity, n frames of EV0 or EV+ images are selected for subsequent MFNR processing. These frames should have similar scene content, but there may be slight differences due to hand jitter, object movement, etc. After the n frames of EV0 or EV+ images after the CSID are processed by the MFNR, an image alignment, motion removal, fusion denoising, etc. are performed, a frame of single-frame RAW image with better signal-to-noise ratio and higher clarity is obtained, wherein the image alignment can be selected by the optical flow method, template matching, etc., the motion removal can be detected by n-order frame difference, cross-correlation, etc., and the fusion denoising can be completed by time domain modeling, frequency domain fusion, multi-scale fusion, etc. The role of this step is to remove the motion artifacts and noise differences between multiple frames.

[0107] In step S506, the original data high-frequency denoising processing is performed on the data obtained in the previous step.

[0108] The role of the original data high-frequency denoising processing is to remove the noise in the single frame, and to perform single-frame denoising to obtain the denoised EV0.

[0109] In step S507, the image data after the original data high-frequency denoising processing is backfilled to the image signal processing flow.

[0110] The image signal processing flow is an image processing process provided by an ISP (Image Signal Process) chip, and the image signal processing flow (hereinafter referred to as ISP) includes processing nodes such as an image preprocessing node, an image processing engine node, and an image encoding node. Figure 5 The EV0 is backfilled to the image preprocessing node.

[0111] It should be understood that the backfill node of the EV0 can be set according to actual requirements, Figure 5 The backfill node in the above embodiment is only an example. The backfill node refers to sending the corresponding image data back to the corresponding node of the ISP.

[0112] In step S508, the image data is subjected to image preprocessing.

[0113] The image preprocessing includes black level compensation, pixel calibration, demosaicing, etc., that is, BPS.

[0114] In step S509, the image data is subjected to image processing.

[0115] The image processing includes color correction, contrast enhancement, denoising, sharpening, etc., that is, IPE.

[0116] In step S510, the image data is subjected to image data compression.

[0117] The image data compression format can be JPEG, YUV, or other compressed image format. That is, JPEGEncoder.

[0118] Figure 5 The DDR appearing in the middle is a data storage node, which functions to provide data buffering, synchronization, improve processing efficiency, and dynamically adjust the data processing flow as needed, such as increasing processing steps when higher image quality is needed, or reducing processing steps when fast response is needed. The Storage is also a data storage node, which is used to store the generated preliminary denoised image data and the ISP-processed underexposed image data set. The data storage is not introduced in this embodiment.

[0119] It should be noted that the above steps S501-S510 are steps performed by the glasses 10.

[0120] That is, in the above data flow process, the flow direction of the underexposed image data set is Preview 3A, HDR AE, Sensor capture, CSID, DDR, BPS, DDR, IPE, DDR, JPEG Encoder, storage. The flow direction of the overexposed image data set is Preview 3A, HDR AE, Sensor capture, CSID, DDR, MFNR, RAW NR, BPS, DDR, IPE, DDR, JPEG Encoder, storage.

[0121] Step S511, the glasses 10 sends the preliminary denoised image data and the underexposed image data set to the mobile terminal 20. The following steps are performed by the mobile terminal 20.

[0122] Step S512, high dynamic range fusion is performed on the preliminary denoised image data and the underexposed image data set to obtain corresponding high dynamic image data.

[0123] That is, the EV0, EV-, and EV-- received from the glasses 10 are fused by HDR fusion, and specifically, exposure fusion, unet, etc. can be used for fusion to generate a frame of high dynamic image. Exposure Fusion is an algorithm based on multi-exposure fusion, which generates a high-quality HDR image by weighted averaging of images with different exposure levels according to local contrast, saturation, and other features of the image.

[0124] Step S513, tone mapping is performed on the high dynamic image data.

[0125] Specifically, CLAHE, HDRnet, etc. can be used for tone mapping.

[0126] Step S514, color space low-frequency noise reduction processing is performed on the data obtained in the previous step to obtain a corresponding target image.

[0127] That is, YUV NR is performed.

[0128] The processing methods of the above steps are described in the above embodiments and will not be repeated here.

[0129] The glasses of the present embodiment can perform original data high-frequency noise reduction processing on the original image data, and the mobile terminal can further perform color space low-frequency noise reduction processing on the data processed by the glasses, which significantly improves the image quality on the premise of ensuring low power consumption of the device. The present embodiment arranges the links that remove high-frequency noise, dark noise, and other links that require higher input image quality and have less computational complexity before the ISP processing of the shooting terminal, which can achieve a signal-to-noise ratio comparable to a pure RAW algorithm and significantly reduce the amount of data to be transmitted. The data transmission amount is also significantly less than the pure YUV algorithm. The algorithm links such as wide dynamic synthesis, low-frequency noise reduction, and tone mapping, which are more sensitive to the final human senses, are deployed to the mobile terminal (also can be deployed in the cloud), which can achieve better viewing effect through more complex algorithms and does not significantly increase the power consumption of the glasses. The present embodiment mixes the RAW algorithm and the YUV algorithm and reasonably cooperates with the platform ISP, changes the traditional one-stop pipeline to a pipeline arranged in the shooting terminal and the mobile terminal or the cloud, and realizes the performance and effect balance that the traditional pipeline cannot achieve. The high-frequency noise and the low-frequency noise are distributed to the front processing and the post-processing of the platform ISP, respectively, which reduces the overall computational complexity.

[0130] Another embodiment of the present application relates to a non-volatile storage medium configured to store a computer-readable program configured to cause a computer to execute part or all of the method embodiments described above.

[0131] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing related hardware, the programs are stored in a storage medium, and include a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0132] The above descriptions are only the preferred embodiments of the present application, and are not configured to limit the present application, and the present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image data processing system, characterized by, The system comprises: eyeglasses, comprising a camera sensor configured to acquire raw image data, the raw image data comprising an underexposed image data set and an overexposed image data set, a camera processor configured to perform raw data high-frequency noise reduction processing on the overexposed image data set to obtain corresponding preliminary noise-reduced image data, and a first communication unit configured to transmit the preliminary noise-reduced image data and the underexposed image data set; a mobile terminal, comprising a second communication unit configured to receive the preliminary noise-reduced image data and the underexposed image data set, and a mobile processor configured to perform color space low-frequency noise reduction processing on the preliminary noise-reduced image data and the underexposed image data set to obtain a corresponding target image.

2. The system of claim 1, wherein, The underexposed image data set comprises underexposed image data with different degrees of exposure reduction, and the overexposed image data set comprises standard exposure image data and overexposed image data with different degrees of exposure increase.

3. The system of claim 1, wherein, The camera processor is further configured to perform multi-frame noise reduction processing on the overexposed image data set.

4. The system of claim 1, wherein, The camera sensor is further configured to acquire image regulation parameters, and the camera processor is further configured to determine to-be-acquired image parameters and frame numbers according to the image regulation parameters, and control the camera sensor to acquire corresponding raw image data according to the to-be-acquired image parameters and frame numbers, the image regulation parameters comprising exposure parameters, brightness distribution histograms, and / or self-disturbance control parameters.

5. The system of claim 1, wherein, The camera processor is further configured to respectively transmit the preliminary noise-reduced image data and the underexposed image data set to respective corresponding processing nodes in an image signal processing flow for image signal processing to obtain processed preliminary noise-reduced image data and underexposed image data sets.

6. The system of claim 5, wherein, The processing nodes comprise image preprocessing nodes, image processing engine nodes, and / or image encoding nodes, and the processing node corresponding to the underexposed image data set is the image preprocessing node.

7. The system of claim 1 or 5, wherein, The mobile processor is further configured to perform high dynamic range fusion on the preliminary noise-reduced image data and the underexposed image data set to obtain corresponding high dynamic image data, and perform color space low-frequency noise reduction processing on the high dynamic image data to obtain a corresponding target image.

8. The system of claim 7, wherein, The mobile processor is further configured to perform tone mapping on the high dynamic image data before performing color space low-frequency noise reduction processing.

9. Eyeglasses, characterized in that, The eyeglasses comprise: a camera sensor configured to acquire raw image data, the raw image data comprising an underexposed image data set and an overexposed image data set; a camera processor configured to perform raw data high-frequency noise reduction processing on the overexposed image data set to obtain corresponding preliminary noise-reduced image data; a first communication unit configured to transmit the preliminary noise-reduced image data and the underexposed image data set to a mobile terminal, so that the mobile terminal performs color space low-frequency noise reduction processing on the preliminary noise-reduced image data and the underexposed image data set to obtain a corresponding target image.

10. A mobile terminal, characterized by The mobile terminal comprises: a second communication unit configured to receive a preliminary de-noised image data and an underexposed image data set from the glasses, the preliminary de-noised image data being image data obtained by performing raw data high-frequency de-noising on an overexposed image data set by the glasses; a mobile processor configured to perform color space low-frequency de-noising on the preliminary de-noised image data and the underexposed image data set to obtain a corresponding target image.

Images