A white point estimation method and apparatus

Abstract

This application provides a white point estimation method and device for estimating the white point of a light source based on multimodal data, thereby improving the accuracy of white point estimation. The method includes: acquiring multimodal data, the multimodal data including a first multispectral image and hardware parameters of an imaging device, wherein the first multispectral image is obtained by capturing a scene using the imaging device, and the hardware parameters are used to indicate the brightness of the light source in the scene; processing the multimodal data to obtain white points, the white points being used for white balance processing of an RGB image and / or a second multispectral image.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to a white spot estimation method and device. Background Technology

[0002] Currently, most cameras and smartphones are equipped with color image sensors, often referred to as RGB sensors. RGB sensors aim to directly simulate the spectral response of the three types of cone cells (L / M / S) in the human eye. However, due to discrepancies between the sensor's response curve and the LMS visual response curve, the sensor's raw response (i.e., the raw RAW image signal) typically requires color correction to ensure that the image's colors accurately reproduce what the human eye perceives at the shooting location. Furthermore, due to the color constancy of the human eye, white balance and other operations are usually required to correct the RAW image response.

[0003] With the development of deep learning, existing solutions have proposed sensor-independent lighting estimation schemes based on deep learning. These schemes analyze raw RGB images using convolutional neural network models to estimate lighting conditions in a scene. Subsequently, the lighting estimation results can be used to perform color correction, brightness adjustment, and other adjustments on the image. However, since the raw RGB images are obtained from RGB sensors, which cannot fully capture spectral information, the accuracy of the lighting estimation results is relatively low. Therefore, to capture the spectral information of the light source more completely, multispectral image sensors have been introduced into terminal devices. Existing solutions analyze multispectral images to obtain white point estimation results, and then perform white balance and other processing on the image based on the white points.

[0004] However, white point estimation based solely on multispectral images is susceptible to the signal-to-noise ratio (SNR) of the multispectral image sensor. When the SNR of the multispectral image sensor is low, it will affect the quality and resolution of the multispectral image, thereby reducing the accuracy of white point estimation. Summary of the Invention

[0005] This application provides a white point estimation method and apparatus for estimating the white point of a light source based on multimodal data, thereby improving the accuracy of white point estimation.

[0006] In a first aspect, this application provides a white point estimation method, comprising: acquiring multimodal data, the multimodal data including a first multispectral image and hardware parameters of an imaging device, wherein the first multispectral image is obtained by capturing a scene using the imaging device, and the hardware parameters are used to indicate the brightness of light sources in the scene; subsequently, processing the multimodal data to obtain white points, the white points being used for white balance processing of RGB images and / or a second multispectral image.

[0007] In this embodiment, the hardware parameters of the shooting device are parameters that affect the pixel values ​​in the multispectral image. The hardware parameters of the shooting device can reflect the brightness of the light source in the current scene. Since the white point values ​​are different under light sources with different brightness, the hardware parameters of the shooting device can be introduced in addition, and the white point can be estimated by combining the hardware parameters of the shooting device and the multispectral image, so as to obtain richer information about the brightness of the scene light source, thereby improving the accuracy of white point estimation.

[0008] In one possible implementation, the aforementioned processing of multimodal data to obtain white points may include: determining a first weight coefficient for each modality in the multimodal data using a convolutional neural network; and processing the multimodal data based on the multiple first weight coefficients to obtain white points.

[0009] In this embodiment, weights can be assigned to each modality in the multimodal data according to the scenario. Greater weights can be assigned to modal data that have a greater impact on white point estimation, thereby improving the accuracy of white point estimation based on multimodal data.

[0010] In one possible implementation, the aforementioned determination of the first weight coefficient of each modality in the multimodal data using a convolutional neural network may include: obtaining a second weight coefficient of each modality using a convolutional neural network based on the feature vector of each modality in the multimodal data; obtaining an attention matrix based on the feature vector of each modality, the attention matrix being used to represent the importance of each modality in the multimodal data; and calculating the attention matrix and the second weight coefficient to obtain the first weight coefficient of each modality.

[0011] In one possible implementation, the aforementioned process of processing multimodal data according to multiple first weight coefficients to obtain white points may include: processing multimodal data according to multiple first weight coefficients to obtain fused data; and processing the fused data through a neural network model to obtain white points.

[0012] In one possible implementation, the aforementioned multimodal data further includes one or more of the following: main camera image, scene category, scene segmentation map, and human detection bounding box; the main camera image is an image captured by the main camera sensor of the shooting device, the scene category is used to represent the category information of objects in the scene, and the scene segmentation map is used to represent the region information in the scene.

[0013] In this embodiment, since the multispectral sensor acquires less image information in low-sensitivity or high-dynamic-range scenes, while the main camera sensor produces higher-quality images with more information in low-sensitivity or high-dynamic-range scenes, the multimodal data can also include the main camera image to provide more image information and increase the accuracy of white point estimation. Furthermore, the scene category can determine the object category, and the corresponding color temperature can be determined based on the object category. The color temperature can then be converted to white point using a conversion formula. Therefore, obtaining the scene category provides more scene information, thereby improving the accuracy of white point estimation. The scene segmentation map can obtain region information within the scene. When the region information indicates the presence of multiple regions in the scene, white point estimation can be adaptively performed based on different regions to enhance the robustness of white point estimation. Also, since the colors of people in the image are usually more vibrant and saturated when a person is present in the scene, a person detection box can be added as a multimodal input to make the output white point value more consistent with reality.

[0014] In one possible implementation, the aforementioned hardware parameters include one or more of the following: ISO sensitivity, LV brightness, aperture size, and shutter speed.

[0015] Secondly, embodiments of this application provide a white point estimation device, comprising: an acquisition module for acquiring multimodal data, the multimodal data including a first multispectral image and hardware parameters of an imaging device, the first multispectral image being obtained by capturing a scene using the imaging device, and the values ​​of the hardware parameters being used to represent the brightness of the light source in the scene; and a processing module for processing the multimodal data to obtain white points, the white points being used for white balance processing of the RGB image and / or the second multispectral image.

[0016] In one possible implementation, the aforementioned processing module is specifically used to: determine the first weight coefficient of each modality in the multimodal data through a convolutional neural network; and process the multimodal data according to the multiple first weight coefficients to obtain white points.

[0017] In one possible implementation, the aforementioned processing module is specifically used to: obtain a second weight coefficient for each modality data through a convolutional neural network based on the feature vector of each modality data in the multimodal data; obtain an attention matrix based on the feature vector of each modality data, the attention matrix being used to represent the importance of each modality data in the multimodal data; and calculate a first weight coefficient for each modality data by calculating the attention matrix and the second weight coefficient.

[0018] In one possible implementation, the aforementioned processing module is specifically used to: process the multimodal data according to multiple first weight coefficients to obtain fused data; and process the fused data through a neural network model to obtain white points.

[0019] In one possible implementation, the aforementioned multimodal data further includes one or more of the following: main camera image, scene category, scene segmentation map, and human detection bounding box; the main camera image is an image captured by the main camera sensor of the shooting device, the scene category is used to represent the category information of objects in the scene, and the scene segmentation map is used to represent the region information in the scene.

[0020] In one possible implementation, the aforementioned hardware parameters include one or more of the following: ISO sensitivity, LV brightness, aperture size, and shutter speed.

[0021] Thirdly, this application provides an electronic device comprising: a processor for executing a program, causing the electronic device to perform the method as described in the first aspect or any implementation thereof as claimed above.

[0022] Optionally, the electronic device further includes a memory, with the processor coupled to the memory; the memory is used to store programs.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0024] Fifthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0025] In a sixth aspect, embodiments of this application provide a chip or chip system, the chip or chip system including at least one processor and a communication interface, the communication interface and at least one processor being interconnected via a line, the at least one processor being used to run computer programs or instructions to perform the method as described in the first aspect or any possible implementation of the first aspect.

[0026] The communication interface in the chip can be an input / output interface, pins, or circuits.

[0027] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself, such as a read-only memory or random access memory.

[0028] The technical effects of the second to sixth aspects or any of their possible implementations can be found in the first aspect or the technical effects of its related possible implementations, and will not be repeated here. Attached Figure Description

[0029] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0030] Figure 2 A flowchart illustrating a white spot estimation method provided in this application;

[0031] Figure 3 This is a schematic diagram of confidence estimation based on convolutional layers and fully connected layers;

[0032] Figure 4 A schematic diagram of a white point estimation device provided in this application;

[0033] Figure 5 A schematic diagram of another white point estimation device provided in this application;

[0034] Figure 6 This is a schematic diagram of the structure of a chip provided in this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0036] To facilitate understanding, some concepts involved in this application will be introduced first.

[0037] 1. Multimodal data

[0038] Multimodal data refers to data acquired through different domains or perspectives, with each domain or perspective referred to as a modality. For example, images, text, audio, and video can all be considered different modalities.

[0039] 2. Multispectral images

[0040] A multispectral image is an image that contains many bands, each of which is a grayscale image. Each band represents the scene brightness based on the sensitivity of the sensor used to generate that band.

[0041] 3. White point

[0042] The white point, also known as reference white or target white, is a set of tristimulus values ​​or chromaticity values. It is used to define white during image capture, encoding, and reproduction. The value of the white point is not constant and needs to be adjusted according to the usage. For example, indoor photography often uses incandescent lamps for illumination. The color temperature of incandescent light is relatively low compared to sunlight, thus appearing orange-toned. If the white point value based on sunlight is used to define white in this situation, the photographic result will have a color cast problem.

[0043] 4. White balance (WB)

[0044] White balance is a crucial concept in photography, videography, and display technology, used to ensure that white objects in an image are accurately rendered as white under varying lighting conditions. Adjusting white balance aims to eliminate the influence of light source color temperature on color, resulting in more realistic and natural colors in the image. White balance includes automatic white balance (AWB), preset white balance, manual white balance, and custom white balance. It is primarily used in photography, videography, display technology, and image processing.

[0045] 5. Attention Mechanism

[0046] The core idea of ​​attention mechanisms is to enable models to selectively focus on different parts of the input sequence, assigning different weights to each part to highlight information more critical to the task. In natural language processing, attention mechanisms are widely used in tasks such as machine translation, text classification, and question answering systems. In computer vision, attention mechanisms are also used in tasks such as image classification and object detection, allowing models to focus on important regions or features in images.

[0047] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0048] See Figure 1 This application provides a system architecture 100. For example... Figure 1As shown, the data acquisition device 160 can be used to collect training data. After the data acquisition device 160 collects the training data, it stores the training data in the database 130. The training device 120 trains the neural network model 102 based on the training data maintained in the database 130.

[0049] It should be noted that in practical applications, the training data maintained in database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that training device 120 may not necessarily train the neural network model 102 entirely based on the training data maintained in database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0050] The calculation module 111 includes a confidence estimation module 101 and a neural network model 102. The white point estimation method mentioned in this application can be executed by the calculation module 111. Specifically, the confidence estimation module 101 is used to determine the weight coefficient of each modality in the multimodal data, and weights the multimodal data according to the weight coefficient of each modality to obtain fused data; the neural network model 102 is used to process the fused data to obtain white points. The white point estimation method proposed in this application embodiment can be applied to different devices, such as... Figure 1 The execution device 110 shown can be a smart terminal device such as a mobile phone, smart car, robot vacuum cleaner, AR / VR glasses, or a device with computing capabilities such as a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), tensor processor (TPU), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA). Figure 1 In this embodiment, the execution device 110 is equipped with a transceiver 112, which may include an input / output (I / O) interface or other wireless or wired communication interfaces for data interaction with external devices. Taking the I / O interface as an example, the user can input data to the I / O interface through the client device 140. For example, in the following embodiments of this application, the client device 140 can send the required multimodal data to the execution device 110, requesting the execution device 110 to output white points obtained by processing the multimodal data.

[0051] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0052] Finally, the I / O interface returns the processing result to the client device 140, thus providing it to the user.

[0053] It is worth noting that, attached Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0054] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0055] See Figure 2 The following is a flowchart illustrating a method for restoring the spectrum of a light source provided in this application.

[0056] Step 201: Obtain multimodal data;

[0057] In this embodiment, multimodal data is first acquired, including a first multispectral image and hardware parameters of the capturing device. The first multispectral image is obtained by the capturing device capturing a scene; exemplarily, the capturing device can be a mobile phone or camera equipped with a multispectral image sensor. The multispectral image sensor can be a multispectral image sensor based on a filter array (or filter coating). The hardware parameters of the capturing device are parameters that affect the pixel values ​​in the multispectral image; a higher pixel value indicates higher brightness.

[0058] For example, the hardware parameters of the shooting device can be parameters such as the camera's ISO (International Standards Organization), light value (LV), aperture size, shutter speed (or exposure time), or dynamic range. ISO refers to the camera's sensitivity to light; a higher ISO value results in a brighter image, and vice versa. LV reflects the brightness of the scene; a higher LV value indicates a brighter scene, and vice versa. Aperture size and shutter speed can both adjust the image brightness by controlling the amount of light entering the camera. The camera's dynamic range is the range of brightness between the brightest and darkest signals it can capture. The brightest signal is the maximum light intensity that a multispectral image sensor can record in a single exposure, and the darkest signal is the minimum effective signal intensity that can be resolved.

[0059] Furthermore, since the values ​​of hardware parameters are determined based on the brightness of the light sources in the scene, for example, when the light source in the scene is dim, the camera's ISO value is higher, and when the light source in the scene is bright, the camera's ISO value is lower. When the light source in the scene is dim, the camera's LV value is smaller, and the aperture can be increased and the shutter speed decreased to increase the amount of light entering the camera; when the light source in the scene is bright, the camera's LV value is larger, and the aperture can be decreased and the shutter speed increased to decrease the amount of light entering the camera. When there are very bright areas and very dark areas in the scene, the dynamic range is very wide, for example, in backlighting (the subject is facing away from a window, where the background is relatively bright, while the subject is dark due to backlighting). Therefore, it can be seen that the values ​​of hardware parameters are related to the brightness of the light sources in the scene. Thus, hardware parameters can be used to indicate the brightness of the light sources in the scene, and the brightness of the light sources in the scene can be determined based on the hardware parameters of the shooting device.

[0060] In this embodiment, since the white point value varies under different light sources, estimating the white point solely through multispectral images can lead to issues. Furthermore, the low signal-to-noise ratio of the multispectral sensor results in low-quality multispectral images, potentially causing increased noise, a darker image, or insufficient clarity, thus affecting the accuracy of white point estimation. However, by introducing additional hardware parameters from the imaging device, more brightness information about the light sources in the scene can be obtained. Combining the multispectral images with the hardware parameters of the imaging device for white point estimation yields richer information about the brightness of the scene's light sources, thereby improving the accuracy of white point estimation.

[0061] Furthermore, since multispectral image sensors acquire less image information in low-sensitivity or high-dynamic-range scenes, while the main camera sensor acquires higher-quality images with more information in the same conditions, the main camera image can also be acquired. The main camera image is the image captured by the main camera sensor of the shooting device. Optionally, the multimodal data may also include the main camera image, allowing for subsequent white point estimation based on the first multispectral image, the main camera image, and the hardware parameters of the shooting device. High-dynamic-range scenes refer to scenes with a wide brightness range, from very dark shadows to very bright highlights, all with detail.

[0062] In this embodiment, since the main camera sensor can acquire higher quality images in low-sensitivity or high-dynamic-range scenes, and the obtained main camera image can provide more image details and information, the additional introduction of the main camera image as multimodal data can improve the accuracy of white point estimation.

[0063] In one possible implementation, the multimodal data may further include one or more of the following: scene categories, scene segmentation maps, or human detection bounding boxes. The scene category represents the category information of objects in the scene, allowing the classification of objects present in the scene to be determined. The scene category can be a 1×N array [x1, x2, ..., x...]. N N represents the maximum number of categories that can appear in the defined scene. For example, a scene can include categories such as flowers, trees, grass, sky, or human figures. The scene segmentation map is used to represent the regional information in the scene. Based on the scene segmentation map, we can know the object information corresponding to multiple regions in the scene. For example, if the scene includes N regions, we can obtain the object information of each region based on the corresponding identifiers of the N regions (e.g., region 1 is the blue sky, region 2 is flowers, region 3 is greenery, region 4 is a human figure, etc.).

[0064] Scene categories can be obtained by classifying multispectral images using a neural network model. Scene segmentation maps can be obtained by classifying each pixel in the multispectral image using a neural network model. Each pixel in the scene segmentation map corresponds to an identifier, which indicates the object category of the pixel, and pixels of the same category belong to the same region. Face detection boxes can be obtained by detecting multispectral images using a convolutional neural network model, with one face corresponding to one face detection box.

[0065] In this embodiment, since different categories of objects correspond to different color temperatures, the color temperature of the scene can be determined based on the scene category, and the color temperature can be converted into white point using a conversion formula. Therefore, by adding scene categories to the multimodal data, more scene information can be obtained, thereby improving the accuracy of white point estimation. Furthermore, the scene segmentation map divides the scene into multiple regions, allowing for the determination of the presence of multiple different light sources within the scene based on these regions. When multiple different light sources exist, white point estimation can be adaptively performed based on different regions to enhance the robustness of white point estimation. Also, since the colors of people in the image are usually more vibrant and saturated when a person is present in the scene, a person detection bounding box can be added as a multimodal input to make the output white point value more consistent with reality.

[0066] Step 202: Process the multimodal data to obtain white points.

[0067] After acquiring multimodal data, a neural network model can be used to process the multimodal data to obtain white points, which can be used for white balance processing of the image. The image can be an RGB image and / or a second multispectral image. The second multispectral image can be the first multispectral image, or other multispectral images obtained by the camera capturing the current scene; specific details are not limited here.

[0068] White balance processing for RGB images can eliminate the interference of light source color temperature on colors, making the colors of objects in the image closer to the true colors perceived by the human eye.

[0069] Optionally, before processing the multimodal data, the weight coefficients corresponding to each modality in the multimodal data can be determined.

[0070] One possible implementation is to determine the first weight coefficient corresponding to each modality data through a first convolutional neural network.

[0071] One possible implementation is to obtain a second weight coefficient for each modality data based on the feature vector of each modality data in the multimodal data through a second convolutional neural network; and to obtain an attention matrix based on the feature vector of each modality data, which is used to represent the importance of each modality data in the multimodal data; subsequently, matrix multiplication can be performed on the attention matrix and the second weight coefficient to obtain a first weight coefficient.

[0072] The feature vector of each modality data can be obtained by feature extraction from the multimodal data through the convolutional layer in the second convolutional neural network. For example, the input multimodal data includes N modality data m1, m2, ..., m N Through convolutional layers, N feature vectors f1, f2, ..., f can be obtained. N Then, the N feature vectors are analyzed through a fully connected layer in the second convolutional neural network to obtain multiple second weight coefficients σ1, σ2, ..., σ N .

[0073] One possible implementation is based on an attention mechanism, calculating an attention matrix from the feature vectors of each modality. Specifically, for each modality's feature vector, a query vector (Q vector), a key vector (K vector), and a value vector (V vector) are generated, and then the matrix is ​​applied according to the formula... The attention matrix is ​​calculated. Where d k is the dimension of the key vector.

[0074] One possible implementation is to pre-set the first weighting coefficients for different modal data in the acquired multimodal data.

[0075] After obtaining the first weight coefficient of each modality in the multimodal data, the multimodal data can be processed according to the first weight coefficient corresponding to each modality to obtain fused data. For example, after obtaining the weight coefficients of the first multispectral image, brightness value, and main camera image, the first multispectral image, brightness value, and main camera image can be multiplied by their respective weight coefficients, and then stacked according to the dimension of each data to obtain fused data.

[0076] One possible implementation is that the main image is a three-channel RGB image with dimensions H×W×3, and the first multispectral image has dimensions H×W×N, where N is the number of channels of the multispectral image sensor. N can be a positive integer greater than 3. The brightness values ​​can be converted into image dimensions H×W×1, and then stacked to obtain fused data with dimensions H×W×(3+N+1). Here, H×W represent the width and height (spatial resolution) of the image, respectively.

[0077] For example, such as Figure 3 The diagram shown illustrates confidence estimation based on convolutional and fully connected layers. Figure 3 It is known that when the multimodal data includes the first multispectral image, the main camera image, and the hardware parameters of the imaging device, the multimodal data can be analyzed using a confidence estimation module to obtain the weight coefficients a1 for the first multispectral image, a2 for the main camera image, and a3 for the hardware parameters. The confidence estimation module can be a convolutional neural network, which includes convolutional layers and fully connected layers. Subsequently, the data is weighted according to the weight coefficients of each modality and stacked as described above to obtain the fused data. The fused data is then processed by a neural network model to obtain the white points.

[0078] In this embodiment, the weight coefficients of multiple data points in the multimodal data can be dynamically adjusted according to the scene, so that factors that have a greater impact on white point estimation in different scenes can obtain higher weights, thereby improving the accuracy of white point estimation. For example, when the multimodal data includes a first multispectral image, a main camera image, and a brightness value, the brightness value is relatively small, indicating that the current light source may be dark. At this time, the multispectral image sensor collects less image information, while the main camera image is captured by the main camera sensor and contains more image information. Therefore, the weight coefficient of the main camera image can be increased.

[0079] Optionally, the fused data can be processed using a neural network model to obtain white points.

[0080] After obtaining the white point, white balance processing can be performed on the RGB image and / or the second multispectral image based on the value of the white point under the current scene light source.

[0081] Specifically, one possible implementation is to calculate the gain coefficients of each channel of the RGB image sensor based on the value of the white point and the value of the target white point. These gain coefficients are the factors that ensure the pixel values ​​of the white point are equal in each channel. After obtaining the gain coefficients for each channel, each pixel of the RGB image can be corrected by multiplying the pixel value of each channel of each pixel by the gain coefficient, thereby achieving color correction of the RGB image.

[0082] In this embodiment of the application, since the hardware parameters of the shooting device can reflect the brightness of the light source in the current scene, the additional introduction of the hardware parameters of the shooting device for white point estimation can obtain more light source information in the current scene, thereby improving the accuracy of white point estimation, and thus making the image based on white point color correction closer to the scene seen by the human eye.

[0083] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0084] See Figure 4 The following is a schematic diagram of the structure of a white point estimation device provided in this application.

[0085] The acquisition module 401 is used to acquire multimodal data, which includes a first multispectral image and hardware parameters of the imaging device. The first multispectral image is obtained by capturing the scene using the imaging device, and the values ​​of the hardware parameters are used to represent the brightness of the light source in the scene.

[0086] The processing module 402 is used to process multimodal data to obtain white points, which are used for white balance processing of RGB images and / or second multispectral images.

[0087] In one possible implementation, the aforementioned processing module 402 is specifically used to: determine the first weight coefficient of each modality data in the multimodal data through a convolutional neural network; and process the multimodal data according to the multiple first weight coefficients to obtain white points.

[0088] In one possible implementation, the aforementioned processing module 402 is specifically used to: obtain a second weight coefficient for each modality data through a convolutional neural network based on the feature vector of each modality data in the multimodal data; obtain an attention matrix based on the feature vector of each modality data, the attention matrix being used to represent the importance of each modality data in the multimodal data; and calculate the attention matrix and the second weight coefficient to obtain a first weight coefficient for each modality data.

[0089] In one possible implementation, the aforementioned processing module 402 is specifically used to: process multimodal data according to multiple first weight coefficients to obtain fused data; and process the fused data through a neural network model to obtain white points.

[0090] In one possible implementation, the aforementioned multimodal data further includes one or more of the following: main camera image, scene category, scene segmentation map, and human detection bounding box; the main camera image is an image captured by the main camera sensor of the shooting device, the scene category is used to represent the category information of objects in the scene, and the scene segmentation map is used to represent the region information in the scene.

[0091] In one possible implementation, the aforementioned hardware parameters include one or more of the following: ISO sensitivity, LV brightness, aperture size, and shutter speed.

[0092] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 2 The steps in the method described in the illustrated embodiment.

[0093] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 2 The method steps described in the illustrated embodiment.

[0094] Please see Figure 5 The following is a schematic diagram of another white point estimation device provided in this application.

[0095] The white point estimation device may include a processor 501 and a memory 502. The processor 501 and the memory 502 are interconnected via a circuit. The memory 502 stores program instructions and data.

[0096] The aforementioned memory 502 stores Figure 2 The steps in the code include the corresponding program instructions and data.

[0097] Processor 501 is used to perform the aforementioned Figure 2 The method steps performed by the white point estimation apparatus shown in any of the embodiments.

[0098] Optionally, the white point estimation device may also include a transceiver 503 for receiving or sending data.

[0099] Optional, the aforementioned Figure 5 The white point estimation device shown is a chip.

[0100] This application also provides a white point estimation device, which can also be called a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figure 2 The method steps performed by the white point estimation apparatus shown in any of the embodiments.

[0101] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the functions of the processor 501 described above, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the white point estimation device in the foregoing embodiments based on the program code stored in the external memory.

[0102] The parameter optimization device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the server to perform the aforementioned operations. Figure 2 The method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.

[0103] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0104] For example, please refer to Figure 6 , Figure 6 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 600. The NPU 600 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 603, which is controlled by a controller 604 to extract matrix data from the memory and perform multiplication operations.

[0105] In some implementations, the arithmetic circuit 603 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 603 is a two-dimensional pulsating array. The arithmetic circuit 603 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 603 is a general-purpose matrix processor.

[0106] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 602 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 608.

[0107] Unified memory 606 is used to store input and output data. Weight data is directly transferred to weight memory 602 via direct memory access controller (DMAC) 605. Input data is also transferred to unified memory 606 via DMAC.

[0108] The bus interface unit (BIU) 610 is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 609.

[0109] The bus interface unit 610 (BIU) is used by the instruction fetch memory 609 to fetch instructions from external memory, and also by the memory access controller 605 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0110] The DMAC is mainly used to move input data from external memory DDR to unified memory 606, or to weight data to weight memory 602, or to input data to input memory 601.

[0111] The vector computation unit 607 includes multiple arithmetic processing units that further process the output of the computation circuit as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0112] In some implementations, the vector computation unit 607 can store the processed output vector in the unified memory 606. For example, the vector computation unit 607 can apply linear and / or nonlinear functions to the output of the computation circuit 603, such as linear interpolation of feature planes extracted by convolutional layers, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 607 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 603, for example, for use in subsequent layers of the neural network.

[0113] The instruction fetch buffer 609 connected to the controller 604 is used to store the instructions used by the controller 604;

[0114] Unified memory 606, input memory 601, weighted memory 602, and instruction fetch memory 609 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0115] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the above. Figure 2 The procedure of the method.

[0116] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0123] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A white point estimation method, characterized in that, include: Acquire multimodal data, the multimodal data including a first multispectral image and hardware parameters of the imaging device, the first multispectral image being obtained by capturing a scene using the imaging device, and the hardware parameters being used to indicate the brightness of the light source in the scene; The multimodal data is processed to obtain white points, which are used for white balance processing of RGB images and / or second multispectral images.

2. The method according to claim 1, characterized in that, The process of processing the multimodal data to obtain white points includes: The first weight coefficient of each modality in the multimodal data is determined by a convolutional neural network; The multimodal data is processed based on multiple first weighting coefficients to obtain the white points.

3. The method according to claim 2, characterized in that, The step of determining the first weight coefficient of each modality in the multimodal data using a convolutional neural network includes: Based on the feature vector of each modality in the multimodal data, the second weight coefficient of each modality is obtained through the convolutional neural network; An attention matrix is ​​obtained based on the feature vector of each modality data, and the attention matrix is ​​used to represent the importance of each modality data in the multimodal data; The attention matrix and the second weight coefficient are calculated to obtain the first weight coefficient for each modality data.

4. The method according to claim 2 or 3, characterized in that, The step of processing the multimodal data according to multiple first weighting coefficients to obtain the white points includes: The multimodal data is processed according to multiple first weighting coefficients to obtain fused data; The fused data is processed using a neural network model to obtain the white points.

5. The method according to any one of claims 1 to 4, characterized in that, The multimodal data also includes one or more of the following: main camera image, scene category, scene segmentation map, and human detection bounding box; the main camera image is an image captured by the main camera sensor of the shooting device, the scene category is used to represent the category information of objects in the scene, and the scene segmentation map is used to represent the region information in the scene.

6. The method according to any one of claims 1 to 5, characterized in that, The hardware parameters include one or more of the following: ISO sensitivity, LV brightness, aperture size, and shutter speed.

7. A white point estimation device, characterized in that, include: The acquisition module is used to acquire multimodal data, which includes a first multispectral image and hardware parameters of the imaging device. The first multispectral image is obtained by capturing a scene using the imaging device, and the values ​​of the hardware parameters are used to represent the brightness of the light source in the scene. The processing module is used to process the multimodal data to obtain white points, which are used for white balance processing of the RGB image and / or the second multispectral image.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used for: The first weight coefficient of each modality in the multimodal data is determined by a convolutional neural network; The multimodal data is processed based on multiple first weighting coefficients to obtain the white points.

9. The apparatus according to claim 8, characterized in that, The processing module is specifically used for: Based on the feature vector of each modality in the multimodal data, the second weight coefficient of each modality is obtained through the convolutional neural network; An attention matrix is ​​obtained based on the feature vector of each modality data, and the attention matrix is ​​used to represent the importance of each modality data in the multimodal data; The attention matrix and the second weight coefficient are calculated to obtain the first weight coefficient for each modality data.

10. The apparatus according to claim 8 or 9, characterized in that, The processing module is specifically used for: The multimodal data is processed according to multiple first weighting coefficients to obtain fused data; The fused data is processed using a neural network model to obtain the white points.

11. The apparatus according to any one of claims 7 to 10, characterized in that, The multimodal data also includes one or more of the following: main camera image, scene category, scene segmentation map, and human detection bounding box; the main camera image is an image captured by the main camera sensor of the shooting device, the scene category is used to represent the category information of objects in the scene, and the scene segmentation map is used to represent the region information in the scene.

12. The apparatus according to any one of claims 7 to 11, characterized in that, The hardware parameters include one or more of the following: ISO sensitivity, LV brightness, aperture size, and shutter speed.

13. An electronic device, characterized in that, include: A processor for executing a program that causes the electronic device to perform the method as described in any one of claims 1 to 6.

14. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6.

15. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

16. A chip, characterized in that, It includes at least one processing unit and an interface circuit, the interface circuit being used to provide program instructions or data to the at least one processing unit, the at least one processing unit being used to execute the program instructions to implement the method of any one of claims 1 to 6.

Images