Image brightness adjusting method and device, equipment, medium and product

Abstract

The invention provides an image brightness adjusting method and device, equipment, a medium and a product, and the method comprises the steps: obtaining the image brightness features of an original image and the current illumination intensity value of light supplement equipment, and enabling the image brightness features to comprise features obtained through the brightness evaluation of the original image based on all dimensions, the dimensions comprise an image quality dimension, a semantic dimension, a spatial dimension, a global dimension and an image block dimension; based on the image brightness feature and a current illumination intensity value of the light supplementing device, determining a target illumination intensity value of the light supplementing device for performing illumination intensity adjustment on a next frame of image of the original image; and based on the target illumination intensity value, adjusting the illumination intensity value of the light supplementing equipment when the next frame of image is shot.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image brightness adjustment method, device, equipment, medium and product. Background Art

[0002] Image brightness refers to the brightness of the picture. Appropriate brightness adjustment can better display the details and colors in the image and improve the image clarity. Therefore, it is necessary to adjust the image brightness.

[0003] At present, the image quality score is mainly calculated based on image index values ​​(such as grayscale value and contrast, etc.), and the image quality score is used to determine whether the image brightness is in an appropriate range. If the image brightness is not in an appropriate range, the light intensity of the fill light device is adjusted according to the prior information to make the image brightness meet the expectations.

[0004] However, this image brightness adjustment method is relatively simple, and the image brightness adjustment effect is not good, resulting in poor image quality. Summary of the invention

[0005] Based on the above technical status, the present application provides an image brightness adjustment method, device, equipment, medium and product, which can improve the image brightness adjustment effect and enhance the image quality.

[0006] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a method for adjusting image brightness is provided, comprising: obtaining image brightness features of an original image and a current light intensity value of a fill light device, the image brightness features comprising features obtained by performing brightness evaluation on the original image based on various dimensions, the various dimensions comprising an image quality dimension, a semantic dimension, a spatial dimension, a global dimension, and an image block dimension; based on the image brightness features and the current light intensity value of the fill light device, determining a target light intensity value for the fill light device to adjust the light intensity of a next frame of the original image; based on the target light intensity value, adjusting the light intensity value of the fill light device when shooting the next frame of the image.

[0008] In some implementations, the step of acquiring image brightness features of an original image; and determining a target brightness value for brightness adjustment of a next frame image of the original image based on the image brightness features, comprises: inputting the original image into a brightness adjustment model, and acquiring the image brightness features of the original image by the brightness adjustment model; and determining a target brightness value for brightness adjustment of a next frame image of the original image based on the image brightness features and a current light intensity value of the fill light device.

[0009] In some implementations, the image brightness feature includes an image quality feature, the image quality feature corresponds to each image quality dimension, the brightness adjustment model includes an image quality feature extraction module, and the image quality feature extraction module includes each image quality assessment unit and a KAN network; wherein, the image brightness feature of the original image is obtained by the brightness adjustment model, including: obtaining the image quality assessment results of the original image in each image quality assessment dimension through the each image quality assessment unit; determining the weights corresponding to the image quality assessment results of the original image in each image quality assessment dimension through the KAN network, and fusing the image quality assessment results in each image quality assessment dimension based on the weights to obtain the image quality feature of the original image.

[0010] In some implementations, the image brightness features include semantic spatial features, and the semantic spatial features correspond to the semantic dimension and the spatial dimension. The brightness adjustment model includes a semantic spatial feature extraction module, and the semantic spatial feature extraction module includes a semantic feature extraction unit, a spatial feature extraction unit and a first fusion unit; wherein, the image brightness features of the original image are obtained by the brightness adjustment model, including: extracting the semantic features corresponding to the original image from the semantic dimension by the semantic feature extraction unit, and extracting the spatial features corresponding to the original image from the spatial dimension by the spatial feature extraction unit; and fusing the semantic features and the spatial features by the first fusion unit to obtain the semantic spatial features corresponding to the original image.

[0011] In some implementations, the first fusion unit includes a first fusion unit, a multiplication unit, and a second fusion unit; wherein, the semantic features and the spatial features are fused through the first fusion module to obtain the semantic spatial features corresponding to the original image, including: fusing the semantic features and the spatial features through the first fusion unit to obtain initial semantic spatial features; multiplying the initial semantic spatial features and the spatial features through the multiplication unit to obtain shallow semantic spatial features; fusing the initial semantic spatial features and the shallow semantic spatial features through the second fusion unit to obtain the semantic spatial features.

[0012] In some implementations, the image brightness feature includes an image sequence feature, the image sequence feature corresponds to the image block dimension, the brightness adjustment model includes an image sequence feature extraction module, and the image sequence feature extraction module includes an image block division unit and an image block visual feature extraction unit; wherein, obtaining the image brightness feature of the original image by the brightness adjustment model includes: dividing the semantic space feature into individual image blocks by the image block division unit; and extracting visual features from the image block dimension based on the semantic space features corresponding to each image block by the image block visual feature extraction unit to obtain the image sequence feature.

[0013] In some implementations, the image brightness feature includes a global feature, the brightness adjustment model includes a global feature extraction module, and the global feature extraction module includes a spatial pyramid pooling convolutional network, a hole convolutional network, an attention mechanism network and a third fusion unit; wherein, the image brightness feature of the original image is obtained by the brightness adjustment model, including: extracting an output feature of a preset size from the global dimension of the original image through the spatial pyramid pooling convolutional network; performing a convolution operation on the output feature from the global dimension through the hole convolutional network to obtain a hole convolution feature; determining the weights corresponding to the output feature of the preset size and the hole convolution feature respectively through the attention mechanism network, and fusing the output feature of the preset size and the hole convolution feature based on the weights corresponding to the output feature of the preset size and the hole convolution feature respectively to obtain an attention feature; fusing the attention feature and the hole convolution feature through the third fusion unit to obtain the global feature.

[0014] In some implementations, the brightness adjustment model is trained through the following steps: obtaining image brightness feature samples corresponding to the original image samples, current light intensity value samples of the fill light device and their corresponding training labels, the training labels including annotated brightness levels, annotated target brightness values ​​and annotated semantic segmentation maps; predicting the semantic segmentation map, current brightness levels and target light intensity values ​​to be adjusted of the original image samples based on the image brightness feature samples corresponding to the original image samples and the current light intensity value samples of the fill light device by the brightness adjustment model to be trained, to obtain predicted brightness levels, predicted target light intensity values ​​and predicted semantic segmentation maps; determining a comprehensive loss based on the difference between the predicted brightness levels and the annotated brightness levels, the difference between the predicted target light intensity values ​​and the annotated target light intensity values, and the difference between the predicted semantic segmentation maps and the annotated semantic segmentation maps, and converging the brightness adjustment model to be trained based on the comprehensive loss to obtain the brightness adjustment model.

[0015] In some implementations, the annotation brightness level and the annotation target brightness value are determined using the following steps: obtaining multiple image brightness scores corresponding to each of multiple training image samples, each of the multiple training image samples corresponding to a different light intensities, and the multiple training image samples and the original image sample are images taken of the same object; for each training image sample in the multiple training image samples, determining a comprehensive brightness score corresponding to the training image sample based on an average value of the multiple image brightness scores corresponding to the training image sample; selecting the light intensity value of the training image sample with the highest comprehensive brightness score among the multiple training image samples, determining it as the annotation target light intensity value corresponding to the original image sample, and determining the highest comprehensive brightness score as the annotation brightness level corresponding to the original image sample.

[0016] According to a second aspect of an embodiment of the present application, an image brightness adjustment device is provided, comprising: an acquisition unit, configured to acquire image brightness features of an original image and a current light intensity value of a fill light device, the image brightness features comprising features obtained by performing brightness evaluation on the original image based on various dimensions, the various dimensions comprising an image quality dimension, a semantic dimension, a spatial dimension, a global dimension and an image block dimension; a brightness adjustment determination unit, configured to determine a target light intensity value for the fill light device to perform light intensity adjustment on a next frame of the original image based on the image brightness features and the current light intensity value of the fill light device; and a brightness adjustment unit, configured to adjust the light intensity value of the fill light device when shooting the next frame of the original image based on the target light intensity value.

[0017] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the image brightness adjustment method as described in the first aspect by running the program in the memory.

[0018] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image brightness adjustment method as described in the first aspect is implemented.

[0019] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, cause the processor to execute: the image brightness adjustment method as described in the first aspect.

[0020] The embodiments of the present application provide an image brightness adjustment method, device, equipment, medium and product, which obtains the image brightness feature of the original image and the current light intensity value of the fill light device, wherein the image brightness feature includes features obtained by evaluating the brightness of the original image based on various dimensions, and the various dimensions include image quality dimension, semantic dimension, spatial dimension, global dimension and image block dimension; based on the image brightness feature and the current light intensity value of the fill light device, determines the target light intensity value for the fill light device to adjust the light intensity of the next frame of the original image; based on the target light intensity value, adjusts the light intensity value of the fill light device when shooting the next frame of the image. Since the image brightness feature is obtained by comprehensively evaluating the original image based on various dimensions, richer and more comprehensive feature information can be obtained, which can not only improve the prediction accuracy of the target light intensity value of the next frame of the image, improve the image brightness adjustment effect, but also improve the imaging quality of the image, thereby helping to improve the subsequent image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0022] Figure 1 A flowchart of an image brightness adjustment method provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of a brightness adjustment model provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the framework of the image quality feature extraction module provided in the embodiment of the present application;

[0025] Figure 4 A schematic diagram of the framework of the semantic space feature extraction module provided in the embodiment of the present application;

[0026] Figure 5 A schematic diagram of the framework of the first fusion unit provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the framework of the image sequence feature extraction module provided in the embodiment of the present application;

[0028] Figure 7 A schematic diagram of the framework of the global feature extraction module provided in the embodiment of the present application;

[0029] Figure 8A schematic diagram of the framework of the fusion module provided in the embodiment of the present application;

[0030] Fig. 9 A schematic diagram of a process flow of a brightness adjustment model training process provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of the structure of an image brightness adjustment device provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solution proposed in the embodiment of the present application is applicable to any application scenario that requires image brightness adjustment, such as medical imaging, industrial detection, photography post-processing, news photography, mobile photography, video production, scientific photography, etc. The technical solution of the embodiment of the present application can improve the image brightness adjustment effect in these application scenarios, thereby improving the image quality and subsequent image processing effect.

[0034] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] Before introducing this application solution, the relevant technologies are first introduced:

[0037] Image brightness is the degree of brightness of the pixels in an image, which determines the overall brightness of the image. Insufficient brightness will result in loss of details in dark areas, while excessive brightness may cause glare. Adjusting image brightness can change the details of the bright or dark areas of the image, thereby improving image quality and subsequent image processing effects.

[0038] In medical imaging and industrial inspection scenarios, an endoscope is needed to observe the internal conditions of the observed object. When using an endoscope, the lens needs to be inserted deep into the observed object, so the image quality is very sensitive to lighting conditions. In such a low-light environment, the fill light effect directly affects the user experience. If the light intensity is too high, the object will reflect light to a large extent, making it difficult for the user to observe the actual original appearance of the observed object through the endoscope. If the light intensity is too low, it is difficult to obtain image detail information because the image is too dark.

[0039] At present, the main method is to calculate the image brightness score based on traditional evaluation methods such as grayscale value, contrast and other image indicators, and then determine whether the image brightness is in an appropriate range based on the score, and adjust the lighting of the fill light device based on prior information to make the image brightness meet the observer's expectations. This method relies on prior information and is often only effective for some workpieces. It has poor robustness and it is difficult to achieve accurate results in the dimming process.

[0040] In addition, some dimming algorithms use object recognition as a dimming reference object. This method divides the dimming into two stages, making it difficult to achieve real-time dimming operations and is not universal.

[0041] In addition, the dimming algorithm needs to be continuously iterated, cross-calculating the image quality score and dimming operation in real time, and the dimming value needs to rely on the image evaluation operation of the previous frame. This method will double the overall dimming time and make it difficult to achieve the effect of real-time dimming.

[0042] In view of this, the embodiments of the present application are dedicated to providing an image brightness adjustment method, device, equipment and storage medium, which can improve the image brightness adjustment effect, and are described in detail one by one in the following embodiments.

[0043] Exemplary Methods

[0044] Figure 1 This is a flow chart of a method for adjusting image brightness provided by an embodiment of the present application. Figure 1 As shown, the image brightness adjustment method provided in this embodiment includes steps S101 to S103:

[0045] S101, obtaining image brightness characteristics of an original image and a current light intensity value of a fill light device.

[0046] The image brightness feature includes features obtained by evaluating the brightness of the original image based on various dimensions, including image quality dimension, semantic dimension, spatial dimension, global dimension and image block dimension. Therefore, the image brightness feature includes image quality features corresponding to the image quality dimension, semantic space features corresponding to the semantic dimension and the spatial dimension, global features corresponding to the global dimension and image sequence features corresponding to the image block dimension.

[0047] The fill light device provides illumination for the image acquisition device to capture the original image. The current illumination intensity value of the fill light device is the illumination intensity value when the image acquisition device captures the original image. The original image is the image currently captured by the image acquisition device.

[0048] In some embodiments, step S101 may be implemented by a brightness adjustment model. Specifically, step S101 includes: inputting the original image into the brightness adjustment model, and obtaining the image brightness features of the original image by the brightness adjustment model.

[0049] The process of implementing step S101 through the brightness adjustment model is described in detail below with reference to the accompanying drawings:

[0050] Figure 2 This is a schematic diagram of the framework of the brightness adjustment model provided in the embodiment of the present application. Figure 2 As shown, the brightness adjustment model includes an image quality feature extraction module 21, a semantic space feature extraction module 22, an image sequence feature extraction module 23, a global feature extraction module 24, a current light intensity acquisition module 25, a fusion module 26 and a classification module 27.

[0051] The image quality feature extraction module 21 is used to obtain the image quality feature Fsck1 of the original image; the semantic space feature extraction module 22 is used to obtain the semantic space feature F AB ; Image sequence feature extraction module 23, used to obtain the image sequence feature Fvbb of the original image; global feature extraction module 24, used to obtain the global feature Fc2 of the original image; current light intensity acquisition module 25, used to obtain the current light intensity value of the fill light device; fusion module 26, used to image quality feature Fsck1, semantic space feature F AB , the image sequence feature Fvbb, the global feature Fc2 and the current light intensity value are merged to obtain the comprehensive image feature Fscma; the classification module 27 is used to determine the target light intensity value for the next frame image of the original image to be adjusted by the fill light device based on the comprehensive image feature Fscma.

[0052] The following is a detailed introduction of each module in conjunction with the attached drawings:

[0053] Figure 3Schematic diagram of the framework of the image quality feature extraction module provided in the embodiment of the present application. Figure 3 As shown, the image quality feature extraction module 21 includes various image quality assessment units 211 and a KAN network 212; wherein each image quality assessment unit 211 is used to obtain the image quality assessment results of the original image under various image quality assessment dimensions; the KAN network 212 is used to determine the weights corresponding to the image quality assessment results of the original image under various image quality assessment dimensions, and fuse the image quality assessment results under various image quality assessment dimensions based on the weights to obtain the image quality features of the original image.

[0054] In some embodiments, the image quality feature extraction module 21 may further include a fusion unit and an activation function unit. The fusion unit is used to fuse the image quality assessment results under each image quality assessment dimension in a concatenated manner to obtain a fused image quality assessment result Fc1, and enable the KAN network 212 to determine the weights corresponding to the image quality assessment results of the original image under each image quality assessment dimension based on the fused image quality assessment result Fc1, and fuse the image quality assessment results under each image quality assessment dimension based on the weights to obtain a fused image quality feature Fck1 of the original image. The activation function unit is used to activate the fused image quality feature Fck1 to obtain the image quality feature Fsck1.

[0055] Among them, each image quality assessment unit 211 includes an average grayscale value assessment unit, a local exposure value assessment unit, a color richness value assessment unit and a contrast assessment unit, which are respectively used to obtain the average grayscale value, local exposure value, color richness value and contrast of the original image as image quality assessment results under each image quality assessment dimension.

[0056] based on Figure 3 The framework schematic diagram of the image quality feature extraction module is shown. The image brightness feature of the original image is obtained by the brightness adjustment model, including: obtaining the image quality assessment results of the original image in each image quality assessment dimension through each image quality assessment unit 211; determining the weights corresponding to the image quality assessment results of the original image in each image quality assessment dimension through the KAN network 212, and fusing the image quality assessment results in each image quality assessment dimension based on the weights to obtain the image quality feature of the original image.

[0057] In this embodiment, the image quality feature Fsck1 represents the degree of quantification of the brightness effect of the original image by using each image quality assessment dimension to evaluate the brightness of the original image. The KAN network can automatically assign weights to each image quality assessment dimension, so that the model learns the importance of each image quality assessment dimension, thereby improving the accuracy of image quality feature extraction.

[0058] Figure 4 The schematic diagram of the framework of the semantic space feature extraction module provided in the embodiment of the present application is as follows. Figure 4 As shown, the semantic spatial feature extraction module 22 includes a semantic feature extraction unit 221, a spatial feature extraction unit 222 and a first fusion unit 223; the semantic feature extraction unit 221 is used to obtain the semantic feature F of the original image. DPR ; Spatial feature extraction unit 222, used to obtain the spatial feature F of the original image CBS ; The first fusion unit 223 is used to fusion the semantic feature F DPR and spatial feature F CBS Fusion is performed to obtain the semantic space feature F corresponding to the original image AB .

[0059] Specifically, the semantic feature extraction unit (DPR Block) 221 includes a depthwise convolution block (DepthwiseConv Block), a pointwise convolution block (PointwiseConv Block) and a RepVGG Block connected in sequence, which are used to perform multiple convolution operations on the original image in sequence to obtain semantic features. Among them, Depthwise Convolution performs convolution on each channel, which can be regarded as a grouped convolution of one channel and one group. It uses a convolution kernel for each channel of the input original image, and then splices the outputs of all convolution kernels. Pointwise Convolution is a 1*1 convolution, which can freely change the number of output channels and perform channel fusion on the features output by Depthwise Convolution. PointwiseConv Block uses 1*1 convolution to convolve the features output by Depthwise Convolution point by point, which is used to change the output dimension of the features output by DepthwiseConv Block and reduce the number of parameters. RepVGG Block includes multiple building blocks, each of which contains a convolution layer and a normalization (BatchNorm) layer, as well as a ReLU activation function. The RepVGG network is a lightweight convolutional neural network architecture for image classification. Through structural parameter reconstruction technology (Structural Re-parameterization), the network adopts different structures in the training and reasoning stages, thereby improving the reasoning speed while maintaining high accuracy.

[0060] In this embodiment, the semantic feature extraction unit 221 can extract the region of interest by performing semantic segmentation on the original image, thereby improving the model's sensitivity to the region of interest. In this process, if the area of ​​the region of interest is less than a preset area threshold, or the detection result is a new object, the radius R is set according to different scenes to re-divide the central area as the region of interest.

[0061] The spatial feature extraction unit 222 may include multiple CBS Blocks connected in series, which are used to perform multiple convolution operations on the original image to obtain spatial features. Each CBS Block includes a convolution layer (Conv), a normalization layer (Batchnormalization, BN) and a SiLU activation function. Take three CBS connected in series as an example. The first CBS can be a 1*1 convolution, which is used to change the number of channels. The second CBS can be a 3*3 convolution with a step size of 1, which is used for feature extraction. The third CBS can be a 3*3 convolution with a step size of 1, which is used for downsampling.

[0062] based on Figure 4 The framework schematic diagram of the semantic spatial feature extraction module is shown. The image brightness feature of the original image is obtained by the brightness adjustment model, including: extracting the semantic feature corresponding to the original image from the semantic dimension by the semantic feature extraction unit, and extracting the spatial feature corresponding to the original image from the spatial dimension by the spatial feature extraction unit; fusing the semantic feature and the spatial feature by the first fusion unit to obtain the semantic spatial feature corresponding to the original image.

[0063] Figure 5 This is a schematic diagram of the framework of the first fusion unit provided in the embodiment of the present application. Figure 5 As shown, the first fusion unit 223 includes a first fusion subunit 2231, a multiplication unit 2232 and a second fusion subunit 2233; wherein the first fusion subunit 2231 is used to use a concatenation fusion method to combine the semantic features F DPR and spatial feature F CBS Fusion is performed to obtain the initial semantic space feature F SS ; Multiplication (MUL) subunit 2232, used for initial semantic space feature F SS and spatial feature F CBS Multiply them to get the shallow semantic space feature F MSS The second fusion subunit 2233 is used to use the ADD method to fusion the initial semantic space feature F SS and shallow semantic space features F MSS Fusion is performed to obtain the semantic space feature F AB .

[0064] In some embodiments, a Sigmoid activation function may be set between the first fusion subunit 2231 and the multiplication subunit 2232. After the spatial features are activated, the activated spatial features are input into the multiplication subunit 2232 for multiplication with the initial semantic spatial features.

[0065] In addition, a global average pooling layer, a convolution layer, a normalization layer, and an activation layer are sequentially arranged between the first fusion subunit 2231 and the second fusion subunit 2233, which are respectively used to perform global average pooling operation, 1*1 convolution operation, normalization operation, and activation operation on the initial semantic space features, so that the initial semantic space features after the activation operation are input into the second fusion subunit 2233 to continue to be fused with the shallow semantic space features to obtain the semantic space features.

[0066] Specifically, the information of semantic features and spatial features carried in the initial semantic spatial features, shallow semantic spatial features, and semantic spatial features gradually increases.

[0067] In this embodiment, the initial semantic spatial feature F is SS Perform multiplication guidance to obtain the shallow semantic space feature F MSS , and the initial semantic space feature F is modified by ADD SS and shallow semantic space features F MSS Fusion can improve the richness of image information representation.

[0068] In some embodiments, the semantic spatial feature extraction module 22 may further include an upsampling unit (including an upsampling block and a deconvolution block) 224, which is connected to the first fusion unit 223, and is used to upsample the fused features after the semantic features and the spatial features are fused to obtain the semantic spatial features corresponding to the original image.

[0069] based on Figure 5 The schematic diagram of the framework of the first fusion unit is shown. The semantic features and the spatial features are fused by the first fusion unit to obtain the semantic spatial features corresponding to the original image, including: fusing the semantic features and the spatial features by the first fusion subunit to obtain the initial semantic spatial features; multiplying the initial semantic spatial features and the spatial features by the multiplication subunit to obtain the shallow semantic spatial features; fusing the initial semantic spatial features and the shallow semantic spatial features by the second fusion subunit to obtain the semantic spatial features.

[0070] Figure 6 Schematic diagram of the framework of the image sequence feature extraction module provided in the embodiment of the present application. Figure 6 As shown, the image sequence feature extraction module 23 includes an image block division unit 231 and an image block visual feature extraction unit 232; wherein the image block division unit 231 is used to divide the semantic space features into individual image blocks; the image block visual feature extraction unit 232 is used to extract visual features from the image block dimension based on the semantic space features corresponding to each image block, so as to obtain image sequence features.

[0071] based on Figure 6 The framework schematic diagram of the image sequence feature extraction module is shown. The image brightness feature of the original image is obtained by the brightness adjustment model, including: dividing the semantic space feature into each image block by the image block division unit; extracting the visual feature from the image block dimension based on the semantic space feature corresponding to each image block by the image block visual feature extraction unit to obtain the image sequence feature.

[0072] The image block visual feature extraction unit 232 includes various visual basic models (Vision Mamba, VIM) and a third fusion sub-unit.

[0073] Continue reading Figure 6Taking 4 VIMs as an example, the image block visual feature extraction unit 232 includes a first visual basis model 2321, a second visual basis model 2322, a third visual basis model 2323, a third fusion subunit 2324 and a fourth visual basis model 2325. The first visual basis model 2321 is used to extract the semantic space features corresponding to each image block to obtain the first image sequence feature Fvm1; the second visual basis model 2322 is used to extract the second image sequence feature Fvm2 based on the first image sequence feature Fvm1; the third visual basis model 2324 is used to extract the third image sequence feature Fvm3 based on the second image sequence feature Fvm2; the third fusion subunit 2323 is used to fuse the first image sequence feature Fvm1 and the third image sequence feature Fvm3 to obtain the fused image sequence feature Fvma; the fourth visual basis model 2325 is used to extract the image sequence feature Fvbb based on the fused image sequence feature Fvma.

[0074] Among them, the richness of the feature information contained in the first image sequence feature Fvm1, the second image sequence feature Fvm2, the third image sequence feature Fvm3, the fused image sequence feature Fvma and the image sequence feature Fvbb gradually increases.

[0075] In some embodiments, the image sequence feature extraction module 23 may also include a Resnet50 network, which is located before the image block division unit 231 and connected to the image block division unit 231, and is used to extract features from the semantic space features to obtain semantic space features with richer feature information, and input them into the image block division unit 231 for image block division, and input them into the image block visual feature extraction unit 232 to extract visual features based on the semantic space features corresponding to each image block to obtain image sequence features.

[0076] In this embodiment, the visual basis model VIM can effectively process the global context of the image by introducing the bidirectional SSM (State Space Model), so that it can more accurately capture the details and context information in the image, and thus can extract image features with richer feature information.

[0077] Figure 7 Schematic diagram of the framework of the global feature extraction module provided in the embodiment of the present application. Figure 7As shown, the global feature extraction module 24 includes a spatial pyramid pooling convolution network 241, a dilated convolution network 242, an attention mechanism network 243 and a third fusion unit 244; wherein the spatial pyramid pooling convolution network (SPPnet) 241 is used to extract an output feature Fspp of a preset size from the original image from a global dimension; the dialeted convolution network (dialeted conv) 242 is used to perform a convolution operation on the output feature Fspp from a global dimension to obtain a dilated convolution feature Fdc; the attention mechanism network 243 is used to determine the weights corresponding to the output feature Fspp of a preset size and the dilated convolution feature Fdc, and fuse the output feature Fspp of a preset size and the dilated convolution feature Fdc based on the weights corresponding to the output feature Fspp of a preset size and the dilated convolution feature Fdc to obtain the attention feature Fsdcc; the third fusion unit 244 is used to fuse the attention feature Fsdcc and the dilated convolution feature Fdc to obtain the global feature Fvbb.

[0078] Among them, the third fusion unit 244 is used to fuse the attention feature Fsdcc and the hole convolution feature Fdc by using the ADD method to obtain the global feature Fvbb.

[0079] Among them, the output feature Fspp of preset size, the hole convolution feature Fdc, the attention feature Fsdcc and the global feature Fvbb are all extracted from the overall perspective of the image, and the richness of the extracted features gradually increases.

[0080] In some embodiments, a fusion unit (ADD) 245 can also be set between the dilated convolution network 242 and the attention mechanism network 243 to perform ADD fusion on the output feature Fspp of a preset size and the dilated convolution feature Fdc, and then input them into the attention mechanism network 243 to extract the attention feature Fsdcc.

[0081] based on Figure 7 The framework schematic diagram of the global feature extraction module shown in the figure obtains the image brightness features of the original image by the brightness adjustment model, including: extracting output features of a preset size from the global dimension of the original image through a spatial pyramid pooling convolution network; performing a convolution operation on the output features from the global dimension through a dilated convolution network to obtain dilated convolution features; determining the weights corresponding to the output features of the preset size and the dilated convolution features respectively through an attention mechanism network, and fusing the output features of the preset size and the dilated convolution features based on the weights corresponding to the output features of the preset size and the dilated convolution features respectively to obtain attention features; fusing the attention features and the dilated convolution features through a third fusion unit to obtain global features.

[0082] In this embodiment, the attention feature Fsdcc and the dilated convolution feature Fdc are fused by the third fusion unit in an ADD manner, and the fusion unit set between the dilated convolution network (dialeted conv) and the attention mechanism network 243 performs ADD fusion on the output feature Fspp of a preset size and the dilated convolution feature Fdc, and then inputs them into the attention mechanism network to extract the attention feature Fsdcc, which is a residual mechanism to ensure that the model prevents the disappearance of knowledge memory during the reasoning process, and can also prevent gradient explosion.

[0083] Continue reading Figure 1 The image brightness adjustment method of this embodiment also includes the following step S102.

[0084] S102: Determine a target illumination intensity value for adjusting illumination intensity of a next frame of the original image by the fill light device based on image brightness characteristics and a current illumination intensity value of the fill light device.

[0085] In this embodiment, the model can automatically determine the target light intensity value for the fill light device to adjust the light intensity of the next frame of the original image based on the current light intensity value of the fill light device, the learned light intensity value of the fill light device corresponding to the original image, and the mapping relationship between the image brightness characteristics and the light intensity value to be adjusted.

[0086] In some embodiments, step S102 may be implemented by a brightness adjustment model. Specifically, step S102 includes: determining a target illumination intensity value for the next frame of the original image to be adjusted by the fill light device based on the image brightness feature and the current illumination intensity value of the fill light device by the brightness adjustment model.

[0087] Figure 8 This is a schematic diagram of the framework of the fusion module provided in the embodiment of the present application. Figure 8 As shown, the fusion module 26 includes a fourth fusion unit 261, a fifth fusion unit 262, a sixth fusion unit 263 and a seventh fusion unit 264; wherein the fourth fusion unit 261 is used to combine the global feature Fc2 and the semantic space feature F ABA concatenation method is used to fuse the first fusion feature Fcms; the fifth fusion unit 262 is used to fuse the first fusion feature Fcms and the image sequence feature Fvbb in a concatenation method to obtain the second fusion feature Fcmwc; the sixth fusion unit 263 is used to multiply the second fusion feature Fcmwc and the current light intensity value to obtain the third fusion feature Fscmk; the seventh fusion unit 264 is used to fuse the third fusion feature Fscmk and the image quality feature Fsck1 in an ADD method to obtain the comprehensive image feature Fscma.

[0088] After the comprehensive image feature Fscma is obtained, it can be input into the classification module 27 to determine the target illumination intensity value for adjusting the illumination intensity value of the next frame of the original image by the fill light device based on the comprehensive image feature Fscma.

[0089] In this embodiment, the global feature is the overall image feature by combining the semantic space feature F AB Concatenating and fusing with global features can better fuse the global and region of interest features. In addition, fusing the features obtained by fusing the global features and the semantic space features of the region of interest with the image sequence features can further improve the richness of the image brightness features.

[0090] S103: Based on the target light intensity value, adjust the light intensity value of the fill light device when capturing the next frame of image.

[0091] This step is to adjust the light intensity value of the fill light device to the target light intensity value when shooting the next frame of image, so as to achieve the purpose of adjusting the image brightness and improve the image brightness adjustment effect.

[0092] After adjusting the illumination intensity value of the next frame of image, the next frame of image can be used as the original image, and steps S101 to S103 are re-executed and iterated in sequence, thereby achieving the effect of adjusting the brightness of each frame of image.

[0093] The above embodiment introduces the reasoning and application process of the brightness adjustment model. Before the reasoning and application, the model needs to be trained. The training process of the brightness adjustment model will be introduced below in conjunction with the accompanying drawings.

[0094] Fig. 9 The flowchart of the training process of the brightness adjustment model provided in the embodiment of the present application is as follows. Fig. 9 As shown, the training process of the brightness adjustment model includes the following steps S901 to S903;

[0095] S901: Obtain image brightness feature samples corresponding to original image samples, current light intensity value samples of a fill light device, and their corresponding training labels.

[0096] Among them, the training labels include labeled brightness levels, labeled target light intensity values, and labeled semantic segmentation maps.

[0097] In some embodiments, the marked brightness level and the marked target illumination intensity value are determined using the following steps a1 to a3:

[0098] Step a1: Obtain multiple image brightness scores corresponding to each of the multiple training image samples.

[0099] The multiple training image samples each correspond to a different light intensity, and the multiple training image samples and the original image sample are images taken of the same object.

[0100] Step a2: for each training image sample among the multiple training image samples, determine the comprehensive brightness score corresponding to the training image sample based on the average value of the multiple image brightness scores corresponding to the training image sample.

[0101] Step a3: Select the illumination intensity value of the training image sample with the highest comprehensive brightness score from multiple training image samples, determine it as the labeled target illumination intensity value corresponding to the original image sample, and determine the highest comprehensive brightness score as the labeled brightness level corresponding to the original image sample.

[0102] For example, by setting different light intensity values ​​for the same object and taking multiple images under different light intensities, multiple training image samples are obtained. Then, a suitable light intensity photo is selected as a reference image anchor, and the brightness of the reference image is set to 0.5. At the same time, multiple labelers compare the brightness of multiple training image samples with the reference image, and score the multiple training image samples according to the comparison results.

[0103] For example, taking one labeler among multiple labelers as an example, for each training image sample among multiple training image samples, if the brightness of the training image sample is better than the brightness of the reference image, a first score (such as "1") is set for the training image sample; if the brightness of the training image sample is worse than the brightness of the reference image, a second score (such as "0") is set for the training image sample. Finally, the comprehensive brightness score of the training image sample is obtained by dividing the total score of the training image sample by the total number of labelers.

[0104] In this way, the comprehensive brightness scores corresponding to the multiple training image samples can be obtained. Then, the illumination intensity value of the fill light device of the training image sample corresponding to the highest comprehensive brightness score is selected as the labeled target illumination intensity value, and the comprehensive brightness score is determined as the labeled brightness level.

[0105] This embodiment adopts a comparative scoring method instead of a direct scoring method, which can avoid the defect that it is difficult to accurately give an image brightness score based on the human eye alone, improve the accuracy of the labeled brightness level, and further assist in model training, thereby improving the model accuracy and image brightness adjustment effect.

[0106] S902. Predict the semantic segmentation map corresponding to the original image sample, the current brightness level and the target light intensity value to be adjusted based on the image brightness feature samples corresponding to the original image sample and the current light intensity value samples of the fill light device through the brightness adjustment model to be trained, and obtain the predicted brightness level, the predicted target light intensity value and the predicted semantic segmentation map.

[0107] Wherein, step S902 includes: obtaining image brightness feature samples corresponding to the original image samples through the brightness adjustment model to be trained. The image brightness feature samples include features obtained by performing brightness evaluation on the original image samples based on various dimensions, and the various dimensions include image quality dimension, semantic dimension, spatial dimension, global dimension and image block dimension. Therefore, the image brightness feature samples include image quality feature samples corresponding to the image quality dimension, semantic space feature samples corresponding to the semantic dimension and the spatial dimension, global feature samples corresponding to the global dimension and image sequence feature samples corresponding to the image block dimension.

[0108] Here, the current illumination intensity value sample of the fill light device is intended to enable the model to learn the mapping relationship between the illumination intensity value of the fill light device and the image brightness, so as to output the illumination intensity value that needs to be adjusted for the fill light device.

[0109] Specifically, image brightness feature samples corresponding to the original image samples are obtained through the brightness adjustment model to be trained, including: inputting the original image samples into the brightness adjustment model to be trained, and obtaining the image brightness feature samples of the original image samples by the brightness adjustment model to be trained; based on the image brightness feature samples corresponding to the original image samples and the current light intensity value samples of the fill light device, a semantic segmentation map corresponding to the original image samples, the current brightness level and the target light intensity value to be adjusted are predicted.

[0110] In this embodiment, the network structure of the brightness adjustment model to be trained can refer to the network structure of the brightness adjustment model in the above-mentioned embodiment.

[0111] Based on the network structure of the brightness adjustment model to be trained, the image quality feature samples, semantic space feature samples, global feature samples and image sequence feature samples can be obtained as follows:

[0112] In some embodiments, quality feature samples are obtained through the brightness adjustment model to be trained, including: obtaining image quality assessment results of the original image samples in each image quality assessment dimension through each image quality assessment unit; determining the weights corresponding to the image quality assessment results of the original image in each image quality assessment dimension through a KAN network, and fusing the image quality assessment results in each image quality assessment dimension based on the weights to obtain image quality feature samples of the original image samples.

[0113] In some embodiments, semantic spatial feature samples are obtained through the brightness adjustment model to be trained, including: extracting semantic feature samples corresponding to the original image from the semantic dimension through a semantic feature extraction unit, and extracting spatial feature samples corresponding to the original image from the spatial dimension through a spatial feature extraction unit; fusing the semantic feature samples and the spatial feature samples through a first fusion unit to obtain semantic spatial feature samples corresponding to the original image samples.

[0114] Among them, the semantic feature samples and the spatial feature samples are fused by the first fusion unit to obtain the semantic spatial feature samples corresponding to the original image samples, including: fusing the semantic features and the spatial features by the first fusion unit to obtain the initial semantic spatial features; multiplying the initial semantic spatial features and the spatial features by the multiplication unit to obtain the shallow semantic spatial features; fusing the initial semantic spatial features and the shallow semantic spatial features by the second fusion unit to obtain the semantic spatial features.

[0115] In some embodiments, global feature samples are obtained through the brightness adjustment model to be trained, including: extracting output feature samples of a preset size from the original image samples from the global dimension through a spatial pyramid pooling convolution network; performing a convolution operation on the output feature samples from the global dimension through a dilated convolution network to obtain dilated convolution feature samples; determining the weights corresponding to the output feature samples of the preset size and the dilated convolution feature samples through an attention mechanism network, and fusing the output feature samples of the preset size and the dilated convolution feature samples based on the weights corresponding to the output feature samples of the preset size and the dilated convolution feature samples to obtain attention feature samples; fusing the attention feature samples and the dilated convolution feature samples through a third fusion unit to obtain global feature samples.

[0116] In some embodiments, image sequence feature samples are obtained through the brightness adjustment model to be trained, including: dividing the semantic space feature samples into individual image blocks through an image block division unit; performing visual feature extraction from the image block dimension based on the semantic space feature samples corresponding to each image block through an image block visual feature extraction unit to obtain image sequence feature samples.

[0117] S903. Determine a comprehensive loss based on the difference between the predicted brightness level and the annotated brightness level, the difference between the predicted target light intensity value and the annotated target light intensity value, and the difference between the predicted semantic segmentation map and the annotated semantic segmentation map, and converge the brightness adjustment model to be trained based on the comprehensive loss to obtain a brightness adjustment model.

[0118] In this embodiment, the labeled semantic segmentation map is half the size of the original image sample, and the predicted semantic segmentation map is the predicted semantic space feature sample, which is half the size of the original image sample.

[0119] Among them, the comprehensive loss can be expressed as the following formula:

[0120] Loss total =αLoss objectivity +βLoss comprehensive +γMSE;

[0121] In the formula, Loss total Loss objectivity Indicates the difference between the predicted brightness and the labeled brightness; Loss comprehensive It represents the difference between the predicted target brightness value and the labeled target brightness value; MSE represents the difference between the predicted semantic segmentation map and the labeled semantic segmentation map.

[0122] Among them, α can be 0.3, β can be 0.5, and γ can be 0.2, which respectively represent the weights of each loss.

[0123] The prediction error of the model can be quantified by calculating the difference between the predicted brightness and the annotated brightness, the difference between the predicted target light intensity and the annotated target light intensity, and the difference between the predicted semantic segmentation map and the annotated semantic segmentation map. This loss value not only reflects the prediction ability of the model in its current state, but also serves as an important basis for guiding the subsequent learning direction of the model.

[0124] In order to minimize this prediction loss, we can use optimization algorithms such as gradient descent to gradually adjust the parameters in the model so that the model can better fit the image brightness features in the training data set. As the training process continues, the comprehensive loss will gradually decrease, indicating that the model's prediction accuracy for the image target brightness value is improving. When the model's prediction accuracy reaches the preset accuracy or the change in the comprehensive loss value tends to be stable, the model is considered to have converged, and the model obtained at this time is the brightness adjustment model that has been optimized and trained.

[0125] Exemplary Devices

[0126] Corresponding to the above-mentioned image brightness adjustment method, the embodiment of the present application also provides an image brightness adjustment device. Fig.10 Schematic diagram of the structure of an image brightness adjustment device provided in an embodiment of the present application. Fig.10 As shown, the image brightness adjustment device provided by the embodiment of the present application includes: an acquisition unit 1101, a brightness adjustment prediction unit 1102 and a brightness adjustment unit 1103; wherein the acquisition unit 1101 is used to acquire the image brightness characteristics of the original image and the current light intensity value of the fill light device, the image brightness characteristics include characteristics obtained by performing brightness evaluation on the original image based on various dimensions, and the various dimensions include image quality dimension, semantic dimension, spatial dimension, global dimension and image block dimension; the brightness adjustment prediction unit 1102 is used to determine, based on the image brightness characteristics and the current light intensity value of the fill light device, a target light intensity value for the fill light device to perform light intensity adjustment on the next frame of the original image; the brightness adjustment unit 1103 is used to adjust the light intensity value of the fill light device when shooting the next frame of the image based on the target light intensity value.

[0127] In some embodiments, the acquisition unit 1101 acquires image brightness features of the original image; the brightness adjustment prediction unit 1102 determines a target light intensity value for the fill light device to adjust the light intensity of the next frame of the original image based on the image brightness features and the current light intensity value of the fill light device, including: inputting the original image into a brightness adjustment model, and obtaining the image brightness features of the original image by the brightness adjustment model; and determining a target light intensity value for the fill light device to adjust the light intensity of the next frame of the original image based on the image brightness features and the current light intensity value of the fill light device.

[0128] In some embodiments, the image brightness feature includes an image quality feature, the image quality feature corresponds to each image quality dimension, the brightness adjustment model includes an image quality feature extraction module, and the image quality feature extraction module includes each image quality assessment unit and a KAN network; wherein, the image brightness feature of the original image is obtained by the brightness adjustment model, including: obtaining the image quality assessment results of the original image in each image quality assessment dimension through each image quality assessment unit; determining the weights corresponding to the image quality assessment results of the original image in each image quality assessment dimension through the KAN network, and fusing the image quality assessment results in each image quality assessment dimension based on the weights to obtain the image quality features of the original image.

[0129] In some embodiments, the image brightness features include semantic spatial features, and the semantic spatial features correspond to the semantic dimension and the spatial dimension. The brightness adjustment model includes a semantic spatial feature extraction module, and the semantic spatial feature extraction module includes a semantic feature extraction unit, a spatial feature extraction unit and a first fusion unit; wherein, the image brightness features of the original image are obtained by the brightness adjustment model, including: extracting the semantic features corresponding to the original image from the semantic dimension by the semantic feature extraction unit, and extracting the spatial features corresponding to the original image from the spatial dimension by the spatial feature extraction unit; and fusing the semantic features and the spatial features by the first fusion unit to obtain the semantic spatial features corresponding to the original image.

[0130] In some embodiments, the first fusion unit includes a first fusion unit, a multiplication unit, and a second fusion unit; wherein, the semantic features and the spatial features are fused through the first fusion module to obtain the semantic spatial features corresponding to the original image, including: fusing the semantic features and the spatial features through the first fusion unit to obtain initial semantic spatial features; multiplying the initial semantic spatial features and the spatial features through the multiplication unit to obtain shallow semantic spatial features; fusing the initial semantic spatial features and the shallow semantic spatial features through the second fusion unit to obtain the semantic spatial features.

[0131] In some embodiments, the image brightness feature includes an image sequence feature, the image sequence feature corresponds to the image block dimension, the brightness adjustment model includes an image sequence feature extraction module, and the image sequence feature extraction module includes an image block division unit and an image block visual feature extraction unit; wherein, the image brightness feature of the original image is obtained by the brightness adjustment model, including: dividing the semantic space feature into individual image blocks by the image block division unit; and extracting visual features from the image block dimension based on the semantic space features corresponding to each image block by the image block visual feature extraction unit to obtain the image sequence feature.

[0132] In some embodiments, the image brightness feature includes a global feature, the brightness adjustment model includes a global feature extraction module, and the global feature extraction module includes a spatial pyramid pooling convolutional network, a hole convolutional network, an attention mechanism network and a third fusion unit; wherein, the image brightness feature of the original image is obtained by the brightness adjustment model, including: extracting an output feature of a preset size from the global dimension of the original image through the spatial pyramid pooling convolutional network; performing a convolution operation on the output feature from the global dimension through the hole convolutional network to obtain a hole convolution feature; determining the weights corresponding to the output feature of the preset size and the hole convolution feature respectively through the attention mechanism network, and fusing the output feature of the preset size and the hole convolution feature based on the weights corresponding to the output feature of the preset size and the hole convolution feature respectively to obtain an attention feature; fusing the attention feature and the hole convolution feature through the third fusion unit to obtain the global feature.

[0133] In some embodiments, the brightness adjustment model is trained by the following steps: obtaining image brightness feature samples corresponding to the original image samples, current light intensity value samples of the fill light device and their corresponding training labels, wherein the training labels include annotated brightness levels, annotated target brightness values ​​and annotated semantic segmentation maps; predicting the semantic segmentation map, current brightness levels and target light intensity values ​​to be adjusted of the original image samples based on the image brightness feature samples corresponding to the original image samples and the current light intensity value samples of the fill light device by the brightness adjustment model to be trained, to obtain predicted brightness levels, predicted target light intensity values ​​and predicted semantic segmentation maps; determining a comprehensive loss based on the difference between the predicted brightness levels and the annotated brightness levels, the difference between the predicted target light intensity values ​​and the annotated target light intensity values, and the difference between the predicted semantic segmentation maps and the annotated semantic segmentation maps, and converging the brightness adjustment model to be trained based on the comprehensive loss to obtain the brightness adjustment model.

[0134] In some embodiments, the annotation brightness level and the annotation target brightness value are determined by the following steps: obtaining multiple image brightness scores corresponding to each of multiple training image samples, each of the multiple training image samples corresponds to a different light intensities, and the multiple training image samples and the original image sample are images taken of the same object; for each training image sample in the multiple training image samples, determining a comprehensive brightness score corresponding to the training image sample based on the average value of the multiple image brightness scores corresponding to the training image sample; selecting the light intensity value of the training image sample with the highest comprehensive brightness score among the multiple training image samples, determining it as the annotation target light intensity value corresponding to the original image sample, and determining the highest comprehensive brightness score as the annotation brightness level corresponding to the original image sample.

[0135] The image brightness adjustment device provided in this embodiment belongs to the same application concept as the image brightness adjustment method provided in the above embodiments of this application, and can execute the image brightness adjustment method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the image brightness adjustment method. For technical details not fully described in this embodiment, please refer to the specific processing content of the image brightness adjustment method provided in the above embodiments of this application, and will not be repeated here.

[0136] The functions implemented by the above acquisition unit 1101, brightness adjustment determination unit 1102 and brightness adjustment unit 1103 may be implemented by the same or different processors respectively, which is not limited in the embodiment of the present application.

[0137] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0138] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0139] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0140] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0141] Exemplary Electronic Devices

[0142] The present application embodiment provides an electronic device, see Fig.11 As shown, the electronic device includes:

[0143] Memory 200 and processor 210;

[0144] The memory 200 is connected to the processor 210 and is used to store programs;

[0145] The processor 210 is used to implement the image brightness adjustment method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0146] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0147] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0148] A bus may include a pathway that transfers information between components of a computer system.

[0149] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0150] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0151] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0152] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0153] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0154] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0155] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any image brightness adjustment method provided in the above embodiments of the present application.

[0156] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the image brightness adjustment method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the above-mentioned image brightness adjustment method.

[0157] Exemplary computer program products and storage media

[0158] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image brightness adjustment method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0159] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0160] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the image brightness adjustment method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0161] Step S101: Obtain facial video data of a target user.

[0162] Step S102: input the facial video data into an image brightness adjustment model to obtain a target cognitive state corresponding to the target user, wherein the image brightness adjustment model is obtained by optimizing and training a pre-trained baseline image brightness adjustment model based on manually annotated video samples, wherein the accuracy of the baseline image brightness adjustment model's prediction result of the cognitive state of the video sample is lower than a set accuracy threshold.

[0163] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0164] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0166] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0168] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0170] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0171] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0173] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for adjusting image brightness, characterized in that: include: Acquire an image brightness feature of the original image and a current light intensity value of the fill light device, wherein the image brightness feature includes features obtained by performing brightness evaluation on the original image based on various dimensions, wherein the various dimensions include an image quality dimension, a semantic dimension, a spatial dimension, a global dimension, and an image block dimension; Based on the image brightness feature and the current light intensity value of the fill light device, determining a target light intensity value for the fill light device to adjust the light intensity of a next frame of the original image; Based on the target light intensity value, the light intensity value of the fill light device when shooting the next frame of image is adjusted.

2. The method according to claim 1, characterized in that The step of acquiring the image brightness feature of the original image and determining, based on the image brightness feature and the current light intensity value of the fill light device, a target light intensity value for adjusting the light intensity of the next frame of the original image by the fill light device comprises: The original image is input into a brightness adjustment model, and the brightness adjustment model obtains the image brightness features of the original image; based on the image brightness features and the current light intensity value of the fill light device, a target light intensity value for the fill light device to adjust the light intensity of the next frame of the original image is determined.

3. The method according to claim 2, characterized in that The image brightness feature includes an image quality feature, the image quality feature corresponds to each image quality dimension, the brightness adjustment model includes an image quality feature extraction module, and the image quality feature extraction module includes each image quality evaluation unit and a KAN network; The step of obtaining the image brightness feature of the original image by the brightness adjustment model includes: Acquire, by means of the respective image quality assessment units, image quality assessment results of the original image in the respective image quality assessment dimensions; The weights corresponding to the image quality assessment results of the original image under each image quality assessment dimension are determined through the KAN network, and the image quality assessment results under each image quality assessment dimension are fused based on the weights to obtain the image quality features of the original image.

4. The method according to claim 2, characterized in that: The image brightness feature includes a semantic space feature, the semantic space feature corresponds to the semantic dimension and the spatial dimension, the brightness adjustment model includes a semantic space feature extraction module, and the semantic space feature extraction module includes a semantic feature extraction unit, a spatial feature extraction unit and a first fusion unit; The step of obtaining the image brightness feature of the original image by the brightness adjustment model includes: Extracting semantic features corresponding to the original image from the semantic dimension by the semantic feature extraction unit, and extracting spatial features corresponding to the original image from the spatial dimension by the spatial feature extraction unit; The semantic feature and the spatial feature are fused by the first fusion unit to obtain the semantic spatial feature corresponding to the original image.

5. The method according to claim 4, characterized in that The first fusion unit includes a first fusion unit, a multiplication unit, and a second fusion unit; The step of fusing the semantic features and the spatial features through the first fusion module to obtain the semantic spatial features corresponding to the original image includes: fusing the semantic feature and the spatial feature through the first fusion unit to obtain an initial semantic spatial feature; The initial semantic space feature and the space feature are multiplied by the multiplication unit to obtain a shallow semantic space feature; The initial semantic space feature and the shallow semantic space feature are fused through the second fusion unit to obtain the semantic space feature.

6. The method according to claim 2, characterized in that The image brightness feature includes an image sequence feature, the image sequence feature corresponds to the image block dimension, the brightness adjustment model includes an image sequence feature extraction module, and the image sequence feature extraction module includes an image block division unit and an image block visual feature extraction unit; The step of obtaining the image brightness feature of the original image by the brightness adjustment model includes: Dividing the semantic space features into individual image blocks by the image block dividing unit; The image block visual feature extraction unit extracts visual features from the image block dimension based on the semantic space features corresponding to each of the image blocks to obtain the image sequence features.

7. The method according to claim 2, characterized in that The image brightness feature includes a global feature, the brightness adjustment model includes a global feature extraction module, and the global feature extraction module includes a spatial pyramid pooling convolution network, a hole convolution network, an attention mechanism network and a third fusion unit; The step of obtaining the image brightness feature of the original image by the brightness adjustment model includes: Extracting output features of a preset size from the original image from the global dimension through the spatial pyramid pooling convolutional network; Performing a convolution operation on the output feature from the global dimension through the dilated convolutional network to obtain a dilated convolutional feature; Determine the weights corresponding to the output features of the preset size and the dilated convolution features respectively through the attention mechanism network, and fuse the output features of the preset size and the dilated convolution features respectively based on the weights corresponding to the output features of the preset size and the dilated convolution features respectively to obtain an attention feature; The attention feature and the hole convolution feature are fused through the third fusion unit to obtain the global feature.

8. The method according to any one of claims 2 to 7, characterized in that: The brightness adjustment model is trained by the following steps: Obtaining image brightness feature samples corresponding to the original image samples, current light intensity value samples of the fill light device and their corresponding training labels, wherein the training labels include a labeled brightness level, a labeled target brightness value and a labeled semantic segmentation map; Predicting a semantic segmentation map of the original image sample, a current brightness level, and a target light intensity value to be adjusted based on the image brightness feature samples corresponding to the original image sample and the current light intensity value samples of the fill light device by a brightness adjustment model to be trained, to obtain a predicted brightness level, a predicted target light intensity value, and a predicted semantic segmentation map; Based on the difference between the predicted brightness level and the annotated brightness level, the difference between the predicted target light intensity value and the annotated target light intensity value, and the difference between the predicted semantic segmentation map and the annotated semantic segmentation map, a comprehensive loss is determined, and based on the comprehensive loss, the brightness adjustment model to be trained is converged to obtain the brightness adjustment model.

9. The method according to claim 8, characterized in that The marked brightness level and the marked target brightness value are determined by the following steps: Obtaining a plurality of image brightness scores corresponding to a plurality of training image samples, each of the plurality of training image samples corresponding to a different light intensity, and the plurality of training image samples and the original image sample being images taken of the same object; For each training image sample in the plurality of training image samples, determining a comprehensive brightness score corresponding to the training image sample based on an average value of a plurality of image brightness scores corresponding to the training image sample; The illumination intensity value of the training image sample with the highest comprehensive brightness score among the multiple training image samples is selected and determined as the labeled target illumination intensity value corresponding to the original image sample, and the highest comprehensive brightness score is determined as the labeled brightness level corresponding to the original image sample.

10. An image brightness adjustment device, characterized in that: include: an acquisition unit, configured to acquire image brightness features of the original image and a current light intensity value of the fill light device, wherein the image brightness features include features obtained by performing brightness evaluation on the original image based on various dimensions, wherein the various dimensions include image quality dimension, semantic dimension, spatial dimension, global dimension, and image block dimension; a brightness adjustment determination unit, configured to determine, based on the image brightness feature and the current light intensity value of the fill light device, a target light intensity value for the fill light device to adjust the light intensity of the next frame of the original image; The brightness adjustment unit is used to adjust the illumination intensity value of the fill light device when shooting the next frame of image based on the target illumination intensity value.

11. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 9 by running the program in the memory.

12. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 9.

Images