Intelligent fill light control method and system

Abstract

The present invention discloses an intelligent fill light control method and system, and relates to the field of electronic technology. An intelligent fill light control system includes an image acquisition module, an image splicing and extraction module, a character portrait fitting module, a gray value calculation module, an emotion heart rate recognition module, and an intelligent fill light parameter adjustment module. The present invention can adjust the fill light strategy according to the user's characteristics and emotional state by performing user portrait fitting and emotion analysis on character information, thereby achieving personalized lighting effects and improving user experience; through the intelligent fill light parameter adjustment model for data fitting, parameters can be adjusted according to the first intelligent fill light strategy, the second intelligent fill light strategy, and the third intelligent fill light strategy to obtain an instant fill light strategy, improve the fill light effect and energy efficiency ratio, and achieve more intelligent intelligent fill light control.

Description

Technical Field

[0001] The present invention relates to the field of electronic technology, and in particular to an intelligent fill light control method and system. Background Art

[0002] The traditional fill light control method adopts manual adjustment of lighting parameters or adjustment of lighting gears based on manually set lighting parameters. However, in actual usage scenarios, users need to frequently adjust the lighting when shooting, which takes a lot of time. In addition, adjusting the lighting based on fixed gears is not personalized and customized, and cannot meet the lighting needs of all users.

[0003] During the shooting process, the character's emotions are an important factor in the beauty of the photo. When the character expresses positive emotions, it will add to the beauty of the photo and bring positive feelings to the photo. Conversely, when the character expresses negative emotions, it will reduce the beauty of the photo and bring negative feelings to the photo. Lighting can help adjust the character's emotions and bring positive effects to the photo.

[0004] Therefore, we urgently need to provide an intelligent fill light control method and system that can be adjusted according to character portraits, scene brightness and character emotions. Summary of the invention

[0005] The present invention aims to provide an intelligent fill light control method and system that are adjusted according to character portraits, scene brightness and character emotions, so as to improve the intelligence and customization of fill light control.

[0006] An intelligent fill light control method, comprising:

[0007] S1. Get the current front camera image and the current timestamp, get the front camera image and the intelligent fill light timestamp; get the three-dimensional scene image in the current scene;

[0008] S2. Segment the front camera image to obtain a character information image and a character background image; fuse the three-dimensional scene image and the character background image to obtain a scene information image; perform image recognition on the character information image to obtain a face area image;

[0009] S3. Fitting the character information image to the user portrait to obtain the character portrait classification information to be tested; matching the character portrait classification information to be tested with the first intelligent fill light strategy library to obtain the matching character portrait classification information, and outputting the first intelligent fill light strategy corresponding to the matching character portrait classification information; the first intelligent fill light strategy library includes standard character portrait classification information and its corresponding first intelligent fill light strategy;

[0010] S4. The scene information image is grayed using a weighted average method to obtain a scene gray value; the scene gray value is compared with the second intelligent fill light strategy library to obtain a second intelligent fill light strategy; the second intelligent fill light strategy library includes a gray threshold range and its corresponding second intelligent fill light strategy;

[0011] S5. Perform non-contact heart rate analysis on the face area image to obtain the non-contact heart rate value of the face; perform emotion analysis on the non-contact heart rate value of the face and the face area image to obtain the face emotion label; match the face emotion label in the third intelligent fill light strategy library, and output the third intelligent fill light strategy corresponding to the face emotion label; the third intelligent fill light strategy library includes standard face emotion labels and their corresponding third intelligent fill light strategies;

[0012] S6. Input the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy into the intelligent fill light parameter adjustment model for data fitting to obtain the real-time fill light strategy, and perform intelligent fill light control based on the real-time fill light strategy.

[0013] As a preferred technical solution of the present invention, the specific steps of step S2 include:

[0014] S21. Preprocess the front camera image to obtain a preprocessed front camera image;

[0015] S22. Use edge detection algorithm to identify the preprocessed front camera image to obtain a strong edge front camera image, and segment the strong edge front camera image into a character information image and a character background image according to the edge;

[0016] S23. Align the character background image and the three-dimensional scene image in terms of camera viewing angle and perspective, and use image fusion technology to stitch the character background image and the three-dimensional scene image to obtain a scene information image;

[0017] S24. Detect the character information image using a face detection algorithm to obtain the face image position in the character information image; cut the character information image based on the face image position to obtain a face area image.

[0018] As a preferred technical solution of the present invention, the specific steps of step S3 include:

[0019] S31. Extracting the character portrait features from the character information image using the feature engineering method to obtain a preliminary character feature vector;

[0020] S32. Fitting the user portrait according to the preliminary character feature vector to obtain the classification information of the character portrait to be tested;

[0021] S33. Calculate the similarity between the portrait classification information of the person to be tested and the standard portrait classification information in the first intelligent fill-light strategy library one by one, output the standard portrait classification information with the highest similarity to the portrait classification information of the person to be tested, and obtain the matching standard portrait classification information; obtain the corresponding first intelligent fill-light strategy in the first intelligent fill-light strategy library based on the matching standard portrait classification information, and obtain the first intelligent fill-light strategy corresponding to the portrait classification information of the person to be tested.

[0022] As a preferred technical solution of the present invention, the specific steps of step S4 include:

[0023] S41. Convert the scene information image into a grayscale image to obtain a scene grayscale image. The scene grayscale image has a total of n pixels, p i Represents the scene grayscale image pixel p i , i=1,2,…,n;

[0024] S42. For the scene grayscale image pixel p i Calculate the value of the RGB color channel and get the red channel pixel value R i , green channel pixel value G i and blue channel pixel value B i ; Using the formula Grey (p i )=0.2989*R i +0.5870*G i +0.1140*B i Calculate the scene grayscale image pixel p i Gray value, where Grey (p i ) represents the scene grayscale image pixel p i Gray value of

[0025] S43. Using formula Calculate the grayscale value of the scene grayscale image; where GREY represents the scene grayscale value of the scene grayscale image, and w i Represents the scene grayscale image pixel p i The weight value of

[0026] S44. Compare the scene grayscale value with the grayscale threshold in the second intelligent fill light strategy library, obtain the grayscale threshold range corresponding to the scene grayscale value in the second intelligent fill light strategy library, and obtain the second intelligent fill light strategy based on the grayscale threshold range;

[0027] As a preferred technical solution of the present invention, the specific steps of step S5 include:

[0028] S51. Inputting the face region image into the face emotion recognition model for emotion recognition to obtain an initial face emotion label;

[0029] S52. The face region image is selected by using the ROI region selection method to obtain m face ROI regions;

[0030] S53. Extract signals from m face ROI regions to obtain m groups of face IPPG signals;

[0031] S54. Remove noise from the m groups of face IPPG signals using a denoising algorithm to obtain m groups of denoised face IPPG signals; screen the m groups of denoised face IPPG signals using a Pearson correlation coefficient fusion algorithm to obtain the best face pulse wave signal;

[0032] S55. The best facial pulse wave signal is transformed by fast FFT algorithm to obtain the best facial pulse wave signal spectrum and the number of Fourier transform points q, and the maximum peak point on the spectrum of the best facial pulse wave signal is obtained by peak method to obtain the maximum peak point max; using formula f h =((1 / f) / q)*maxCalculate the peak frequency f corresponding to the maximum peak point max h , and get the peak frequency f h ; f is the time interval between the last image acquisition and the current image acquisition; using the formula h=f h *60 Calculate the non-contact heart rate value of the face;

[0033] S56. Inputting the initial facial emotion label and the face contactless heart rate value into the facial emotion heart rate auxiliary recognition model to obtain the facial emotion label;

[0034] S57. Calculate the similarity between the facial emotion label and the standard facial emotion labels in the third intelligent fill-light strategy library one by one, output the standard facial emotion label with the highest similarity to the facial emotion label, and obtain the matching facial emotion label; based on the matching facial emotion label, obtain the corresponding third intelligent fill-light strategy in the third intelligent fill-light strategy library to obtain the third intelligent fill-light strategy corresponding to the facial emotion label.

[0035] As a preferred technical solution of the present invention, the facial emotion recognition model includes an image preprocessing layer, an expression feature extraction layer and a facial emotion label output layer;

[0036] The image preprocessing layer is used to denoise the face area image and convert it into an input vector to obtain the face area image input vector;

[0037] The expression feature extraction layer is used to extract features from the face area image input vector to obtain the face area image feature vector;

[0038] The facial emotion label output layer is used to perform emotion recognition on the facial region image feature vector and output the initial facial emotion label.

[0039] As a preferred technical solution of the present invention, a facial emotion recognition model is constructed, and the specific steps are as follows:

[0040] Q1. Prepare several face area images, annotate the face area images with facial emotion labels, obtain a pre-processed face area image set training set, and set the maximum number of training times;

[0041] Q2. Use the Transformer model as the initial model of the facial emotion recognition model to obtain the facial emotion recognition initial model; send the preprocessed face area image set training set into the facial emotion recognition initial model for training, take the facial emotion label as the target, and judge whether the maximum number of training times has been reached. If so, output the trained facial emotion recognition initial model, that is, the facial emotion recognition model; otherwise, continue to train the facial emotion recognition initial model through the preprocessed face area image set training set.

[0042] As a preferred technical solution of the present invention, a facial emotion heart rate auxiliary recognition model is constructed, and the specific steps are as follows:

[0043] Q3. Prepare several face area images with heart rate values, input several face area images with heart rate values ​​into the facial emotion recognition model for emotion recognition, and obtain several initial emotion recognition labels; perform feature fitting on the heart rate values ​​of the face area images with heart rate values ​​and their corresponding initial emotion recognition labels to obtain the characteristic values ​​of the heart rate emotion recognition labels; perform labeling processing on several face area images with characteristic values ​​of the heart rate emotion recognition labels using the facial emotion labels to obtain the preprocessed face heart rate area image set training set, and set the maximum number of training times;

[0044] Q4. Input the preprocessed face-heart rate region image set training set into the facial emotion recognition model for training to obtain the initial model for facial emotion and heart rate assisted recognition; take the facial emotion label as the target, determine whether the maximum number of training times has been reached, if so, output the trained initial model for facial emotion and heart rate assisted recognition, and the initial model for facial emotion and heart rate assisted recognition is the facial emotion and heart rate assisted recognition model; otherwise, continue to train the initial model for facial emotion and heart rate assisted recognition by using the preprocessed face-heart rate region image set training set.

[0045] As a preferred technical solution of the present invention, the intelligent fill light parameter adjustment model includes an intelligent fill light strategy definition layer, a parameter extraction layer, a parameter selection layer, a data output layer and a lighting control layer;

[0046] The intelligent fill light strategy definition layer is used to define the fill light parameter mapping rule as light=f(α(t), β(b), γ(c)), where light represents the intelligent fill light strategy, α(t) represents the color temperature adjustment parameter, β(b) represents the brightness adjustment parameter, and γ(c) represents the color adjustment parameter;

[0047] The parameter extraction layer is used to extract the parameters of the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy to obtain a color temperature adjustment parameter set, a brightness adjustment parameter set and a color adjustment parameter set;

[0048] The parameter selection layer is used to average the parameters of the color temperature adjustment parameter set to obtain the average color temperature adjustment parameter; average the parameters of the color adjustment parameter set to obtain the average color adjustment parameter; extract the maximum parameter of the brightness adjustment parameter set to obtain the maximum brightness adjustment parameter;

[0049] The data output layer is used to construct an intelligent fill light strategy based on the average color temperature adjustment parameter, the maximum brightness adjustment parameter and the average color adjustment parameter to obtain an instant fill light strategy;

[0050] The lighting control layer is used to perform intelligent fill light control based on real-time fill light strategy.

[0051] An intelligent fill light control system, comprising:

[0052] The image acquisition module is used to obtain the front camera image at the current moment and the timestamp at the current moment, obtain the front camera image and the intelligent fill light timestamp; obtain the three-dimensional scene image in the current scene;

[0053] The image stitching and extraction module is used to perform image segmentation on the front camera image to obtain a person information image and a person background image; perform image fusion on the three-dimensional scene image and the person background image to obtain a scene information image; perform image recognition on the person information image to obtain a face area image;

[0054] A character portrait fitting module is used to fit the character information image to a user portrait to obtain the character portrait classification information to be tested; the character portrait classification information to be tested is matched with the first intelligent fill-light strategy library to obtain the matched character portrait classification information, and the first intelligent fill-light strategy corresponding to the matched character portrait classification information is output; the first intelligent fill-light strategy library includes standard character portrait classification information and its corresponding first intelligent fill-light strategy;

[0055] A gray value calculation module is used to grayscale the scene information image using a weighted average method to obtain a scene gray value; compare the scene gray value with a second intelligent fill light strategy library to obtain a second intelligent fill light strategy; the second intelligent fill light strategy library includes a gray threshold range and its corresponding second intelligent fill light strategy;

[0056] The emotional heart rate recognition module is used to perform non-contact heart rate analysis on the face area image to obtain the non-contact heart rate value of the face; perform emotional analysis on the non-contact heart rate value of the face and the face area image to obtain the facial emotion label; match the facial emotion label in the third intelligent fill light strategy library and output the third intelligent fill light strategy corresponding to the facial emotion label; the third intelligent fill light strategy library includes facial emotion labels and their corresponding third intelligent fill light strategies;

[0057] The intelligent fill light parameter adjustment module is used to deploy the intelligent fill light parameter adjustment model, input the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy into the intelligent fill light parameter adjustment model for data fitting, obtain the real-time fill light strategy, and perform intelligent fill light control based on the real-time fill light strategy.

[0058] The present invention has the following advantages:

[0059] 1. The present invention can adjust the fill light strategy according to the user characteristics and emotional state by performing user portrait fitting and emotion analysis on character information, thereby achieving personalized lighting effects and improving user experience; by utilizing three-dimensional scene images and grayscale processing, the brightness and color distribution of the current environment can be perceived, and the fill light strategy can be dynamically adjusted according to the scene characteristics to adapt to the lighting needs in different scenes and ensure picture clarity and color restoration; through non-contact heart rate analysis and emotion analysis, the system can identify the emotional state of the face, match the corresponding fill light strategy according to the emotion label, and help create a more comfortable lighting atmosphere that is more suitable for the current emotion; through the intelligent fill light parameter adjustment model for data fitting, parameters can be adjusted according to the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy to obtain an instant fill light strategy, improve the fill light effect and energy efficiency ratio, and achieve more intelligent lighting control; using the instant fill light strategy, real-time fill light adjustment can be achieved, and the changes in the environment and users can be quickly responded to maintain the stability and consistency of the lighting effect.

[0060] 2. The present invention performs feature extraction and user portrait fitting on character information images. The system can provide personalized fill-light solutions based on each person's appearance characteristics and identity information, so that users can obtain the best lighting effects in photos or videos; through similarity calculation and matching of standard character portrait classification information, the user portrait of the person to be tested can be accurately identified, and the most suitable fill-light strategy can be selected accordingly, thereby ensuring that the lighting effect matches the scene requirements.

[0061] 3. The present invention can understand the user's emotional state by performing emotion recognition on facial information, thereby adjusting the shooting environment or emotional atmosphere; combining emotional tags and non-contact heart rate values, it can provide each shooting subject with a personalized fill light strategy to meet their emotional state and physiological needs, thereby improving shooting effects and user satisfaction.

[0062] 4. The present invention selects the Transformer model to recognize face area portraits, so as to capture the features in face area portraits, match the emotions of the characters according to the features of face area portraits, and add the heart rate value to help assist in recognition, thereby outputting the emotion labels of the characters, which helps to improve the scientificity and accuracy of the emotion labels of the characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a schematic diagram of the structure of an intelligent fill light control system adopted in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable persons skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0065] Example 1

[0066] An intelligent fill light control method, comprising:

[0067] S1. Get the current front camera image and the current timestamp, get the front camera image and the intelligent fill light timestamp; get the three-dimensional scene image in the current scene; the front camera is an infrared high-definition camera, and the three-dimensional scene image is taken by a light field camera;

[0068] S2. Segment the front camera image to obtain a character information image and a character background image; fuse the three-dimensional scene image and the character background image to obtain a scene information image; perform image recognition on the character information image to obtain a face area image;

[0069] The specific steps of step S2 include:

[0070] S21. Preprocessing the front camera image to obtain a preprocessed front camera image;

[0071] S22. Use edge detection algorithm to identify the preprocessed front camera image to obtain a strong edge front camera image, and segment the strong edge front camera image into a character information image and a character background image according to the edge;

[0072] S23. Align the character background image and the three-dimensional scene image in terms of camera viewing angle and perspective, and use image fusion technology to stitch the character background image and the three-dimensional scene image to obtain a scene information image;

[0073] S24. Detecting the character information image using a face detection algorithm to obtain the position of the face image in the character information image; cutting the character information image based on the face image position to obtain a face area image;

[0074] Through edge detection and segmentation, the person can be effectively separated from the background, providing clear person information images and background images for subsequent image processing tasks; by fusing the person background image with the three-dimensional scene image, the realism and immersion of the image can be enhanced, making the person seem to be in a real environment. Through the face detection algorithm, the face position in the person image can be accurately located, and the face area image can be extracted, providing accurate input data for subsequent face recognition, emotion recognition and other tasks; preprocessing the front camera image helps to remove noise and smooth the image, thereby improving the accuracy and robustness of subsequent algorithms;

[0075] S3. Fitting the character information image to the user portrait to obtain the character portrait classification information to be tested; matching the character portrait classification information to be tested with the first intelligent fill light strategy library to obtain the matching character portrait classification information, and outputting the first intelligent fill light strategy corresponding to the matching character portrait classification information; the first intelligent fill light strategy library includes standard character portrait classification information and its corresponding first intelligent fill light strategy;

[0076] The specific steps of step S3 include:

[0077] S31. Extracting the character portrait features from the character information image using feature engineering method to obtain a preliminary character feature vector;

[0078] S32. Fitting the user portrait according to the preliminary character feature vector to obtain the classification information of the character portrait to be tested;

[0079] S33. Calculate the similarity between the portrait classification information of the person to be tested and the standard portrait classification information in the first intelligent fill-light strategy library one by one, output the standard portrait classification information with the highest similarity to the portrait classification information of the person to be tested, and obtain the matching standard portrait classification information; obtain the corresponding first intelligent fill-light strategy in the first intelligent fill-light strategy library based on the matching standard portrait classification information, and obtain the first intelligent fill-light strategy corresponding to the portrait classification information of the person to be tested;

[0080] By using the feature engineering method to extract features from the character information image, the image information can be converted into a numerical feature vector, and the key features of the character, such as facial expressions and postures, can be extracted, providing a basis for subsequent classification and recognition; user portrait fitting can be performed based on the preliminary character feature vector, and the character portrait to be tested can be classified into the corresponding user portrait, thereby realizing personalized processing of different types of characters and improving processing efficiency and accuracy; by calculating the similarity with the standard character portrait classification information in the first intelligent fill light strategy library, the standard classification information that matches the character portrait classification information to be tested can be found, and the corresponding intelligent fill light strategy can be obtained accordingly. This helps to select a suitable fill light strategy based on the characteristics of the character portrait, and improves the quality and aesthetics of the image.

[0081] User portrait fitting is done by establishing a user portrait library, which contains feature vectors of various types of users and their corresponding portrait classification information; the preliminary character feature vector is calculated similarly with the feature vector in the user portrait library, and according to the similarity calculation result, the preliminary character feature vector is matched to the user portrait classification information that is most similar to it, and according to the matched user portrait classification information, the user to be tested is classified into the corresponding user portrait to obtain the user portrait classification information that best matches its characteristics;

[0082] Constructing the first intelligent fill-light strategy library includes obtaining photos of various types of user portraits, analyzing the lighting parameters of the photos of various types of user portraits, and summarizing the lighting parameters with the highest evaluation values ​​corresponding to various types of user portraits by using feature fusion technology to obtain the first intelligent fill-light strategy; combining the standard character portrait classification information and the corresponding first intelligent fill-light strategy to obtain the first intelligent fill-light strategy library;

[0083] S4. The scene information image is grayed using a weighted average method to obtain a scene gray value; the scene gray value is compared with the second intelligent fill light strategy library to obtain a second intelligent fill light strategy; the second intelligent fill light strategy library includes a gray threshold range and its corresponding second intelligent fill light strategy;

[0084] The specific steps of step S4 include:

[0085] S41. Convert the scene information image into a grayscale image to obtain a scene grayscale image. The scene grayscale image has a total of n pixels, p i Represents the scene grayscale image pixel p i , i=1,2,…,n;

[0086] S42. For the scene grayscale image pixel p i Calculate the value of the RGB color channel and get the red channel pixel value R i , green channel pixel value Gi and blue channel pixel value B i ; Using the formula Grey (p i )=0.2989*R i +0.5870*G i +0.1140*B i Calculate the scene grayscale image pixel p i Gray value, where Grey (p i ) represents the scene grayscale image pixel p i Gray value of

[0087] S43. Using formula Calculate the grayscale value of the scene grayscale image; where GREY represents the scene grayscale value of the scene grayscale image, and w i Represents the scene grayscale image pixel p i The weight value of i According to Grey (p i ) to adjust, Grey (p i ) is larger, w i The higher the value, the larger the normalized value corresponding to Grey (pi). The higher the value, the larger the normalized value corresponding to Grey (pi). The normalized value is set to w i ;

[0088] S44. Compare the scene grayscale value with the grayscale threshold in the second intelligent fill light strategy library, obtain the grayscale threshold range corresponding to the scene grayscale value in the second intelligent fill light strategy library, and obtain the second intelligent fill light strategy based on the grayscale threshold range;

[0089] Converting scene information images into grayscale images helps to simplify image information, remove color information, and make subsequent processing more efficient. Grayscale images are usually easier to process and analyze, while retaining the main structure and features of the image; analyzing the color channels of grayscale images can extract richer image information from the pixel values ​​of different channels. By calculating the values ​​of the red, green, and blue channels of each pixel and converting them into grayscale values ​​according to the formula, it helps to retain the details and texture information in the image; based on the grayscale value and weight value of the pixel, the importance of different pixels in the image can be distinguished. By assigning different weight values ​​to pixels, their influence on the entire image can be adjusted, thereby focusing on or ignoring different areas of the image, which helps to improve the accuracy and efficiency of image processing;

[0090] Building the second intelligent fill-light strategy library includes obtaining photos with various grayscale value ranges, analyzing the lighting parameters of the photos with various grayscale value ranges, and using feature fusion technology to summarize the lighting parameters that are used most frequently for the photos with various grayscale value ranges to obtain the second intelligent fill-light strategy; combining the grayscale threshold ranges and the corresponding second intelligent fill-light strategy to obtain the second intelligent fill-light strategy library;

[0091] S5. Perform non-contact heart rate analysis on the face area image to obtain the non-contact heart rate value of the face; perform emotion analysis on the non-contact heart rate value of the face and the face area image to obtain the face emotion label; match the face emotion label in the third intelligent fill light strategy library, and output the third intelligent fill light strategy corresponding to the face emotion label; the third intelligent fill light strategy library includes standard face emotion labels and their corresponding third intelligent fill light strategies;

[0092] The specific steps of step S5 include:

[0093] S51. Inputting the face region image into the face emotion recognition model for emotion recognition to obtain an initial face emotion label;

[0094] S52. The face region image is selected by using the ROI region selection method to obtain m face ROI regions;

[0095] S53. Extract signals from m face ROI regions to obtain m groups of face IPPG signals;

[0096] S54. Remove noise from the m groups of face IPPG signals using a denoising algorithm to obtain m groups of denoised face IPPG signals; screen the m groups of denoised face IPPG signals using a Pearson correlation coefficient fusion algorithm to obtain the best face pulse wave signal;

[0097] S55. The best facial pulse wave signal is transformed by fast FFT algorithm to obtain the best facial pulse wave signal spectrum and the number of Fourier transform points q, and the maximum peak point on the spectrum of the best facial pulse wave signal is obtained by peak method to obtain the maximum peak point max; using formula f h =((1 / f) / q)*maxCalculate the peak frequency f corresponding to the maximum peak point max h , and get the peak frequency f h ; f is the time interval between the last image acquisition and the current image acquisition; using the formula h=f h *60 Calculate the non-contact heart rate value of the face;

[0098] S56. Inputting the initial facial emotion label and the face contactless heart rate value into the facial emotion heart rate auxiliary recognition model to obtain the facial emotion label;

[0099] S57. Calculate the similarity of the facial emotion label and the standard facial emotion label in the third intelligent fill light strategy library one by one, output the standard facial emotion label with the highest similarity to the facial emotion label, and obtain the matching facial emotion label; obtain the corresponding third intelligent fill light strategy in the third intelligent fill light strategy library based on the matching facial emotion label, and obtain the third intelligent fill light strategy corresponding to the facial emotion label;

[0100] By inputting the face region image into the emotion recognition model for emotion recognition, the emotional information of the face can be extracted from the image; this helps to understand the user's emotional state in a specific situation and provide reference and support for subsequent application scenarios; using the ROI region selection method to obtain multiple face ROI regions helps to accurately locate the face position, thereby improving the accuracy and reliability of subsequent signal extraction and analysis; performing signal extraction on the ROI region to obtain the face IPPG signal, physiological characteristics such as heart rate can be extracted from the biological signal, providing data support for subsequent emotion analysis; using the fast FFT algorithm for spectrum transformation, the time domain signal can be converted into a frequency domain signal, and then the spectrum characteristics of the face pulse wave signal can be analyzed to extract the heart rate value from it; the initial face emotion label and the face non-contact heart rate value are input into the emotion heart rate auxiliary recognition model. Through the auxiliary recognition of the model, the emotional state of the face can be more comprehensively understood, and the accuracy and reliability of emotion recognition can be improved;

[0101] Building the third intelligent fill-light strategy library includes obtaining photos with various emotion labels, analyzing the light parameters of the photos with various emotion labels, and summarizing the light parameters with the highest evaluation values ​​corresponding to various emotion labels by using feature fusion technology to obtain the third intelligent fill-light strategy; combining the standard character emotion labels and their corresponding third intelligent fill-light strategies to obtain the third intelligent fill-light strategy library;

[0102] The facial emotion recognition model includes an image preprocessing layer, an expression feature extraction layer, and a facial emotion label output layer;

[0103] The image preprocessing layer is used to denoise the face area image and convert it into an input vector to obtain the face area image input vector;

[0104] The expression feature extraction layer is used to extract features from the face area image input vector to obtain the face area image feature vector;

[0105] The facial emotion label output layer is used to perform emotion recognition on the facial region image feature vector and output the initial facial emotion label;

[0106] Build a facial emotion recognition model, specific steps:

[0107] Q1. Prepare several face area images, annotate the face area images with facial emotion labels, obtain a pre-processed face area image set training set, and set the maximum number of training times;

[0108] Q2. Use the Transformer model as the initial model of the facial emotion recognition model to obtain the facial emotion recognition initial model; send the preprocessed facial region image set training set to the facial emotion recognition initial model for training, take the facial emotion label as the target, and judge whether the maximum number of training times has been reached. If so, output the trained facial emotion recognition initial model, which is the facial emotion recognition model; otherwise, continue to train the facial emotion recognition initial model through the preprocessed facial region image set training set;

[0109] Construct a facial emotion and heart rate auxiliary recognition model. Specific steps:

[0110] Q3. Prepare several face area images with heart rate values, input several face area images with heart rate values ​​into the facial emotion recognition model for emotion recognition, and obtain several initial emotion recognition labels; perform feature fitting on the heart rate values ​​of the face area images with heart rate values ​​and their corresponding initial emotion recognition labels to obtain the characteristic values ​​of the heart rate emotion recognition labels; perform labeling processing on several face area images with characteristic values ​​of the heart rate emotion recognition labels using the facial emotion labels to obtain the preprocessed face heart rate area image set training set, and set the maximum number of training times;

[0111] Q4. Input the preprocessed face heart rate region image set training set into the face emotion recognition model for training, and obtain the face emotion heart rate auxiliary recognition initial model; take the face emotion label as the target, and judge whether the maximum number of training times has been reached. If so, output the trained face emotion heart rate auxiliary recognition initial model, and the face emotion heart rate auxiliary recognition initial model is the face emotion heart rate auxiliary recognition model; otherwise, continue to train the face emotion heart rate auxiliary recognition initial model by using the preprocessed face heart rate region image set training set;

[0112] A facial emotion heart rate auxiliary recognition model was constructed. By preparing a facial region image with heart rate values ​​and inputting it into the facial emotion recognition model for emotion recognition, an initial emotion recognition label was obtained. Then, the heart rate value and the emotion recognition label were feature fitted to construct a training set with heart rate emotion recognition label feature values. These data were input into the emotion recognition model for training to obtain a facial emotion heart rate auxiliary recognition model. This step can enhance the model's understanding of emotions, integrate heart rate information with emotion information, and improve the model's accuracy and comprehensiveness in understanding facial emotions.

[0113] S6. Input the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy into the intelligent fill light parameter adjustment model for data fitting, obtain the real-time fill light strategy, and perform intelligent fill light control based on the real-time fill light strategy;

[0114] The intelligent fill light parameter adjustment model includes the intelligent fill light strategy definition layer, parameter extraction layer, parameter selection layer, data output layer and lighting control layer;

[0115] The intelligent fill light strategy definition layer is used to define the fill light parameter mapping rule as light=f(α(t), β(b), γ(c)), where light represents the intelligent fill light strategy, α(t) represents the color temperature adjustment parameter, β(b) represents the brightness adjustment parameter, and γ(c) represents the color adjustment parameter;

[0116] The parameter extraction layer is used to extract the parameters of the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy to obtain a color temperature adjustment parameter set, a brightness adjustment parameter set and a color adjustment parameter set;

[0117] The parameter selection layer is used to average the parameters of the color temperature adjustment parameter set to obtain the average color temperature adjustment parameter; average the parameters of the color adjustment parameter set to obtain the average color adjustment parameter; extract the maximum parameter of the brightness adjustment parameter set to obtain the maximum brightness adjustment parameter;

[0118] The data output layer is used to construct an intelligent fill light strategy based on the average color temperature adjustment parameter, the maximum brightness adjustment parameter and the average color adjustment parameter to obtain an instant fill light strategy;

[0119] The lighting control layer is used to perform intelligent fill light control based on real-time fill light strategy.

[0120] Example 2

[0121] An intelligent fill light control system, such as Figure 1 As shown, including:

[0122] The image acquisition module is used to obtain the front camera image at the current moment and the timestamp at the current moment, obtain the front camera image and the intelligent fill light timestamp; obtain the three-dimensional scene image in the current scene;

[0123] The image stitching and extraction module is used to perform image segmentation on the front camera image to obtain a person information image and a person background image; perform image fusion on the three-dimensional scene image and the person background image to obtain a scene information image; perform image recognition on the person information image to obtain a face area image;

[0124] A character portrait fitting module is used to fit the character information image to a user portrait to obtain the character portrait classification information to be tested; the character portrait classification information to be tested is matched with the first intelligent fill-light strategy library to obtain the matched character portrait classification information, and the first intelligent fill-light strategy corresponding to the matched character portrait classification information is output; the first intelligent fill-light strategy library includes standard character portrait classification information and its corresponding first intelligent fill-light strategy;

[0125] A gray value calculation module is used to grayscale the scene information image using a weighted average method to obtain a scene gray value; compare the scene gray value with a second intelligent fill light strategy library to obtain a second intelligent fill light strategy; the second intelligent fill light strategy library includes a gray threshold range and its corresponding second intelligent fill light strategy;

[0126] The emotional heart rate recognition module is used to perform non-contact heart rate analysis on the face area image to obtain the non-contact heart rate value of the face; perform emotional analysis on the non-contact heart rate value of the face and the face area image to obtain the facial emotion label; match the facial emotion label in the third intelligent fill light strategy library and output the third intelligent fill light strategy corresponding to the facial emotion label; the third intelligent fill light strategy library includes facial emotion labels and their corresponding third intelligent fill light strategies;

[0127] The intelligent fill light parameter adjustment module is used to deploy the intelligent fill light parameter adjustment model, input the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy into the intelligent fill light parameter adjustment model for data fitting, obtain the real-time fill light strategy, and perform intelligent fill light control based on the real-time fill light strategy.

[0128] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention. Parts not described in detail in this specification belong to the prior art known to those skilled in the art.

Claims

1. An intelligent fill light control method, characterized in that: include: S1. Get the current front camera image and the current timestamp, and get the front camera image and the smart fill light timestamp; Get the three-dimensional scene image in the current scene; S2. Segment the front camera image to obtain a character information image and a character background image; The three-dimensional scene image and the character background image are subjected to image fusion to obtain a scene information image; the character information image is subjected to image recognition to obtain a face area image; S3. Fitting the character information image to the user portrait to obtain the character portrait classification information to be tested; matching the character portrait classification information to be tested with the first intelligent fill light strategy library to obtain the matching character portrait classification information, and outputting the first intelligent fill light strategy corresponding to the matching character portrait classification information; the first intelligent fill light strategy library includes standard character portrait classification information and its corresponding first intelligent fill light strategy; S4. graying the scene information image using a weighted average method to obtain a scene gray value; comparing the scene gray value with the second intelligent fill light strategy library to obtain a second intelligent fill light strategy; The second intelligent fill light strategy library includes a grayscale threshold range and its corresponding second intelligent fill light strategy; S5. Perform non-contact heart rate analysis on the face area image to obtain the non-contact heart rate value of the face; Perform emotion analysis on the non-contact heart rate value of the face and the face area image to obtain the face emotion label; match the face emotion label in the third intelligent fill light strategy library and output the third intelligent fill light strategy corresponding to the face emotion label; the third intelligent fill light strategy library includes standard face emotion labels and their corresponding third intelligent fill light strategies; S6. Input the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy into the intelligent fill light parameter adjustment model for data fitting, obtain the real-time fill light strategy, and perform intelligent fill light control based on the real-time fill light strategy; The specific steps of step S3 include: S31. Extracting the character portrait features from the character information image using feature engineering method to obtain a preliminary character feature vector; S32. Fitting the user portrait according to the preliminary character feature vector to obtain the classification information of the character portrait to be tested; S33. Calculate the similarity between the portrait classification information of the person to be tested and the standard portrait classification information in the first intelligent fill-light strategy library one by one, output the standard portrait classification information with the highest similarity to the portrait classification information of the person to be tested, and obtain the matching standard portrait classification information; obtain the corresponding first intelligent fill-light strategy in the first intelligent fill-light strategy library based on the matching standard portrait classification information, and obtain the first intelligent fill-light strategy corresponding to the portrait classification information of the person to be tested.

2. The intelligent fill light control method according to claim 1, characterized in that: The specific steps of step S2 include: S21. Preprocess the front camera image to obtain a preprocessed front camera image; S22. Use edge detection algorithm to identify the preprocessed front camera image to obtain a strong edge front camera image, and segment the strong edge front camera image into a character information image and a character background image according to the edge; S23. Align the character background image and the three-dimensional scene image in terms of camera viewing angle and perspective, and use image fusion technology to stitch the character background image and the three-dimensional scene image to obtain a scene information image; S24. Detect the character information image using a face detection algorithm to obtain the position of the face image in the character information image; The character information image is cut based on the position of the face image to obtain a face area image.

3. The intelligent fill light control method according to claim 2, characterized in that: The specific steps of step S4 include: S41. Convert the scene information image into a grayscale image to obtain a scene grayscale image. The scene grayscale image has a total of n pixels, p i Represents the scene grayscale image pixel p i , i=1,2,…,n; S42. For the scene grayscale image pixel p i Calculate the value of the RGB color channel and get the red channel pixel value R i , green channel pixel value G i and blue channel pixel value B i ; Using the formula Grey(p i )=0.2989*R i +0.5870*G i +0.1140*B i Calculate the scene grayscale image pixel p i Gray value, where Grey(p i ) represents the scene grayscale image pixel p i Gray value of S43. Using formula Calculate the grayscale value of the scene grayscale image; where GREY represents the scene grayscale value of the scene grayscale image, and w i Represents the scene grayscale image pixel p i The weight value of S44. Compare the scene grayscale value with the grayscale threshold in the second intelligent fill light strategy library to obtain the grayscale threshold range corresponding to the scene grayscale value in the second intelligent fill light strategy library, and obtain the second intelligent fill light strategy based on the grayscale threshold range.

4. The intelligent fill light control method according to claim 3, characterized in that: The specific steps of step S5 include: S51. Inputting the face region image into the face emotion recognition model for emotion recognition to obtain an initial face emotion label; S52. The face region image is selected by using the ROI region selection method to obtain m face ROI regions; S53. Extract signals from m face ROI regions to obtain m groups of face IPPG signals; S54. Remove noise from the m groups of face IPPG signals using a denoising algorithm to obtain m groups of denoised face IPPG signals; screen the m groups of denoised face IPPG signals using a Pearson correlation coefficient fusion algorithm to obtain the best face pulse wave signal; S55. The best facial pulse wave signal is transformed by fast FFT algorithm to obtain the best facial pulse wave signal spectrum and the number of Fourier transform points q, and the maximum peak point on the spectrum of the best facial pulse wave signal is obtained by peak method to obtain the maximum peak point max; using formula f h =((1 / f) / q)*max calculates the peak frequency f corresponding to the maximum peak point max h , and get the peak frequency f h ; f is the time interval between the last image acquisition and the current image acquisition; using the formula h = f h *60 Calculate the non-contact heart rate value of the face; S56. Inputting the initial facial emotion label and the face contactless heart rate value into the facial emotion heart rate auxiliary recognition model to obtain the facial emotion label; S57. Calculate the similarity between the facial emotion label and the standard facial emotion labels in the third intelligent fill-light strategy library one by one, output the standard facial emotion label with the highest similarity to the facial emotion label, and obtain the matching facial emotion label; based on the matching facial emotion label, obtain the corresponding third intelligent fill-light strategy in the third intelligent fill-light strategy library to obtain the third intelligent fill-light strategy corresponding to the facial emotion label.

5. The intelligent fill light control method according to claim 4, characterized in that: The facial emotion recognition model includes an image preprocessing layer, an expression feature extraction layer, and a facial emotion label output layer; The image preprocessing layer is used to denoise the face area image and convert it into an input vector to obtain the face area image input vector; The expression feature extraction layer is used to extract features from the face area image input vector to obtain the face area image feature vector; The facial emotion label output layer is used to perform emotion recognition on the facial region image feature vector and output the initial facial emotion label.

6. The intelligent fill light control method according to claim 5, characterized in that: Build a facial emotion recognition model, specific steps: Q1. Prepare several face area images, annotate the face area images with facial emotion labels, obtain a pre-processed face area image set training set, and set the maximum number of training times; Q2. Use the Transformer model as the initial model of the facial emotion recognition model to obtain the facial emotion recognition initial model; send the preprocessed face area image set training set into the facial emotion recognition initial model for training, take the facial emotion label as the target, and judge whether the maximum number of training times has been reached. If so, output the trained facial emotion recognition initial model, that is, the facial emotion recognition model; otherwise, continue to train the facial emotion recognition initial model through the preprocessed face area image set training set.

7. The intelligent fill light control method according to claim 6, characterized in that: Construct a facial emotion and heart rate auxiliary recognition model. Specific steps: Q3. Prepare several face area images with heart rate values, input several face area images with heart rate values ​​into the facial emotion recognition model for emotion recognition, and obtain several initial emotion recognition labels; perform feature fitting on the heart rate values ​​of the face area images with heart rate values ​​and their corresponding initial emotion recognition labels to obtain the characteristic values ​​of the heart rate emotion recognition labels; perform labeling processing on several face area images with characteristic values ​​of the heart rate emotion recognition labels using the facial emotion labels to obtain the preprocessed face heart rate area image set training set, and set the maximum number of training times; Q4. Input the preprocessed face-heart rate region image set training set into the facial emotion recognition model for training to obtain the initial model for facial emotion and heart rate assisted recognition; take the facial emotion label as the target, determine whether the maximum number of training times has been reached, if so, output the trained initial model for facial emotion and heart rate assisted recognition, and the initial model for facial emotion and heart rate assisted recognition is the facial emotion and heart rate assisted recognition model; otherwise, continue to train the initial model for facial emotion and heart rate assisted recognition by using the preprocessed face-heart rate region image set training set.

8. The intelligent fill light control method according to claim 7, characterized in that: The intelligent fill light parameter adjustment model includes the intelligent fill light strategy definition layer, parameter extraction layer, parameter selection layer, data output layer and lighting control layer; The intelligent fill light strategy definition layer is used to define the fill light parameter mapping rule as light = f(α(t), β(b), γ(c)), where light represents the intelligent fill light strategy, α(t) represents the color temperature adjustment parameter, β(b) represents the brightness adjustment parameter, and γ(c) represents the color adjustment parameter; The parameter extraction layer is used to extract the parameters of the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy to obtain a color temperature adjustment parameter set, a brightness adjustment parameter set and a color adjustment parameter set; The parameter selection layer is used to average the parameters of the color temperature adjustment parameter set to obtain an average color temperature adjustment parameter; Taking an average value of the parameters in the color adjustment parameter set to obtain an average color adjustment parameter; Extracting the maximum parameter in the brightness adjustment parameter set to obtain the maximum brightness adjustment parameter; The data output layer is used to construct an intelligent fill light strategy based on the average color temperature adjustment parameter, the maximum brightness adjustment parameter and the average color adjustment parameter to obtain an instant fill light strategy; The lighting control layer is used to perform intelligent fill light control based on real-time fill light strategy.

9. An intelligent fill light control system, characterized in that: The system applies an intelligent fill light control method according to any one of claims 1 to 8, including: The image acquisition module is used to obtain the front camera image at the current moment and the timestamp at the current moment, obtain the front camera image and the intelligent fill light timestamp; obtain the three-dimensional scene image in the current scene; The image stitching and extraction module is used to perform image segmentation on the front camera image to obtain a person information image and a person background image; perform image fusion on the three-dimensional scene image and the person background image to obtain a scene information image; perform image recognition on the person information image to obtain a face area image; A character portrait fitting module is used to fit the character information image to a user portrait to obtain the character portrait classification information to be tested; the character portrait classification information to be tested is matched with the first intelligent fill-light strategy library to obtain the matched character portrait classification information, and the first intelligent fill-light strategy corresponding to the matched character portrait classification information is output; the first intelligent fill-light strategy library includes standard character portrait classification information and its corresponding first intelligent fill-light strategy; A gray value calculation module is used to grayscale the scene information image using a weighted average method to obtain a scene gray value; compare the scene gray value with a second intelligent fill light strategy library to obtain a second intelligent fill light strategy; the second intelligent fill light strategy library includes a gray threshold range and its corresponding second intelligent fill light strategy; The emotional heart rate recognition module is used to perform non-contact heart rate analysis on the face area image to obtain the non-contact heart rate value of the face; perform emotional analysis on the non-contact heart rate value of the face and the face area image to obtain the facial emotion label; match the facial emotion label in the third intelligent fill light strategy library and output the third intelligent fill light strategy corresponding to the facial emotion label; the third intelligent fill light strategy library includes facial emotion labels and their corresponding third intelligent fill light strategies; The intelligent fill light parameter adjustment module is used to deploy the intelligent fill light parameter adjustment model, input the first intelligent fill light strategy, the second intelligent fill light strategy and the third intelligent fill light strategy into the intelligent fill light parameter adjustment model for data fitting, obtain the real-time fill light strategy, and perform intelligent fill light control based on the real-time fill light strategy.

Images