Stereoscopic scene automatic light regulation and control method and system thereof

By constructing a panoramic image of a three-dimensional scene and adjusting light using artificial intelligence algorithms, the problem of unnatural and lack of flexibility in traditional light control methods is solved, and a more natural lighting effect and higher interaction are achieved, which is suitable for all A kind of scenario.

CN118918236AActive Publication Date: 2024-11-08WUHAN LAZY CAT DIGITAL TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410950088.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Traditional light control methods have problems such as unnatural light and shadow effects, insufficient interaction and lack of flexibility in game scenes, making it difficult to adapt to the lighting needs of different scenarios.

Method used

By constructing a panoramic image of a three-dimensional scene, using an artificial intelligence-based image processing and analysis algorithm to extract the foreground and background features, and using the foreground-background semantic joint interactive network to adjust the light intensity, color and irradiation direction to achieve adaptive light regulation.

Benefits of technology

It enhances the realism and interactivity of the game scene, can adapt to the lighting conditions of indoor and outdoor environments, and meets different shooting or rendering needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918236A_ABST
    Figure CN118918236A_ABST
Patent Text Reader

Abstract

The invention relates to the field of intelligent light control, and particularly discloses an automatic light regulation and control method and system for a three-dimensional scene. Carrying out feature analysis and extraction on foreground features and background features of the panoramic image of the stereoscopic scene by adopting an image processing and analysis algorithm based on artificial intelligence; therefore, the light intensity, the light color and the light irradiation direction are adaptively adjusted based on the semantic joint interaction characteristics between the foreground and the background of the stereoscopic scene. In this way, the light can be intelligently adjusted, so that the scene where the person is located better meets the actual illumination condition, the sense of reality is enhanced, and the light can be adaptively adjusted to adapt to different shooting or rendering requirements, whether indoor scenes or outdoor environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent light control, and more specifically, to a method and system for automatic light control of a stereoscopic scene. Background Art

[0002] With the development of technology, people have higher requirements for the visual realism of game scenes. Ray tracing, as the most commonly used realistic rendering algorithm, plays a huge role in 3D animation, virtual reality, digital twins and other fields.

[0003] However, in traditional light control methods, light sources and shadows are usually pre-calculated and rendered, which may limit the interactivity of the game world and cause the light and shadow effects to appear stiff and unnatural. In addition, some traditional light control techniques are only suitable for specific types of game scenes, but not effective in other scenes and lack flexibility.

[0004] Therefore, an automatic light control solution for a stereoscopic scene is desired. Summary of the invention

[0005] In view of the shortcomings in the prior art, the present application provides a method and system for automatic light control of a stereoscopic scene, which constructs a panoramic image of the stereoscopic scene and uses an image processing and analysis algorithm based on artificial intelligence to perform feature analysis and extraction of the foreground features and background features of the panoramic image of the stereoscopic scene, so as to adaptively adjust the light intensity, light color and light irradiation direction based on the semantic joint interaction features between the foreground and background of the stereoscopic scene. In this way, the light can be intelligently adjusted to make the scene where the characters are located more in line with the actual lighting conditions, enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment.

[0006] According to one aspect of the present application, a method for automatic light control of a stereoscopic scene is provided, comprising:

[0007] Constructing a panoramic image of a stereoscopic scene;

[0008] Extracting a foreground portion and a background portion of the panoramic image to obtain a stereoscopic scene foreground portion and a stereoscopic scene background portion;

[0009] Performing foreground feature extraction and background feature extraction on the foreground part of the stereoscopic scene and the background part of the stereoscopic scene respectively to obtain a feature map of the foreground part of the stereoscopic scene and a feature map of the background part of the stereoscopic scene;

[0010] Inputting the stereoscopic scene foreground feature map and the stereoscopic scene background feature map into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground feature map and an enhanced stereoscopic scene background feature map;

[0011] Inputting the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map into a foreground-background semantic joint interaction network to obtain a stereoscopic scene foreground-background semantic joint interaction feature map as a stereoscopic scene foreground-background semantic joint interaction feature;

[0012] Based on the foreground-background semantic joint interaction feature of the stereoscopic scene, a light parameter decoding result is obtained, and the light parameter decoding result includes light intensity, light color and light irradiation direction.

[0013] According to another aspect of the present application, a stereoscopic scene automatic light control system is provided, comprising:

[0014] A panoramic image construction module, used to construct a panoramic image of a stereoscopic scene;

[0015] A panoramic image segmentation module, used for extracting a foreground portion and a background portion of the panoramic image to obtain a stereoscopic scene foreground portion and a stereoscopic scene background portion;

[0016] A panoramic image segmentation part feature extraction module, used for performing foreground feature extraction and background feature extraction on the stereoscopic scene foreground part and the stereoscopic scene background part respectively to obtain a stereoscopic scene foreground part feature map and a stereoscopic scene background part feature map;

[0017] A panoramic image segmentation part feature enhancement module, used to input the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground part feature map and an enhanced stereoscopic scene background part feature map;

[0018] A panoramic image segmentation part semantic joint interaction module, used for inputting the enhanced stereo scene foreground part feature map and the enhanced stereo scene background part feature map into a foreground-background semantic joint interaction network to obtain a stereo scene foreground-background semantic joint interaction feature map as a stereo scene foreground-background semantic joint interaction feature;

[0019] The light parameter decoding result generating module is used to obtain the light parameter decoding result based on the foreground-background semantic joint interaction feature of the stereoscopic scene, and the light parameter decoding result includes light intensity, light color and light irradiation direction.

[0020] This application has significant technical effects due to the adoption of the above technical solutions:

[0021] The present application provides a method and system for automatic light control of a stereoscopic scene, which constructs a panoramic image of the stereoscopic scene and uses an image processing and analysis algorithm based on artificial intelligence to perform feature analysis and extraction of foreground features and background features of the panoramic image of the stereoscopic scene, so as to adaptively adjust the light intensity, light color and light irradiation direction based on the semantic joint interaction features between the foreground and background of the stereoscopic scene. In this way, the light can be intelligently adjusted to make the scene where the characters are located more in line with the actual lighting conditions, enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 Flow chart of a method for automatic light control of a stereoscopic scene according to an embodiment of the present application.

[0024] Figure 2 Schematic diagram of the architecture of the automatic light control method for a stereoscopic scene according to an embodiment of the present application.

[0025] Figure 3 A flowchart of inputting the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map into the foreground-background semantic joint interaction network to obtain the stereoscopic scene foreground-background semantic joint interaction feature map in the stereoscopic scene automatic light control method according to an embodiment of the present application.

[0026] Figure 4 The present invention is a flowchart for training the foreground feature extractor based on the hole convolutional neural network model, the background feature extractor based on the hole convolutional neural network model, the feature multi-scale perception enhancement module, the foreground-background semantic joint interaction network and the decoder-based light parameter configurator in the automatic light control method for a stereoscopic scene according to an embodiment of the present application.

[0027] Figure 5 4 is a system block diagram of a stereoscopic scene automatic light control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] As technology advances, people have put forward higher standards for the visual realism of game scenes. Ray tracing technology, as a widely used rendering algorithm, has brought significant impacts to application fields such as 3D animation, virtual reality, and digital twins. However, traditional light control methods have limitations. The light source and shadow effects are often pre-set, which may affect the interactivity of the game environment and make the light and shadow effects appear not smooth and realistic enough. In addition, some traditional light control technologies may only be suitable for specific types of game scenes and may perform poorly in other scenes, showing a lack of adaptability.

[0033] Therefore, in response to the above technical problems, the technical concept of the present application is to construct a panoramic image of a stereoscopic scene, and use an artificial intelligence-based image processing and analysis algorithm to perform feature analysis and extraction of the foreground features and background features of the panoramic image of the stereoscopic scene, so as to adaptively adjust the light intensity, light color and light irradiation direction based on the semantic joint interaction features between the foreground and background of the stereoscopic scene. In this way, the light can be intelligently adjusted to make the scene where the character is located more in line with the actual lighting conditions, enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment.

[0034] Figure 1FIG. 1 is a flow chart of a method for automatic light control of a stereoscopic scene according to an embodiment of the present application. Figure 1 As shown, the automatic light control method for a stereoscopic scene according to an embodiment of the present application includes: S110, constructing a panoramic image of a stereoscopic scene; S120, extracting the foreground part and the background part of the panoramic image to obtain a stereoscopic scene foreground part and a stereoscopic scene background part; S130, performing foreground feature extraction and background feature extraction on the stereoscopic scene foreground part and the stereoscopic scene background part respectively to obtain a stereoscopic scene foreground part feature map and a stereoscopic scene background part feature map; S140, inputting the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map into a feature multi-scale perception enhancement module respectively to obtain an enhanced image. The foreground feature map of the stereoscopic scene and the background feature map of the enhanced stereoscopic scene are enhanced; S150, the foreground feature map of the enhanced stereoscopic scene and the background feature map of the enhanced stereoscopic scene are input into the foreground-background semantic joint interaction network to obtain the foreground-background semantic joint interaction feature map of the stereoscopic scene as the foreground-background semantic joint interaction feature of the stereoscopic scene; S160, the foreground feature map of the enhanced stereoscopic scene and the background feature map of the enhanced stereoscopic scene are input into the foreground-background semantic joint interaction network to obtain the foreground-background semantic joint interaction feature map of the stereoscopic scene as the foreground-background semantic joint interaction feature of the stereoscopic scene.

[0035] In step S110, a panoramic image of the stereoscopic scene is constructed. It should be understood that the panoramic image provides 360-degree image information of the stereoscopic scene, which can completely record every detail in the stereoscopic scene, including lighting conditions, object layout, spatial relationships, etc., which provides rich data support for subsequent light control. Based on this, in the technical solution of the present application, it is necessary to first construct a panoramic image of the stereoscopic scene. In particular, in a specific embodiment of the present application, one feasible way to construct a panoramic image of a stereoscopic scene can be to acquire multi-angle scene images around the center of the scene where light control is required, and then use image stitching and stereo correction technology to form a complete panoramic image.

[0036] In step S120, the foreground part and the background part of the panoramic image are extracted to obtain the foreground part and the background part of the stereoscopic scene. Accordingly, considering that the panoramic image of the stereoscopic scene contains the viewing angle information of the entire stereoscopic scene within a range of 360 degrees, and the foreground part and the background part of the panoramic image usually have different light requirements. In detail, the foreground part in the panoramic image refers to the area part that is closer to the observer in the scene and is usually located in the center of the image, while the background part refers to the area part that is farther away from the observer in the scene and is usually located at the edge of the image or at a longer distance. That is, stronger light may be required for the foreground to highlight the details of the object, while soft light may be required for the background to create an atmosphere. Based on this, in order to make the foreground of the stereoscopic scene and the background of the stereoscopic scene more coordinated and unified in light, color and illumination direction, in the technical solution of the present application, the foreground part and the background part of the panoramic image are extracted to obtain the foreground part and the background part of the stereoscopic scene. In particular, in a specific embodiment of the present application, the foreground part and the background part of the panoramic image can be extracted by image segmentation technology to obtain the foreground part and the background part of the stereoscopic scene.

[0037] In step S130, foreground feature extraction and background feature extraction are performed on the foreground part of the stereoscopic scene and the background part of the stereoscopic scene respectively to obtain a feature map of the foreground part of the stereoscopic scene and a feature map of the background part of the stereoscopic scene. Specifically, in an embodiment of the present application, foreground feature extraction and background feature extraction are performed on the foreground part of the stereoscopic scene and the background part of the stereoscopic scene respectively to obtain a feature map of the foreground part of the stereoscopic scene and a feature map of the background part of the stereoscopic scene, including: inputting the foreground part of the stereoscopic scene into a foreground feature extractor based on a dilated convolutional neural network model to obtain the feature map of the foreground part of the stereoscopic scene; inputting the background part of the stereoscopic scene into a background feature extractor based on a dilated convolutional neural network model to obtain the feature map of the background part of the stereoscopic scene. Accordingly, considering that the foreground part of the stereoscopic scene contains important implicit feature information about the foreground of the stereoscopic scene, the dilated convolutional neural network model can expand the receptive field without increasing the number of parameters by introducing the dilated convolution operation, and capture deeper feature information, which is particularly important for processing the foreground part of the stereoscopic scene. Therefore, in the technical solution of the present application, the foreground part of the stereoscopic scene is input into a foreground feature extractor based on a hole convolutional neural network model to extract and capture the deeper foreground feature information hidden in the foreground part of the stereoscopic scene, thereby obtaining a feature map of the foreground part of the stereoscopic scene. Similarly, considering that the background part of the stereoscopic scene also reflects the implicit feature information of the background part, based on this, in order to more accurately extract and mine the deep implicit background feature information of the background part of the stereoscopic scene, in the technical solution of the present application, the background part of the stereoscopic scene is input into a background feature extractor based on a hole convolutional neural network model to obtain a feature map of the background part of the stereoscopic scene.

[0038] In step S140, the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map are respectively input into the feature multi-scale perception enhancement module to obtain the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map. It should be understood that considering that the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map both express the local detail feature information and the overall global features about the foreground and background of the stereoscopic scene. Therefore, in order to understand and analyze the details and overall feature information expressed by different local areas of the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map in a more fine-grained manner to obtain a comprehensive and detailed feature representation, in the technical solution of the present application, the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map are respectively input into the feature multi-scale perception enhancement module to obtain the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map.

[0039] It is worth mentioning that the feature multi-scale perception enhancement module performs feature enhancement processing on the input feature map by performing feature processing of three different branches on the input feature map to capture and extract local fine-grained information and overall global information in the input feature map respectively. Specifically, in the first branch, the local fine-grained feature information about the foreground part is extracted by performing channel compression, global mean pooling and nonlinear activation processing on the foreground part feature map of the stereoscopic scene. In the second branch, the foreground part feature map of the stereoscopic scene is subjected to point convolution processing to retain features useful for specific tasks, making the foreground part features more compact and efficient. In the third branch, the foreground part feature map of the stereoscopic scene is subjected to hole convolution coding to expand the global perception information of the foreground part features. Then, the features obtained from the first branch and the third branch are fused and added with the features of the second branch, and then the receptive field is expanded again to enhance the important details and overall structural feature information in the foreground part feature map of the stereoscopic scene, thereby obtaining an enhanced foreground part feature map of the stereoscopic scene containing rich semantic information. In particular, the processing method of the feature map of the background part of the stereoscopic scene is consistent with the processing method of the feature map of the foreground part of the stereoscopic scene.

[0040] Specifically, in an embodiment of the present application, the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map are respectively input into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground part feature map and an enhanced stereoscopic scene background part feature map, including: in a first branch, performing first channel feature extraction on the stereoscopic scene foreground part feature map to obtain a first channel stereoscopic scene foreground part local activation feature vector; in a second branch, performing point convolution processing on the stereoscopic scene foreground part feature map to obtain a second channel compressed stereoscopic scene foreground part feature map; in a third branch, performing receptive field expansion on the stereoscopic scene foreground part feature map to obtain a stereoscopic scene foreground part receptive field expansion global activation feature matrix; and combining the stereoscopic scene foreground part receptive field expansion global activation feature matrix with the second channel compressed stereoscopic field The corresponding feature matrices of the foreground part feature map of the scene along the channel dimension are multiplied by position to obtain the global activation feature map of the foreground part of the second channel compressed stereo scene; the local activation feature vector of the foreground part of the first channel stereo scene is multiplied by position with the feature matrices of the foreground part of the second channel compressed stereo scene to obtain the local activation feature map of the foreground part of the second channel compressed stereo scene; the global activation feature map of the foreground part of the second channel compressed stereo scene and the local activation feature map of the foreground part of the second channel compressed stereo scene are added by position to obtain the multi-scale fusion activation feature map of the foreground part of the second channel compressed stereo scene; the multi-scale fusion activation feature map of the foreground part of the second channel compressed stereo scene is subjected to hole convolution coding to obtain the enhanced stereo scene foreground feature map.

[0041] More specifically, in an embodiment of the present application, in a first branch, a first channel feature extraction is performed on the stereoscopic scene foreground part feature map to obtain a first channel stereoscopic scene foreground part local activation feature vector, including: performing point convolution processing on the stereoscopic scene foreground part feature map to obtain a first channel stereoscopic scene foreground part compression feature map; performing global mean pooling on each feature matrix along the channel dimension in the first channel stereoscopic scene foreground part compression feature map to obtain a first channel stereoscopic scene foreground part compression feature vector; performing nonlinear activation on the first channel stereoscopic scene foreground part compression feature vector to obtain the first channel stereoscopic scene foreground part local activation feature vector.

[0042] More specifically, in an embodiment of the present application, in the third branch, the receptive field of the feature map of the foreground part of the stereoscopic scene is expanded to obtain a global activation feature matrix of the receptive field expansion of the foreground part of the stereoscopic scene, including: performing hole convolution encoding on the feature map of the foreground part of the stereoscopic scene to obtain a feature map of the receptive field expansion of the foreground part of the stereoscopic scene; performing point convolution processing on the receptive field expansion feature map of the foreground part of the stereoscopic scene to obtain a global feature matrix of the receptive field expansion of the foreground part of the stereoscopic scene; performing nonlinear activation on the global feature matrix of the receptive field expansion of the foreground part of the stereoscopic scene to obtain the global activation feature matrix of the receptive field expansion of the foreground part of the stereoscopic scene.

[0043] In an embodiment of the present application, specifically, the stereoscopic scene foreground feature map and the stereoscopic scene background feature map are respectively input into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground feature map and an enhanced stereoscopic scene background feature map, including: inputting the stereoscopic scene foreground feature map into the feature multi-scale perception enhancement module, and processing it with the following enhancement formula to obtain the enhanced stereoscopic scene foreground feature map; wherein the enhancement formula is:

[0044]

[0045] Wherein, F represents the feature map of the foreground of the stereoscopic scene, C1(·) represents point convolution processing on the feature map, Avg(·) represents global mean pooling processing on each feature matrix along the channel dimension in the feature map, δ(·) represents nonlinear activation processing, and C 3,2 and C 3,3 They are 3×3 dilated convolution operations with dilation numbers of 3 and 2, respectively. is added by position, is multiplication by position, F cIn particular, the encoding method of the feature map of the background part of the stereoscopic scene is consistent with the encoding method of the feature map of the foreground part of the stereoscopic scene.

[0046] In step S150, the enhanced stereo scene foreground feature map and the enhanced stereo scene background feature map are input into the foreground-background semantic joint interaction network to obtain the stereo scene foreground-background semantic joint interaction feature map as the stereo scene foreground-background semantic joint interaction feature. It should be understood that in order to promote the feature information exchange and interaction between the enhanced stereo scene foreground feature map and the enhanced stereo scene background feature map, so that the model can better understand the semantic interaction relationship between the stereo scene foreground feature and the stereo scene background feature, so as to make the subsequent light adjustment more accurate, in the technical solution of the present application, the enhanced stereo scene foreground feature map and the enhanced stereo scene background feature map are input into the foreground-background semantic joint interaction network to obtain the stereo scene foreground-background semantic joint interaction feature map. That is to say, the foreground-background semantic joint interaction network can effectively integrate the enhanced stereo scene foreground feature map and the enhanced stereo scene background feature map, and strengthen the complex semantic association interactivity between the stereo scene foreground and the stereo scene background, so as to more accurately capture the overall semantic information of the stereo scene, improve the model's ability to understand the entire stereo scene, and make light regulation and visual effect optimization more accurate and effective.

[0047] Specifically, Figure 3 The flowchart of the method for automatic light control of a stereoscopic scene according to an embodiment of the present application is to input the enhanced stereoscopic scene foreground feature map and the enhanced stereoscopic scene background feature map into the foreground-background semantic joint interaction network to obtain the stereoscopic scene foreground-background semantic joint interaction feature map. Figure 3As shown, the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map are input into the foreground-background semantic joint interaction network to obtain the stereoscopic scene foreground-background semantic joint interaction feature map, including: S151, shape transformation of the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map to obtain the enhanced stereoscopic scene foreground part feature matrix and the enhanced stereoscopic scene background part feature matrix; S152, calculating the matrix multiplication between the enhanced stereoscopic scene foreground part feature matrix and the enhanced stereoscopic scene background part feature matrix to obtain the enhanced stereoscopic scene foreground background correlation feature matrix; S153, after inputting the enhanced stereoscopic scene foreground background correlation feature matrix into the Softmax function to obtain the enhanced stereoscopic scene foreground background dependency matrix, linear interpolation is performed on the enhanced stereoscopic scene foreground background dependency matrix to obtain the enhanced stereoscopic scene foreground background dependency interaction matrix; S154, shape transformation is performed on the feature matrix obtained by matrix multiplication of the enhanced stereoscopic scene foreground background dependency interaction matrix and the enhanced stereoscopic scene foreground part feature matrix to obtain the stereoscopic scene foreground-background semantic joint interaction feature map.

[0048] In step S160, based on the foreground-background semantic joint interaction feature of the stereoscopic scene, a light parameter decoding result is obtained, and the light parameter decoding result includes light intensity, light color and light irradiation direction. Specifically, in an embodiment of the present application, based on the foreground-background semantic joint interaction feature of the stereoscopic scene, a light parameter decoding result is obtained, including: inputting the foreground-background semantic joint interaction feature map of the stereoscopic scene into a light parameter configurator based on a decoder to obtain the light parameter decoding result. That is, the foreground-background semantic joint interaction feature of the stereoscopic scene obtained by semantic joint interaction of the foreground part feature map of the enhanced stereoscopic scene and the background part feature map of the enhanced stereoscopic scene is decoded and processed, so as to adaptively adjust the light intensity, light color and light irradiation direction. In this way, the light can be intelligently adjusted to make the scene where the character is located more in line with the actual lighting conditions, enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment.

[0049] It is worth mentioning that those skilled in the art should be aware that before applying a deep neural network model for inference, the deep neural network model must be trained so that the deep neural network can implement specific functional capabilities.

[0050] Specifically, in an embodiment of the present application, a training step is also included: for training the foreground feature extractor based on the hole convolutional neural network model, the background feature extractor based on the hole convolutional neural network model, the feature multi-scale perception enhancement module, the foreground-background semantic joint interaction network and the decoder-based light parameter configurator.

[0051] Figure 4 FIG. 1 is a flow chart of the training steps in the method for automatic light control of a stereoscopic scene according to an embodiment of the present application. Figure 4 As shown, the training step includes: S210, obtaining training data, the training data including a training panoramic image for constructing a stereoscopic scene, and a real light parameter decoding result, the real light parameter decoding result being the real value of light intensity, light color and light irradiation direction; S220, extracting the foreground part and the background part of the training panoramic image to obtain a training stereo scene foreground part and a training stereo scene background part; S230, inputting the training stereo scene foreground part into the foreground feature extractor based on the hole convolutional neural network model to obtain a training stereo scene foreground part feature map; S240, inputting the training stereo scene background part into the background feature extractor based on the hole convolutional neural network model to obtain a training stereo scene background part feature map; S250, respectively inputting the training stereo scene foreground part feature map and the training stereo scene background part feature map into the feature multi-scale perception enhancement module to obtain a training enhanced stereo scene foreground part feature map and Training and enhancing the background feature map of the stereoscopic scene; S260, inputting the training enhanced stereoscopic scene foreground feature map and the training enhanced stereoscopic scene background feature map into the foreground-background semantic joint interaction network to obtain the training stereoscopic scene foreground-background semantic joint interaction feature map; S270, inputting the training stereoscopic scene foreground-background semantic joint interaction feature map into the decoder-based light parameter configurator to obtain the training light parameter decoding result; S280, calculating the cross entropy loss function value between the training light parameter decoding result and the real light parameter decoding result to obtain the decoding loss function value; S290, based on the decoding loss function value and through gradient descent back propagation, the foreground feature extractor based on the void convolutional neural network model, the background feature extractor based on the void convolutional neural network model, the feature multi-scale perception enhancement module, the foreground-background semantic joint interaction network and the decoder-based light parameter configurator are trained.

[0052] It should be understood that, here, the training enhanced stereo scene foreground feature map and the training enhanced stereo scene background feature map respectively express the image semantic features of the multi-scale perceptual enhancement of the image semantic spatial distribution of the foreground part and the background part of the training panoramic image. In this way, after the training enhanced stereo scene foreground feature map and the training enhanced stereo scene background feature map are input into the foreground-background semantic joint interaction network, the training stereo scene foreground-background semantic joint interaction feature map will also have feature aggregation decoding regression imbalance due to the difference in foreground-background semantic interaction weight distribution, and make the probability density distribution regression convergence logic of in-class features and out-of-class features after the clustering operation inconsistent.

[0053] Based on this, in the preferred embodiment, when the foreground feature extractor based on the hole convolutional neural network model, the background feature extractor based on the hole convolutional neural network model, the feature multi-scale perception enhancement module, the foreground-background semantic joint interaction network and the decoder-based light parameter configurator are trained based on the classification loss function value, wherein, in each iteration of the training, the foreground-background semantic joint interaction feature map of the training stereo scene is iteratively optimized,

[0054] The iterative optimization process includes: performing a clustering operation on the training stereo scene foreground-background semantic joint interaction feature map, for example, performing a clustering operation based on the distance between eigenvalues, and determining the number of eigenvalues ​​in the clustering set obtained by the clustering operation, that is, the number of in-class features, and subtracting the number of in-class features from the total number of eigenvalues ​​of the training stereo scene foreground-background semantic joint interaction feature map to obtain the number of out-of-class features; dividing the total number of eigenvalues ​​by the number of in-class features and the number of out-of-class features respectively to obtain a class importance value and an out-of-class proportion value, and calculating the inverse of the class importance value to obtain a class constraint value; calculating the training stereo scene foreground-background semantic joint interaction feature map; Each eigenvalue of the feature map is a power function with the class importance value as the exponent, and is added to an exponential value with the natural function as the base and the class constraint value as the exponent, and then multiplied by the out-of-class proportion value to obtain a training stereo scene foreground-background semantic joint interaction modulation map; the training stereo scene foreground-background semantic joint interaction feature map is dot-multiplied with the class importance value to obtain a training stereo scene foreground-background semantic joint interaction ontology map; the weighted sum of the training stereo scene foreground-background semantic joint interaction modulation map and the training stereo scene foreground-background semantic joint interaction ontology map is calculated with a weight hyperparameter to obtain an optimized training stereo scene foreground-background semantic joint interaction feature map.

[0055] Specifically, in this preferred embodiment, the training stereo scene foreground-background semantic joint interaction feature map is optimized to obtain an optimized training stereo scene foreground-background semantic joint interaction feature map, and the process formula is expressed as follows:

[0056]

[0057] Wherein, F is the training stereo scene foreground-background semantic joint interaction feature map, n is the total number of feature values ​​of the training stereo scene foreground-background semantic joint interaction feature map, k is the number of intra-class features of the training stereo scene foreground-background semantic joint interaction feature map, ⊙ is the position point multiplication, β is the weight hyperparameter, is added by position, represents an exponential operation on the feature values ​​at each position in the feature map, and F′ is the optimized training stereo scene foreground-background semantic joint interaction feature map.

[0058] Therefore, while performing clustering operation on the training stereo scene foreground-background semantic joint interaction feature map, the clustering importance measure of the feature value of the training stereo scene foreground-background semantic joint interaction feature map is combined with the class constraint measure of the clustering operation to further perform responsive modulation with the out-of-class scale factor, and based on the clustering scaling of the feature ontology representation of the training stereo scene foreground-background semantic joint interaction feature map, the clustering operation is established for the overall feature purification simplicity and effectiveness of the feature set of the training stereo scene foreground-background semantic joint interaction feature map, and the probability density distribution regression convergence logic inconsistency caused by the clustering operation is suppressed, so as to improve the decoding regression iteration effect of the light parameter configurator based on the decoder of the feature set input of the training stereo scene foreground-background semantic joint interaction feature map, that is, to improve the speed of decoding training and the accuracy of decoding results. In this way, the light can be intelligently adjusted to make the scene more in line with the actual lighting conditions, enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment.

[0059] In summary, the automatic light control method for a stereoscopic scene based on the embodiment of the present application is explained, which constructs a panoramic image of the stereoscopic scene and uses an artificial intelligence-based image processing and analysis algorithm to perform feature analysis and extraction of the foreground features and background features of the panoramic image of the stereoscopic scene, so as to adaptively adjust the light intensity, light color and light irradiation direction based on the semantic joint interaction features between the foreground and background of the stereoscopic scene. In this way, the light can be intelligently adjusted to make the scene where the character is located more in line with the actual lighting conditions and enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment.

[0060] Figure 5 FIG. 1 is a system block diagram of a stereoscopic scene automatic light control system according to an embodiment of the present application. Figure 5 As shown, according to the embodiment of the present application, the automatic light control system 100 for a stereoscopic scene includes: a panoramic image construction module 110, which is used to construct a panoramic image of a stereoscopic scene; a panoramic image segmentation module 120, which is used to extract the foreground part and the background part of the panoramic image to obtain a stereoscopic scene foreground part and a stereoscopic scene background part; a panoramic image segmentation part feature extraction module 130, which is used to perform foreground feature extraction and background feature extraction on the stereoscopic scene foreground part and the stereoscopic scene background part respectively to obtain a stereoscopic scene foreground part feature map and a stereoscopic scene background part feature map; a panoramic image segmentation part feature enhancement module 140, which is used to extract the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map. The segmented feature maps are respectively input into the feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground feature map and an enhanced stereoscopic scene background feature map; the panoramic image segmentation part semantic joint interaction module 150 is used to input the enhanced stereoscopic scene foreground feature map and the enhanced stereoscopic scene background feature map into the foreground-background semantic joint interaction network to obtain a stereoscopic scene foreground-background semantic joint interaction feature map as a stereoscopic scene foreground-background semantic joint interaction feature; the light parameter decoding result generation module 160 is used to obtain a light parameter decoding result based on the stereoscopic scene foreground-background semantic joint interaction feature, and the light parameter decoding result includes light intensity, light color and light irradiation direction.

[0061] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the above-mentioned stereoscopic scene automatic light control system 100 have been described in detail above. Figures 1 to 4 It has been introduced in detail in the description of the automatic light control method for a stereoscopic scene, and therefore, its repeated description will be omitted.

[0062] In summary, the automatic light control system 100 for a stereoscopic scene based on the embodiment of the present application is explained, which constructs a panoramic image of a stereoscopic scene and uses an image processing and analysis algorithm based on artificial intelligence to perform feature analysis and extraction of foreground features and background features of the panoramic image of the stereoscopic scene, so as to adaptively adjust the light intensity, light color and light irradiation direction based on the semantic joint interaction features between the foreground and background of the stereoscopic scene. In this way, the light can be intelligently adjusted to make the scene where the character is located more in line with the actual lighting conditions, enhance the sense of reality, and the adaptive adjustment of the light can adapt to different shooting or rendering requirements, whether it is an indoor scene or an outdoor environment.

[0063] As described above, the stereoscopic scene automatic light control system 100 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server for stereoscopic scene automatic light control. In one example, the stereoscopic scene automatic light control system 100 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the stereoscopic scene automatic light control system 100 can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the stereoscopic scene automatic light control system 100 can also be one of the many hardware modules of the wireless terminal.

[0064] Alternatively, in another example, the stereoscopic scene automatic light control system 100 and the wireless terminal may also be separate devices, and the stereoscopic scene automatic light control system 100 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0065] The above is merely an example of the principles of the present disclosure, and various modifications may be made by those skilled in the art without departing from the scope of the present disclosure. The above embodiments are presented for the purpose of illustration rather than limitation. The present disclosure may also take many forms other than those explicitly described herein.

Claims

1. A method for automatic light control of a stereoscopic scene, characterized in that: include: Constructing a panoramic image of a stereoscopic scene; Extracting a foreground portion and a background portion of the panoramic image to obtain a stereoscopic scene foreground portion and a stereoscopic scene background portion; Performing foreground feature extraction and background feature extraction on the foreground part of the stereoscopic scene and the background part of the stereoscopic scene respectively to obtain a feature map of the foreground part of the stereoscopic scene and a feature map of the background part of the stereoscopic scene; Inputting the stereoscopic scene foreground feature map and the stereoscopic scene background feature map into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground feature map and an enhanced stereoscopic scene background feature map; Inputting the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map into a foreground-background semantic joint interaction network to obtain a stereoscopic scene foreground-background semantic joint interaction feature map as a stereoscopic scene foreground-background semantic joint interaction feature; Based on the foreground-background semantic joint interaction feature of the stereoscopic scene, a light parameter decoding result is obtained, and the light parameter decoding result includes light intensity, light color and light irradiation direction.

2. The method for automatic light control of a stereoscopic scene according to claim 1, characterized in that: The foreground feature extraction and the background feature extraction are respectively performed on the foreground part of the stereoscopic scene and the background part of the stereoscopic scene to obtain a feature map of the foreground part of the stereoscopic scene and a feature map of the background part of the stereoscopic scene, including: Inputting the foreground part of the stereoscopic scene into a foreground feature extractor based on a dilated convolutional neural network model to obtain a feature map of the foreground part of the stereoscopic scene; The background part of the stereoscopic scene is input into a background feature extractor based on a hole convolutional neural network model to obtain a feature map of the background part of the stereoscopic scene.

3. The method for automatic light control of a stereoscopic scene according to claim 2, characterized in that: Inputting the stereoscopic scene foreground feature map and the stereoscopic scene background feature map into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground feature map and an enhanced stereoscopic scene background feature map, including: In the first branch, first channel feature extraction is performed on the stereoscopic scene foreground part feature map to obtain a first channel stereoscopic scene foreground part local activation feature vector; In the second branch, point convolution processing is performed on the stereoscopic scene foreground part feature map to obtain a second channel compressed stereoscopic scene foreground part feature map; In the third branch, the receptive field of the foreground part feature map of the stereoscopic scene is expanded to obtain a receptive field expansion global activation feature matrix of the foreground part of the stereoscopic scene; Multiplying the receptive field expansion global activation feature matrix of the stereoscopic scene foreground part by position with each corresponding feature matrix of the second channel compressed stereoscopic scene foreground part feature map along the channel dimension to obtain the second channel compressed stereoscopic scene foreground part global activation feature map; Multiplying the local activation feature vector of the foreground part of the stereoscopic scene of the first channel by each feature matrix along the channel dimension of the feature map of the foreground part of the compressed stereoscopic scene of the second channel by position to obtain the local activation feature map of the foreground part of the compressed stereoscopic scene of the second channel; Adding the second channel compressed stereo scene foreground part global activation feature map and the second channel compressed stereo scene foreground part local activation feature map according to position to obtain a second channel compressed stereo scene foreground part multi-scale fused activation feature map; The multi-scale fused activation feature map of the foreground part of the second channel compressed stereoscopic scene is subjected to dilated convolution coding to obtain the enhanced stereoscopic scene foreground part feature map.

4. The method for automatic light control of a stereoscopic scene according to claim 3, characterized in that: In the first branch, performing first channel feature extraction on the stereoscopic scene foreground part feature map to obtain a first channel stereoscopic scene foreground part local activation feature vector, including: Performing point convolution processing on the stereoscopic scene foreground part feature map to obtain a first channel stereoscopic scene foreground part compressed feature map; Performing global mean pooling on each feature matrix along the channel dimension in the first channel stereoscopic scene foreground part compression feature map to obtain a first channel stereoscopic scene foreground part compression feature vector; Nonlinear activation is performed on the compressed feature vector of the foreground part of the stereoscopic scene of the first channel to obtain a local activated feature vector of the foreground part of the stereoscopic scene of the first channel.

5. The method for automatic light control of a stereoscopic scene according to claim 4, characterized in that: In the third branch, the receptive field expansion is performed on the feature map of the foreground part of the stereoscopic scene to obtain a receptive field expansion global activation feature matrix of the foreground part of the stereoscopic scene, including: Performing dilated convolution coding on the feature map of the foreground part of the stereoscopic scene to obtain a feature map of the receptive field expansion of the foreground part of the stereoscopic scene; Performing point convolution processing on the receptive field expansion feature map of the foreground part of the stereoscopic scene to obtain a global feature matrix of the receptive field expansion of the foreground part of the stereoscopic scene; The receptive field expansion global feature matrix of the foreground part of the stereoscopic scene is nonlinearly activated to obtain the receptive field expansion global activation feature matrix of the foreground part of the stereoscopic scene.

6. The method for automatic light control of a stereoscopic scene according to claim 5, characterized in that: Inputting the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map into a foreground-background semantic joint interaction network to obtain a stereoscopic scene foreground-background semantic joint interaction feature map, including: Performing shape transformation on the enhanced stereoscopic scene foreground part feature map and the enhanced stereoscopic scene background part feature map to obtain an enhanced stereoscopic scene foreground part feature matrix and an enhanced stereoscopic scene background part feature matrix; Calculating matrix multiplication between the enhanced stereoscopic scene foreground part feature matrix and the enhanced stereoscopic scene background part feature matrix to obtain an enhanced stereoscopic scene foreground-background correlation feature matrix; After inputting the enhanced stereoscopic scene foreground-background correlation feature matrix into a Softmax function to obtain an enhanced stereoscopic scene foreground-background dependency matrix, linear interpolation is performed on the enhanced stereoscopic scene foreground-background dependency matrix to obtain an enhanced stereoscopic scene foreground-background dependency interaction matrix; The feature matrix obtained by matrix multiplication of the enhanced stereoscopic scene foreground-background dependency interaction matrix and the enhanced stereoscopic scene foreground part feature matrix is ​​transformed in shape to obtain the stereoscopic scene foreground-background semantic joint interaction feature map.

7. The method for automatic light control of a stereoscopic scene according to claim 6, characterized in that: Based on the stereoscopic scene foreground-background semantic joint interaction feature, a light parameter decoding result is obtained, including: inputting the stereoscopic scene foreground-background semantic joint interaction feature map into a decoder-based light parameter configurator to obtain the light parameter decoding result.

8. The method for automatic light control of a stereoscopic scene according to claim 7, characterized in that: It also includes a training step: for training the foreground feature extractor based on the hole convolutional neural network model, the background feature extractor based on the hole convolutional neural network model, the feature multi-scale perception enhancement module, the foreground-background semantic joint interaction network and the decoder-based light parameter configurator.

9. The method for automatic light control of a stereoscopic scene according to claim 8, characterized in that: The training step comprises: Acquire training data, where the training data includes a training panoramic image for constructing a stereoscopic scene, and a real light parameter decoding result, where the real light parameter decoding result is a real value of light intensity, light color, and light irradiation direction; Extracting the foreground portion and the background portion of the training panoramic image to obtain a training stereoscopic scene foreground portion and a training stereoscopic scene background portion; Inputting the foreground part of the training stereoscopic scene into the foreground feature extractor based on the hole convolutional neural network model to obtain a feature map of the foreground part of the training stereoscopic scene; Inputting the training stereoscopic scene background part into the background feature extractor based on the hole convolutional neural network model to obtain a training stereoscopic scene background part feature map; Inputting the training stereo scene foreground part feature map and the training stereo scene background part feature map into the feature multi-scale perception enhancement module to obtain the training enhanced stereo scene foreground part feature map and the training enhanced stereo scene background part feature map; Inputting the training enhanced stereo scene foreground part feature map and the training enhanced stereo scene background part feature map into the foreground-background semantic joint interaction network to obtain a training stereo scene foreground-background semantic joint interaction feature map; Inputting the training stereoscopic scene foreground-background semantic joint interaction feature map into the decoder-based light parameter configurator to obtain a training light parameter decoding result; Calculating a cross entropy loss function value between the training light parameter decoding result and the real light parameter decoding result to obtain a decoding loss function value; The foreground feature extractor based on the hole convolutional neural network model, the background feature extractor based on the hole convolutional neural network model, the feature multi-scale perception enhancement module, the foreground-background semantic joint interaction network and the decoder-based light parameter configurator are trained based on the decoding loss function value and through back propagation of gradient descent.

10. A three-dimensional scene automatic light control system, characterized in that: include: A panoramic image construction module, used to construct a panoramic image of a stereoscopic scene; A panoramic image segmentation module, used for extracting a foreground portion and a background portion of the panoramic image to obtain a stereoscopic scene foreground portion and a stereoscopic scene background portion; A panoramic image segmentation part feature extraction module, used for performing foreground feature extraction and background feature extraction on the stereoscopic scene foreground part and the stereoscopic scene background part respectively to obtain a stereoscopic scene foreground part feature map and a stereoscopic scene background part feature map; A panoramic image segmentation part feature enhancement module, used to input the stereoscopic scene foreground part feature map and the stereoscopic scene background part feature map into a feature multi-scale perception enhancement module to obtain an enhanced stereoscopic scene foreground part feature map and an enhanced stereoscopic scene background part feature map; A panoramic image segmentation part semantic joint interaction module, used for inputting the enhanced stereo scene foreground part feature map and the enhanced stereo scene background part feature map into a foreground-background semantic joint interaction network to obtain a stereo scene foreground-background semantic joint interaction feature map as a stereo scene foreground-background semantic joint interaction feature; The light parameter decoding result generating module is used to obtain the light parameter decoding result based on the foreground-background semantic joint interaction feature of the stereoscopic scene, and the light parameter decoding result includes light intensity, light color and light irradiation direction.

Citation Information

Patent Citations

  • Image semantic segmentation method and system based on semantic propagation and foreground and background perception

    CN114494699A

  • Multi-scale feature fusion bridge detection method based on depth separable convolution

    CN115223017A

  • Plant chlorophyll fluorescence three-dimensional imaging device and method

    CN116678862A

  • Multi-modal ocean scene video description algorithm based on instance segmentation auxiliary information

    CN116778382A

  • Image illumination enhancement method and system based on saliency region

    CN118195983A