A method for setting photosensitivity of digital twin scenes

Abstract

The present invention relates to a method for setting photosensitivity of a digital twin scene, which comprises the following steps: Step 1: obtaining the perception results of light intensity and color by a camera installed in the real world and feeding back the parameters, and storing them in a digital twin computing unit; Step 2: presetting a set of light parameters in the digital twin unit according to real-world data; Step 3: starting the camera and obtaining the camera environment perception results, and sending them to the digital twin computing unit; Step 4: the digital twin computing unit performs weighted balancing according to the camera environment perception results based on a scene-by-scene photometry method; Step 5: performing secondary compensation on the digital twin world according to the light intensity of the real world; Step 6: displaying the digital twin world; Step 7: recording the currently set light parameters as historical data. Compared with the prior art, the present invention has the advantages of real-time adjustment of the light effect of the virtual world to achieve a visual effect close to that of the real world.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and in particular to a method for setting photosensitivity of a digital twin scene. Background Art

[0002] Digital twins make full use of physical models, sensor updates, operation history and other data, integrate multi-disciplinary, multi-physical quantity, multi-scale, and multi-probability simulation processes, complete mapping in virtual space, and thus reflect the entire life cycle of the corresponding physical equipment. Digital twins are a concept that transcends reality and can be regarded as a digital mapping system of one or more important and interdependent equipment systems. Digital twins are a universally adaptable theoretical and technical system that can be applied in many fields. They are widely used in product design, product manufacturing, medical analysis, engineering construction and other fields. The most in-depth application in China is in the field of engineering construction, and the field of intelligent manufacturing has received the highest attention and the hottest research.

[0003] The most important inspiration of digital twins lies in their ability to enable feedback from real-world physical systems to digital models in cyberspace. This is a feat of reverse thinking in the industrial field, attempting to embed everything that happens in the physical world into the digital space. Only full-life tracking with feedback loops can truly achieve a full lifecycle concept. This ensures the coordination and consistency between the digital and physical worlds throughout the entire lifecycle. Simulations, analyses, data accumulation, mining, and even AI applications based on digital models can ensure their applicability to real-world physical systems. This is the significance of digital twins for intelligent manufacturing.

[0004] Applications of digital twins: The intelligence of intelligent systems must first be based on perception and modeling, followed by analysis and reasoning. Without a digital twin that accurately models the real-world production system, the so-called intelligent manufacturing system is useless and cannot be implemented. Simulating optimal solutions for smart transportation, and combining real-world scenarios, demonstrates the value of digital twin technology in improving autonomous driving training efficiency, facilitating traffic accident analysis, and traffic control. Autonomous driving virtual simulation testing, used to test autonomous driving perception algorithms, requires lighting settings that are as close to real-world conditions as possible. Traffic accident analysis: Again, taking a truck accident as an example, once the accident scene and the trajectories of traffic participants are tracked and restored, the accident process can be observed from multiple angles. Through freeze-frame processing, it can be seen from inside the vehicle that the truck driver actually did not see the passing cyclist. Therefore, digital twin technology has great application value in traffic accident analysis, helping to trace and analyze the specific causes of accidents and identify those responsible. Traffic control: Using digital twin technology to simulate urban traffic conditions and then optimize traffic control strategies through evaluation and deduction is a key application scenario for digital twins in empowering smart transportation. This involves four main functional aspects:

[0005] The first is monitoring and discovery. Through the digital twin system, a closed loop of information acquisition and control can be created to achieve full process control. More importantly, in a very large and complex scenario, some key problems can be discovered and dealt with in a timely manner. For example, in autumn and winter, fog frequently occurs on some sections of Anhui Expressway. Fog has the characteristics of low visibility, strong suddenness, and difficult weather forecasting, which can easily cause traffic accidents. Digital twin technology enables real-time monitoring of dynamic sensor data, enabling timely detection of fog formation and warnings. Second, it enables deduction and prediction. Once data is in hand, micro-behavioral models can be created for some participants. Simulating a large number of traffic agents yields macro-level simulation results, allowing us to deduce the course of events and achieve predictive capabilities. Third, it enables evaluation and optimization of countermeasures. Through massively parallel computing, simulation results from numerous parallel worlds can be simultaneously evaluated. Traffic control plans can then be continuously refined through techniques like reinforcement learning. Fourth, historical tracing and retrospective research: After an incident occurs, a digital twin system can be used to reconstruct the entire process of the accident, exploring whether each response was effective and whether there was room for improvement. This is a unique capability of digital twin technology.

[0006] The current digital twin virtual scene lighting conditions are set according to preset parameters and cannot simulate the real-world lighting intensity and light and dark changes in different areas in real time. For example, in the virtual scene, the light intensity at different times and in different weather conditions is pre-set, and the scene is modeled and displayed according to the orientation. It can also show the impact of the surrounding environment on light: refraction, reflection, and shadows. However, the actual light intensity in the real scene and the light intensity blocked by clouds in different weather conditions are not set as real parameters. As a result, there is a large deviation in light brightness between the virtual scene and the real scene under the digital twin. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for setting the photosensitivity of a digital twin scene in order to overcome the defects of the above-mentioned prior art.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A method for setting photosensitivity of a digital twin scene, comprising the following steps:

[0010] Step 1: Obtain the perception results of light intensity and color from the camera installed in the real world and feedback the parameters, and store the camera parameters in the digital twin computing unit;

[0011] Step 2: Record real-world data and preset a set of light parameters in the digital twin unit based on the real-world data for presetting the light in the virtual world;

[0012] Step 3: Start multiple cameras distributed in different locations in the real world, obtain the camera environment perception results of the cameras, and send them to the digital twin computing unit;

[0013] Step 4: The digital twin computing unit uses a scene-by-scene photometry method to perform a weighted balance based on the camera's environmental perception results, adjusting the virtual scene in the digital twin world in real time to achieve a lighting effect close to that of the real world.

[0014] Step 5: Measure the lighting conditions of the virtual scene surrounding the digital twin world and perform secondary compensation based on the light intensity in the real world;

[0015] Step 6: Display the digital twin world;

[0016] Step 7: Record the currently set light parameters as historical data.

[0017] In step 1, the camera parameters include the current ISO, shutter speed, white balance, aperture size, and color temperature.

[0018] In step 2, the real-world data includes the current time, current season, current weather, current ambient temperature and historical data, which is used to obtain the historical scene that best matches the current environment through big data, and call the parameters in the real-world data as preset data.

[0019] In step 2, the light parameters include the intensity of direct light, the intensity of reflected light, the spectrum of reflected light, the direction of refracted light, diffracted light, the intensity and spectral composition of light.

[0020] In step 3, the camera environment perception result is the perception result obtained by the CCD or CMOS sensor in the camera, including the ambient light intensity and the captured image.

[0021] In step 4, the digital twin world is adjusted by the principle of weighted balance of multiple cameras. The principle of weighted balance of multiple cameras is equivalent to the metering principle of multiple photosensitive area arrays in a camera. That is, all cameras in an environmental area are regarded as a huge single camera photosensitive device. The relevant photosensitive processing technology of a single digital camera is used to process the original image output by the image sensor to obtain the effect of restoring the scene. Among them, the 3A algorithms for processing the original image output by the image sensor are automatic white balance, automatic exposure and autofocus. The digital twin calculation unit performs weighted balance according to the camera environment perception results based on the scene-based photometry method. Specifically:

[0022] In the digital twin world, the light of a static object or a type of dynamic object is set, and the light of other environments is weighted and balanced based on the set static object or dynamic object and the light intensity in the real world. In different scenarios, different digital twin world photosensitivity methods and metering modes are switched to adjust the light conditions of the digital twin world: when adjusting the digital twin world globally, metering is performed based on the partition metering algorithm, and the multi-camera comprehensive metering is weighted for the light intensity measured by each camera; when adjusting an object in the virtual scene, metering is performed based on the spot metering algorithm and automatic white balance through the camera closest to the object or the camera that captures the object to obtain the brightness; when adjusting a set street scene, metering is performed based on a combination of multiple cameras that capture the street scene based on the local metering algorithm, and the average brightness is obtained.

[0023] The partitioned metering algorithm specifically divides the image into multiple zones, first measures the brightness of each zone, and then performs a comprehensive calculation to determine the metering weight of each zone, obtaining an exposure value that takes all zones into account. The partitioned metering algorithm also has automatic backlight compensation capabilities.

[0024] The automatic white balance is specifically: in different color temperature environments, the ISP algorithm adjusts and offsets the color cast caused by the color temperature, so that the image effect is close to the visual habits of the human eye;

[0025] The local metering algorithm specifically comprises: combining center-weighted metering with spot metering, wherein the metering area is larger than that of the spot metering;

[0026] The brightness is the ratio of the luminous intensity of the light source to the area of ​​the light source visible to the human eye, and is defined as the brightness of the light source unit, that is, the luminous intensity per unit projected area; the luminous intensity is the luminous flux emitted per unit solid angle in a set direction; the luminous flux is the radiant power that can be perceived by the human eye, that is, the product of the radiant energy in a certain band per unit time and the relative visibility of the band.

[0027] In step 5, the process of measuring the light conditions of the virtual scene surrounding the digital twin world and performing secondary compensation according to the light intensity in the real world is as follows:

[0028] When the environment in which the digital twin world is observed is too bright or too dark, the brightness of the projection screen is adjusted to measure the lighting conditions of the digital twin world's environment. Secondary compensation is performed based on the light intensity in the real world, and the color RGB is further adjusted to achieve a lighting effect close to the real world when observing the digital twin world in different environments.

[0029] In step 5, the light conditions include light intensity and light type, the light types include incident light, reflected light, refracted light and diffracted light, and the light intensity measurement methods include infrared photometry, CCD photometry, CMOS photometry and photosensor photometry.

[0030] In step 7, the combination of light intensities measured by multiple cameras is saved as historical data to display the generalized virtual scene, that is, it is reused as an experience value.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. By obtaining the intensity of light in the real scene, the lighting effect of the virtual world is adjusted in real time to achieve a visual effect close to the real world;

[0033] 2. The specific shooting fields of view of multiple cameras are combined to form a set of light intensity images, which are used to subdivide the light settings of the virtual world;

[0034] 3. Set the lighting settings for a static object or a certain type of dynamic object in the virtual world, and perform weighted balance of other ambient lights based on the set object and the actual light intensity;

[0035] 4. The light intensity combination of multiple cameras can be saved as historical data, used for further display of generalized virtual scenes, and reused as experience values;

[0036] 5. Perform secondary compensation for objects that are too bright or too dark in the real world to enhance the visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of the method for setting up lighting for an existing digital twin scene.

[0038] Figure 2 Flow chart of the method of the present invention.

[0039] Figure 3 Schematic diagram of the local photometry algorithm of the present invention.

[0040] Figure 4 Schematic diagram of the partitioned photometry algorithm of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example

[0043] This invention provides a method for setting the light sensitivity of a digital twin scene. By acquiring light-sensing information captured by a camera in the real environment, the ambient brightness of the digital twin scene is dynamically adjusted in real time, making the digital twin environment closer to the real environment. In the virtual scene of the digital twin world, light intensities are pre-set for different times and weather conditions, and modeled based on orientation. The digital twin image in the digital twin world is displayed, and the effects of the surrounding environment on light, including refraction, reflection, and shadows, are displayed.

[0044] In the present invention, the digital twin world is adjusted through the weighted principle of multiple cameras, which is equivalent to the metering principle of multiple photosensitive area arrays (Bayer array, or pixel array) in a camera. Each pixel point in the pixel array of the camera photosensitive device corresponds to the information provided by a camera in the present invention. The method of the present invention is to treat all cameras in an environmental area as a huge single camera photosensitive device, and then use the relevant photosensitive processing technology of a single digital camera, such as the image signal processing algorithm (Image Signal Process, ISP), to stitch and integrate the images collected by all cameras, and then process the original image output by the image sensor to obtain a better scene restoration effect. Among them, the 3A algorithm for processing the original image output by the image sensor includes AWB (automatic white balance), AE (automatic exposure) and AF (autofocus).

[0045] The method comprises the following steps:

[0046] Step 1: Obtain the camera parameters installed in the real world and store them in the digital twin computing unit;

[0047] Step 2: Record real-world data and preset a set of light parameters in the digital twin unit based on the real-world data for presetting the light in the virtual world;

[0048] Step 3: Start multiple cameras distributed in different locations in the real world, obtain the camera environment perception results of the cameras, and send them to the digital twin computing unit;

[0049] Step 4: The digital twin computing unit uses a scene-by-scene photometry method to perform a weighted balance based on the camera's environmental perception results, adjusting the virtual scene in the digital twin world in real time to achieve a lighting effect close to that of the real world.

[0050] Step 5: Measure the lighting conditions of the virtual scene surrounding the digital twin world and perform secondary compensation based on the light intensity in the real world;

[0051] Step 6: Display the digital twin world;

[0052] Step 7: Record the currently set light parameters as historical data.

[0053] The digital twin projection environment is specifically as follows:

[0054] First, a map that is the same as the real scene is constructed in the virtual world. It is generally based on a high-precision map, including various buildings. In the surveying and mapping field, it is called the Building Information Model (BIM) or the City Information Model (CIM), that is, the digital base. On this basis, the lighting effect is configured to achieve a virtual scene that is close to the real scene, that is, the digital twin projection environment.

[0055] By combining the specific shooting fields of view of multiple cameras, a set of image combinations with different light intensities is formed to subdivide the lighting settings of the virtual world. For example, for the same building or car, different cameras will perceive different reflected light from different shooting angles, but they are all reflected light from the building. In this way, the lighting settings for the building are set based on the perception results of multiple cameras. In other words, by observing buildings from different angles in the virtual world, the light and dark changes are more detailed and realistic, just like in the real world.

[0056] In step 4, different digital twin world photometric methods and metering modes are switched through the cloud control platform in different scenarios to adjust the lighting conditions of the digital twin world. The metering modes include:

[0057] 1. Average metering: takes the average brightness of the image as the basis for exposure. If the brightness difference between the main subject and the background is large, it is easy to cause overexposure or underexposure.

[0058] 2. Center Spot Metering: Only measures the brightness of a small central area as the basis for automatic exposure.

[0059] 3. Multi-Spot Metering: Simply put, it's spot metering with a memory device. When shooting, different parts of the subject are positioned at the center of the viewfinder for spot metering. The camera's electronic circuitry memorizes the results and uses the average of these points for exposure. This allows for balanced brightness across the entire frame, offering high accuracy but requiring minimal operation. Multi-Spot Metering is suitable for photographing still subjects like landscapes and portraits, but not for moving subjects.

[0060] 4. Center-weighted metering: This metering mode is a compromise between average metering and spot metering. It gives a higher weight to the brightness of the center portion of the image and a lower weight to the edges. This mode takes into account the brightness of both the subject and the surrounding scenery. Therefore, it has a high metering accuracy for the subject and is particularly suitable for photographing people in landscapes. However, it can easily cause overexposure or underexposure in scenes with uneven brightness or high contrast.

[0061] 5. Partial metering: This is a metering mode that is a compromise between center-weighted metering and spot metering. The metering area is larger than spot metering.

[0062] 6. Multi-Pattern Metering: Also known as multi-format metering, multi-mode metering or area split metering, the main principle is to divide the picture into several areas, first measure the brightness of each area, and then through comprehensive calculation, determine the metering weighted ratio of each area to obtain an exposure value that takes into account each area. The zone metering method has the ability to automatically compensate for backlighting.

[0063] In step 4, the digital twin computing unit performs weighted balancing based on the camera environment perception results based on the scene-by-scene photometry method. Specifically:

[0064] When adjusting the global digital twin world, light metering is performed based on the partition metering algorithm, and the light intensity measured by each camera is weighted by multi-camera comprehensive metering; when adjusting an object in the virtual scene, light metering is performed based on the spot metering algorithm and automatic white balance through the camera closest to the object or the camera that captures the object to obtain the brightness; when adjusting a set street scene, light metering is performed based on the local metering algorithm through a combination of multiple cameras that capture the street scene, and the average brightness is obtained.

[0065] Auto White Balance (AWB) specifically involves adjusting the image quality of the camera using the ISP algorithm in different color temperature environments to offset the color cast caused by color temperature, resulting in images that are closer to the human eye's visual perception. This is because cameras differ from the human eye. For the human eye, perception is subjective, and essentially white objects appear different under different color temperatures and reflected light. However, after correction by the human visual system, they remain white. For cameras, on the other hand, perception is objective, and color inconsistencies under different color temperatures and reflected light result in a different effect than what is seen by the human eye, known as color cast. Therefore, AWB is required for adjustment, similar to correction by the human visual system.

[0066] Automatic white balance algorithms include the grayscale world algorithm, the perfect reflection algorithm, and the dynamic threshold algorithm. The grayscale world algorithm is preferred and is an image processing method in the cloud control platform. In addition, when selecting a specific angle and a specific object for observation in the digital twin world, it is equivalent to using spot metering or local metering. In this case, the cloud control platform switches to spot metering or local metering.

[0067] Luminous flux (Φ): refers to the radiation power that can be perceived by the human eye, that is, the product of the radiation energy of a certain band per unit time and the relative visibility of the band, and the unit is lumen;

[0068] Luminous intensity (light intensity): the luminous flux emitted per unit solid angle in a given direction (the radiant intensity in that direction is (1 / 683) watt / steradian)), measured in candela;

[0069] Brightness (Lv): refers to the ratio of the light intensity of a luminous body to the area of ​​the light source seen by the human eye. It is defined as the brightness of the light source unit, that is, the luminous intensity per unit projected area, and the unit is candela / square meter (cd / m2);

[0070] Sensitivity (ISO): A measure of the film's sensitivity to light.

[0071] In step 7, the light intensity combination measured by multiple cameras is saved as historical data to display the generalized virtual scene and reused as experience values. That is, in the same scene but different weather, for example, when it is darker or cloudier than the historical virtual scene, the corresponding light parameters are automatically expanded and calculated based on the historical data.

[0072] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A digital twin scene photosensitivity setting method, characterized in that: The method comprises the following steps: Step 1: Obtain the perception results of light intensity and color from the camera installed in the real world and feedback the parameters, and store the camera parameters in the digital twin computing unit; Step 2: Record real-world data and preset a set of light parameters in the digital twin unit based on the real-world data for presetting the light in the virtual world; Step 3: Start multiple cameras distributed in different locations in the real world, obtain the camera environment perception results of the cameras, and send them to the digital twin computing unit; Step 4: The digital twin computing unit uses a scene-by-scene photometry method to perform a weighted balance based on the camera's environmental perception results, adjusting the virtual scene in the digital twin world in real time to achieve a lighting effect close to that of the real world. Step 5: Measure the lighting conditions of the virtual scene surrounding the digital twin world and perform secondary compensation based on the light intensity in the real world; Step 6: Display the digital twin world; Step 7: Record the currently set light parameters as historical data; In step 5, the process of measuring the light conditions of the virtual scene surrounding the digital twin world and performing secondary compensation according to the light intensity in the real world is as follows: When the environment in which the digital twin world is observed is too bright or too dark, the brightness of the projection screen is adjusted to measure the lighting conditions of the digital twin world's environment. Secondary compensation is performed based on the light intensity in the real world, and the color RGB is further adjusted to achieve a lighting effect close to the real world when observing the digital twin world in different environments.

2. A digital twin scene photosensitivity setting method according to claim 1, characterized in that: In step 1, the camera parameters include the current ISO, shutter speed, white balance, aperture size, and color temperature.

3. The method for setting the light sensitivity of a digital twin scene according to claim 1, wherein: In step 2, the real-world data includes the current time, current season, current weather, current ambient temperature and historical data, which is used to obtain the historical scene that best matches the current environment through big data, and call the parameters in the real-world data as preset data.

4. The method for setting the light sensitivity of a digital twin scene according to claim 1, wherein: In step 2, the light parameters include the intensity of direct light, the intensity of reflected light, the spectrum of reflected light, the direction of refracted light, diffracted light, the intensity and spectral composition of light.

5. The method for setting the light sensitivity of a digital twin scene according to claim 1, wherein: In step 3, the camera environment perception result is the perception result obtained by the CCD or CMOS sensor in the camera, including the ambient light intensity and the captured image.

6. The method for setting the light sensitivity of a digital twin scene according to claim 1, wherein: In step 4, the digital twin world is adjusted by the principle of weighted balance of multiple cameras. The principle of weighted balance of multiple cameras is equivalent to the metering principle of multiple photosensitive area arrays in a camera. That is, all cameras in an environmental area are regarded as a huge single camera photosensitive device. The relevant photosensitive processing technology of a single digital camera is used to process the original image output by the image sensor to obtain the effect of restoring the scene. Among them, the 3A algorithms for processing the original image output by the image sensor are automatic white balance, automatic exposure and autofocus. The digital twin calculation unit performs weighted balance according to the camera environment perception results based on the scene-based photometry method. Specifically: In the digital twin world, the light of a static object or a type of dynamic object is set, and the light of other environments is weighted and balanced based on the set static object or dynamic object and the light intensity in the real world. In different scenarios, different digital twin world photosensitivity methods and metering modes are switched to adjust the light conditions of the digital twin world: when adjusting the digital twin world globally, metering is performed based on the partition metering algorithm, and the multi-camera comprehensive metering is weighted for the light intensity measured by each camera; when adjusting an object in the virtual scene, metering is performed based on the spot metering algorithm and automatic white balance through the camera closest to the object or the camera that captures the object to obtain the brightness; when adjusting a set street scene, metering is performed based on a combination of multiple cameras that capture the street scene based on the local metering algorithm, and the average brightness is obtained.

7. The method for setting the light sensitivity of a digital twin scene according to claim 6, wherein: The partitioned metering algorithm specifically divides the image into multiple zones, first measures the brightness of each zone, and then performs a comprehensive calculation to determine the metering weight of each zone, obtaining an exposure value that takes all zones into account. The partitioned metering algorithm also has automatic backlight compensation capabilities. The automatic white balance is specifically: in different color temperature environments, the ISP algorithm adjusts and offsets the color cast caused by the color temperature, so that the image effect is close to the visual habits of the human eye; The local metering algorithm specifically comprises: combining center-weighted metering with spot metering, wherein the metering area is larger than that of the spot metering; The brightness is the ratio of the luminous intensity of the luminous body to the area of ​​the light source seen by the human eye, and is defined as the brightness of the light source unit, that is, the luminous intensity per unit projected area; The luminous intensity is the luminous flux emitted per unit solid angle in a set direction; the luminous flux is the radiation power that can be perceived by the human eye, that is, the product of the radiation energy of a certain band per unit time and the relative visibility of the band.

8. The method for setting the light sensitivity of a digital twin scene according to claim 1, wherein: In step 5, the light conditions include light intensity and light type, the light types include incident light, reflected light, refracted light and diffracted light, and the light intensity measurement methods include infrared photometry, CCD photometry, CMOS photometry and photosensor photometry.

9. The method for setting the light sensitivity of a digital twin scene according to claim 1, wherein: In step 7, the combination of light intensities measured by multiple cameras is saved as historical data to display the generalized virtual scene, that is, it is reused as an experience value.

Images