Fragmentation depth map correction system with temporal smoothness

Abstract

An image simulation system includes an image acquisition circuit, memory, and a processor. The image acquisition circuit is used to acquire images. The memory is used to store multiple instructions and a fragmentation depth map. The processor is connected to the image acquisition circuit and accesses multiple instructions to perform the following operations: generating a quaternion map and classification parameters based on the image, wherein the classification parameters are related to the background and foreground in the image; adjusting the quaternion map based on the classification parameters, and generating a guide mask based on the adjusted quaternion map, wherein the guide mask is used to indicate the background and foreground in the image; and generating a special effects image based on the image, the guide mask, and the fragmentation depth map, wherein the special effects image is used to simulate weather corresponding to the fragmentation depth map in the image. This allows for the effective simulation of near-realistic weather effects on the image.

Description

Technical Field

[0001] This disclosure relates to an image processing technique, and more particularly to a fragmented depth map correction system with temporal smoothness. Background Technology

[0002] In recent years, thanks to the rapid development of deep learning in the field of computer vision, many complex problems that were previously difficult to describe with equations can now be solved effectively using convolutional neural networks. However, when it comes to generating weather effects from street view images, current technologies encounter various problems when applied to street view images, resulting in less than ideal synthesized effects. Therefore, how to simulate near-realistic weather effects on images is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0003] This disclosure presents an image simulation system, including an image acquisition circuit, memory, and a processor. The image acquisition circuit is used to acquire images. The memory is used to store multiple instructions and a fragmentation depth map. The processor is connected to the image acquisition circuit and accesses multiple instructions to perform the following operations: generating a quaternion map and classification parameters based on the image, wherein the classification parameters are related to the background and foreground in the image; adjusting the quaternion map based on the classification parameters, and generating a guide mask based on the adjusted quaternion map, wherein the guide mask is used to indicate the background and foreground in the image; and generating a special effects image based on the image, the guide mask, and the fragmentation depth map, wherein the special effects image is used to simulate weather corresponding to the fragmentation depth map in the image.

[0004] In one embodiment, the operation of generating a four-connected graph and classification parameters based on an image includes: performing semantic segmentation processing on the image to generate a semantically segmented image, and generating multiple classification labels corresponding to multiple pixels of the image based on the semantically segmented image; and generating histograms corresponding to multiple numerical categories based on the RGB values ​​of the multiple pixels, and generating classification parameters based on the multiple numerical categories and the multiple classification labels.

[0005] In one embodiment, the multiple classification labels include foreground labels, background labels, and undetermined labels, and the operation of generating classification parameters based on multiple numerical categories and multiple classification labels includes: selecting the minimum number from the number of foreground labels and the number of undetermined labels corresponding to each of the multiple numerical categories; and adding the minimum numbers of each of the multiple numerical categories to generate a minimum sum value, and generating classification parameters based on the minimum sum value and classification cost parameters.

[0006] In one embodiment, the operation of adjusting the quadconnected graph based on classification parameters includes: generating smoothing parameters based on the RGB values ​​of each of the plurality of pixels and the RGB values ​​of the surrounding pixels of each of the plurality of pixels; and adjusting the quadconnected graph based on the smoothing parameters of each of the plurality of pixels and classification parameters.

[0007] In one embodiment, the operation of adjusting the quadconnected graph according to classification parameters includes: converting the image from the RGB domain to the HSV domain to generate the HSV value of the image; performing distance calculation based on the HSV value and shadow value range of the image to generate the shadow probability of multiple pixels of the image; adjusting the classification parameters and smoothing parameters based on the shadow probability of multiple pixels, and adjusting multiple classification labels and the quadconnected graph based on the adjusted smoothing parameters and the adjusted classification parameters.

[0008] In one embodiment, the operation of adjusting multiple classification labels and a quadconnected graph based on adjusted smoothing parameters and adjusted classification parameters includes: performing a maximum flow minimum segmentation operation on the quadconnected graph using the adjusted smoothing parameters and adjusted classification parameters to adjust multiple classification labels corresponding to multiple pixels of the image, and generating a guide mask based on the adjusted classification labels and the quadconnected graph.

[0009] In one embodiment, the operation of generating special effects images based on images, guide masks, and broken depth maps includes: generating a depth map based on images, and using guide masks and depth maps to generate weather on images corresponding to the broken depth maps.

[0010] In one embodiment, the image capturing circuit is further configured to capture another image in the previous frame of the image. The operation of generating a special effects image based on the image, the guide mask, and the broken depth map includes: generating a depth map based on the image, and generating a point cloud map based on the depth map; identifying a virtual ground position corresponding to the horizon in the image from the point cloud map, and adjusting the guide mask based on the virtual ground position; generating timing parameters and edge parameters based on the image, the other image, the adjusted guide mask, and the depth map, wherein the timing parameters are used to resolve depth discontinuities between the image and the other image, and the edge parameters are used to enhance the edge enhancement of the depth of the image and the other image; and adjusting the depth map based on the timing parameters and the edge parameters, and generating a special effects image based on the adjusted depth map and the broken depth map.

[0011] In one embodiment, the operation of adjusting the depth map based on temporal parameters and edge parameters includes: performing conjugate gradient descent processing based on temporal parameters and edge parameters to adjust the depth map.

[0012] In one embodiment, the operation of generating special effects images based on the adjusted depth map and the fragmented depth map includes: generating a sparse point cloud map based on the fragmented depth map, and compositing the sparse point cloud map with the image based on the adjusted depth map to generate special effects images.

[0013] Based on the above, the present disclosure embodiment can generate a guiding mask according to the classification parameters of the image and the four-connected map, and use the guiding mask to optimize the depth map generated by depth prediction to clearly distinguish the foreground and background in the image, thereby preventing the depth of the part of the image close to the ground from being confused with the depth of the ground. Attached Figure Description

[0014] Figure 1 This is a block diagram of the image simulation system disclosed herein.

[0015] Figure 2 This is a flowchart of the image simulation method disclosed herein.

[0016] Figure 3 This is a schematic diagram of an image simulation method according to some embodiments of the present disclosure.

[0017] Figure 4 This is a schematic diagram of auxiliary nodes and classification labels based on some embodiments of this disclosure.

[0018] Figure 5 This is a schematic diagram of a probability graph corresponding to the probability of shadows, based on some embodiments of this disclosure.

[0019] [Explanation of Key Component Symbols]

[0020] 100: Image simulation system; 110: Image acquisition circuit

[0021] 120: Memory 130: Processor

[0022] S210~S230: Step PPM: Preprocessing Model

[0023] MGM: Masking Generation Model; TSSM: Temporal Smoothing Model

[0024] IMG: Image; IMG': Another Image

[0025] SI: Semantic segmentation image; DI: Depth map

[0026] GDI: Virtual Ground Map; GM: Guide Mask

[0027] SYI: Special effects image; k: the kth auxiliary node Detailed Implementation

[0028] Reference Figure 1 , Figure 1This is a block diagram of the image simulation system 100 disclosed herein. In one embodiment, the image simulation system 100 includes an image acquisition circuit 110, a memory 120, and a processor 130. The image acquisition circuit 110 is used to acquire images. The memory 120 is used to store multiple instructions and a fragmentation depth map. The processor 120 is connected to the image acquisition circuit 110 and the memory 120, and is used to access these instructions.

[0029] In some embodiments, the image simulation system 100 may be established by a computer, server, or processing center. In some embodiments, the image capturing circuit 110 may be a camera for capturing images or a camera capable of continuous image capture. In some embodiments, the processor 120 may be implemented by a processing unit, a central processing unit, or a computing unit. In some embodiments, the breakage depth map may be a depth map corresponding to specific weather conditions that may occur on the road. For example, a depth map of rain, fog, snow, or hail on the road.

[0030] In some embodiments, the image simulation system 100 is not limited to including the image capturing circuit 110, the memory 120, and the processor 130. The image simulation system 100 may further include other components required for operation and application. For example, the image simulation system 100 may further include an output interface (e.g., a display panel for displaying information), an input interface (e.g., a touch panel, keyboard, microphone, scanner, or flash memory reader), and communication circuits (e.g., WiFi communication model, Bluetooth communication model, wireless telecommunications network communication model, etc.).

[0031] Reference Figure 2 , Figure 2 This is a flowchart of the image simulation method disclosed herein. Figure 2 The method of the illustrated embodiment is applicable to Figure 1 The image simulation system 100 is applicable, but not limited to it. For convenience and clarity, the following also refers to... Figure 1 as well as Figure 2 The explanation is based on the actuation relationships between the components in the image simulation system 100. Figure 2 The detailed steps of the image simulation method shown are as follows.

[0032] In one embodiment, the image simulation method includes steps S210 to S230, all of which can be executed by the processor 130. First, in step S210, an image is captured, and a quadconnected graph and classification parameters are generated based on the image, wherein the classification parameters are related to the background and foreground in the image.

[0033] In some embodiments, semantic segmentation processing can be performed on the image to generate a semantically segmented image, and multiple classification labels corresponding to multiple pixels of the image can be generated based on the semantically segmented image. Then, histograms corresponding to multiple numerical categories can be generated based on the RGB values ​​of the multiple pixels, and classification parameters can be generated based on the multiple numerical categories and the multiple classification labels.

[0034] In some embodiments, image segmentation can be used to generate weights among multiple pixels in the image. Then, the multiple pixels can be treated as nodes in a quadconnected graph, and these weights can be mapped to the connections between the nodes in the quadconnected graph, wherein the nodes in the quadconnected graph correspond sequentially to pixels in the image (e.g., the nodes in the first column of the quadconnected graph correspond to the pixels in the first column of the image).

[0035] In some embodiments, multiple numerical categories can be used as multiple auxiliary nodes, and the multiple auxiliary nodes can be connected to the corresponding nodes in the four-connected graph to generate an adjusted four-connected graph.

[0036] In some embodiments, the multiple classification labels include foreground labels, background labels, and undetermined labels. In some embodiments, the minimum number of foreground labels and undetermined labels corresponding to each of the multiple numerical categories can be selected. Then, the minimum numbers for each of the multiple numerical categories can be added together to generate a minimum sum value, and a classification parameter can be generated based on the minimum sum value and the classification cost parameter.

[0037] Furthermore, in step S220, the quadconnected graph is adjusted according to the classification parameters, and a guide mask is generated based on the adjusted quadconnected graph, wherein the guide mask is used to indicate the background and foreground in the image.

[0038] In some embodiments, a smoothing parameter can be generated based on the RGB values ​​of each of the plurality of pixels and the RGB values ​​of the surrounding pixels of each of the plurality of pixels. Then, the quadconnected graph can be adjusted based on the smoothing parameter of each of the plurality of pixels and the classification parameter.

[0039] In some embodiments, the image can be converted from the RGB domain to the HSV domain to generate HSV values ​​for the image. Then, distance calculations can be performed based on the HSV values ​​and shadow value ranges of the image to generate shadow probabilities for multiple pixels in the image. Next, classification parameters and smoothing parameters can be adjusted based on the shadow probabilities of the multiple pixels, and multiple classification labels and a quadconnected graph can be adjusted based on the adjusted smoothing parameters and adjusted classification parameters.

[0040] In some embodiments, the shadow value range may include the value range of the hue channel, the value range of the saturation channel, and the value range of the lightness channel, wherein these value ranges may be the average value of shadows obtained from past experience, preset values ​​of shadows given manually, or random values.

[0041] In some embodiments, a maximum flow minimum segmentation operation can be performed on the quadconnected graph using the adjusted smoothing parameters and the adjusted classification parameters to adjust the multiple classification labels corresponding to multiple pixels of the image, and a guide mask can be generated based on the adjusted multiple classification labels and the quadconnected graph.

[0042] Furthermore, in step S230, special effects images are generated based on the images, guide masks, and fragmentation depth maps, wherein the special effects images are used to simulate the weather corresponding to the climate images in the images.

[0043] In some embodiments, a depth map can be generated based on the image, and weather corresponding to the broken depth map can be generated on the image using a guide mask and the depth map.

[0044] In some embodiments, an image capturing circuit 110 can capture another image from the previous frame. In some embodiments, a depth map can be generated based on the image, and a point cloud map can be generated based on the depth map. Then, a virtual ground position corresponding to the horizon in the image can be identified from the point cloud map, and a guide mask can be adjusted based on the virtual ground position. Next, timing parameters and edge parameters are generated based on the image, the other image, the adjusted guide mask, and the depth map, wherein the timing parameters are used to resolve depth discontinuities between the image and the other image, and the edge parameters are used to enhance the edge enhancement of the depth of the image and the other image. Then, the depth map can be adjusted based on the timing parameters and the edge parameters, and a special effects image can be generated based on the adjusted depth map and the fragmented depth map.

[0045] In some embodiments, conjugate gradient descent can be performed based on temporal parameters and edge parameters to adjust the depth map.

[0046] In some embodiments, a sparse point cloud map can be generated based on a fragmented depth map, and the sparse point cloud map can be combined with an image based on an adjusted depth map to produce a special effects image.

[0047] The above steps allow for adjustments to classification parameters to prevent the depth of areas near the ground in the background from being confused with the ground depth, thus enabling the generation of a guiding mask using the adjusted classification parameters. Furthermore, smoothing parameters can be adjusted to prevent noise at object edges in the image. Additionally, temporal and edge parameters can be adjusted to resolve depth discontinuities between the image and another image from the previous frame, and to enhance the edge enhancement of both the current and previous images.

[0048] The following example uses a real-world model with 120 memory units to further illustrate the above process. (See also...) Figure 3 , Figure 3 This is a schematic diagram illustrating an image simulation method according to some embodiments of the present disclosure. In one embodiment, memory 120 may further include a preprocessing model PPM, a mask generation model MGM, and a temporal smoothing model TSSM. Processor 130 can execute these models to perform the above-described... Figure 2 The steps in the process.

[0049] First, the preprocessing model PPM can preprocess the image IMG generated by the image capturing circuit 110, wherein the image IMG is an RGB image, and the preprocessing includes semantic segmentation processing (e.g., by using UpperNet), depth prediction processing (e.g., by using MegaDepth), and point cloud projection processing (e.g., by using PCL).

[0050] In detail, the preprocessing model PPM performs semantic segmentation on the image IMG to generate a semantically segmented image SI, where identical objects in the semantically segmented image SI have the same weight. Next, the preprocessing model PPM performs depth prediction on the image IMG to generate a depth map DI. Then, the preprocessing model PPM performs point cloud projection on the depth map DI to generate a virtual terrain map GDI, where virtual ground locations in the virtual terrain map GDI have relatively low grayscale values. Thus, the preprocessing model PPM can input the semantically segmented image SI and the depth map DI into the masking generation model MGM, and input the depth map DI and the virtual terrain map GDI into the temporal smoothing model TSSM.

[0051] Furthermore, the masking generation model (MGM) can identify ground-related pixels in the semantically segmented image SI as foreground and non-ground-related pixels as background, and further identify pixels at the boundary between foreground and background as undefined regions. Next, the MGM can generate a four-connected graph of pixels in the image IMG based on the semantically segmented image SI, where each pixel in the image IMG corresponds to a node in the four-connected graph, and the connections between pixels belonging to the same object in the four-connected graph have higher correlation values.

[0052] Furthermore, the Masking Generation Model (MGM) can generate histograms for 16 categories of the red, green, and blue channels based on the image IMG. The histograms are 16x16x16 in size, and the numerical categories within the histograms are also 16x16x16. Next, the MGM can connect the 16x16x16 auxiliary nodes corresponding to these numerical categories to the corresponding cells in the four-connected graph. For example, the numerical range of one category in the red channel is 0–30, and the node corresponding to this range in the four-connected graph can be connected to the auxiliary node for this numerical category.

[0053] Furthermore, the mask generation model MGM can use the following formula (1) to calculate the sum of the classification costs (i.e., classification parameters) of all auxiliary nodes.

[0054]

[0055] Where E1 is the classification parameter, βk is the pre-set classification cost parameter of the kth auxiliary node, S is the number of pixels of the nodes with foreground labels connected to each auxiliary node, S' is the number of pixels of the nodes with background labels connected to each auxiliary node, min(,) is the function to extract the minimum value, and N is the number of auxiliary nodes (e.g., 16x16x16 as mentioned above).

[0056] For example, consider the kth auxiliary node. Also refer to... Figure 4 , Figure 4 This is a schematic diagram of auxiliary nodes and classification labels based on some embodiments of this disclosure. Figure 4 In the given information, pixels from one connected node have a background label, pixels from two connected nodes have an undetermined label, and pixels from four connected nodes have a foreground label. Therefore, the minimum of 1 and 4 can be multiplied by β to calculate the classification cost β of the k-th auxiliary node. Based on this, if all nodes in the k-th auxiliary node have at most one label (foreground or background), the classification cost can be minimized (i.e., reduced to 0).

[0057] Refer to the return Figure 3 The mask generation model MGM can calculate the smoothing term of each pixel using the following formula (2), and use the sum of the smoothing terms as the smoothing parameter.

[0058]

[0059] Where E2 is the smoothing term for one pixel, Vp is the RGB value of one pixel, Vq is the RGB value of one of the surrounding pixels, and σ is the statistical variance of all pixels. It is worth noting that, as shown in formula (2), four smoothing terms can be calculated from one pixel, and these terms can be summed to form a smoothing parameter. These smoothing terms exhibit a Gaussian distribution; the closer the colors of two adjacent pixels are, the smaller the smoothing term, and vice versa. This smoothing parameter can greatly reduce the possibility of incorrect pixel classification labels.

[0060] Furthermore, the Masking Model (MGM) can preset the range of shadow values ​​in the HSV domain of an image. For example, the hue channel can be set to 50–250, the saturation channel to 0–70, and the lightness channel to 25–100.

[0061] Furthermore, the mask generation model MGM can convert the image IMG from the RGB domain to the HSV domain to generate the image's HSV values. Next, the mask generation model MGM can perform distance calculations based on the image's HSV values ​​and shadow value ranges to generate the shadow probabilities of multiple pixels in the image IMG, and can generate a probability map based on these shadow probabilities.

[0062] For example, refer to Figure 5 , Figure 5 This is a schematic diagram of a probability graph corresponding to the probability of shadows, based on some embodiments of this disclosure. For example... Figure 5 As shown, different shadow probabilities can be mapped to different grayscale values ​​based on the shadow probabilities of the aforementioned pixels to generate a probability map, where areas closer to white are more likely to be shadows.

[0063] Refer to the return Figure 3 The mask generation model MGM can adjust the value of βk in formula (1) according to the above shadow probability as shown in the following formula (3).

[0064] βk=1-ShadowProbk…(3)

[0065] Where ShadowProbk represents the shadow probability of all pixels corresponding to the k-th auxiliary node. In other words, areas with higher shadow probabilities are less affected by the classification parameters and are only controlled by the aforementioned smoothing term.

[0066] Furthermore, the mask generation model MGM can generate shadow parameters based on the shadow probability of the above pixels as shown in the following formula (4).

[0067]

[0068] Here, E3 is the shadow parameter, ShadowProbp is the shadow probability of a pixel, the blurred pixel is a pixel with an ambiguous label and surrounding pixels with background labels, and ShadowProbq is the shadow probability of the pixels surrounding the pixel. It's worth noting that for other pixels, there are four pixels surrounding the pixel, thus generating four shadow parameters. Based on this, the shadow parameter E3 can reduce the cost of ground boundaries in the original semantic segmentation, ensuring that the final boundary falls on the edge of the semantic segmentation rather than the edge of the shadow.

[0069] Furthermore, the mask generation model MGM can adjust the above smoothing parameters according to the shadow parameters of each pixel and a preset adjustment parameter, as shown in the following formula (5).

[0070] E2′=E2+w×E3…(5)

[0071] Here, E2' is the adjusted smoothing parameter, and w is the adjustment parameter. By adjusting the above parameter w, the relationship between shadows and the colors of the image IMG can be controlled.

[0072] Furthermore, the mask generation model MGM performs maximum flow minimum segmentation operations on E1, the adjusted smoothing parameters E2' of each pixel, multiple auxiliary nodes, and the correlation values ​​in the quadconnected graph to generate adjusted classification labels and an adjusted quadconnected graph. Next, the mask generation model MGM generates a binary guided mask GM based on the adjusted classification labels and the adjusted quadconnected graph. The values ​​in the binary guided mask GM corresponding to the background label can be set to 0, and the values ​​corresponding to the foreground and background labels can be set to 1. In this way, the mask generation model MGM can input the binary guided mask GM into the temporal smoothing model TSSM.

[0073] Furthermore, the Temporal Smoothing Model (TSSM) can pre-store the image IMG and another image IMG' from the previous frame. The TSSM can identify the virtual ground position corresponding to the horizon in the image IMG from the point cloud map, adjust the binary guide mask GM according to the virtual ground position, and then use the adjusted binary guide mask GM to filter the depth map DI to generate a filtered depth map DI.

[0074] Furthermore, the temporal smoothing model TSSM can generate photometric parameters based on another image IMG' in the previous frame of image IMG, as shown in the following formula (6).

[0075] min∫w(x)×‖O-warp(O′)‖ 2 dx…(6)

[0076] Where x is the position of each pixel in the image IMG, O is the expected output depth value of each pixel at position x, warp(,) is the displacement of each pixel between the previous and next frames calculated by the optical flow mapping function (DIS flow) on the original unprocessed film (i.e., warp(0') is the displacement of each pixel between the expected output depth value 0 of the pixel at position x and the expected output depth value 0' of the previous frame of x), and w(x) is as shown in the following formula (7).

[0077] w(x) = λ × e -||V-warp(v′)|| …(7)

[0078] Where λ is a preset parameter, and V is the RGB value of the pixel at position x in the image IMG. The greater the color difference between the pixel at position x and the previous frame, the more likely it is to be at the boundary of an object, and the less it needs to be smoothed.

[0079] Furthermore, as the continuous playback time of a series of images increases, the information projected from the previous frame will continuously accumulate, causing the overall depth to approach a certain value. Therefore, the Temporal Smoothing Model (TSSM) can adjust the photometric parameters to generate temporal parameters as shown in the following formula (8).

[0080] E3=min∫w(x)×||O-warp(O′)|| 2 +s×||OP|| 2 dx…(8)

[0081] Where E3 is the time-series parameter, P is the depth value corresponding to position x in the depth map DI, and s is 0.1. This will ensure that the difference between the expected depth value 0 and the depth value P in the depth map DI is not too large, and the value of s will not cause the originally expected value to deviate too much.

[0082] Furthermore, the temporal smoothing model TSSM can perform gradient calculations on the depth value P corresponding to position x in the depth map Dl and the depth values ​​of surrounding pixels to generate edge parameters as shown in the following formula (9).

[0083]

[0084] in Let x be the gradient value of the expected output depth value for each pixel at position x. Let be the gradient value of the depth value corresponding to position x in the depth map DI, and let e be as shown in the following formula (10).

[0085]

[0086] As can be seen from formula (10), when there is a boundary around the pixel, e is 0, which means that the current value is completely unrelated to the boundary, and otherwise it is -1.

[0087] Furthermore, the Temporal Smoothing Model (TSSM) can sum the temporal parameter E3 and the marginal parameter E4. Since all terms after summation are squared, it can be viewed as a matrix least squares problem. Therefore, the TSSM model can use conjugate gradient descent to iteratively derive the expected depth values ​​and adjust the depth values ​​in the depth map DI to these expected output depth values.

[0088] Finally, the temporal smoothing model TSSM can generate sparse point cloud maps based on pre-stored fragmented depth maps (e.g., using Structure from Motion (SfM)), and synthesize the sparse point cloud maps with the image IMG based on the adjusted depth map to produce the special effects image SYI.

[0089] In summary, the image simulation system of this embodiment can generate a guiding mask based on the image's classification parameters and quadconnectivity graph. This guiding mask is then used to optimize the depth map generated by depth prediction, clearly distinguishing the foreground and background in the image, thereby preventing the depth of areas near the ground from being confused with the ground's depth. Furthermore, temporal and edge parameters can be used to smooth the depth and enhance its edges temporally, preventing flickering caused by discontinuity in temporal depth. This effectively simulates near-realistic weather effects on images.

[0090] While specific embodiments of the present disclosure have been disclosed in relation to the above embodiments, these embodiments are not intended to limit the present disclosure. Various alternatives and modifications can be made by those skilled in the art in accordance with the present disclosure without departing from the principles and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure is determined by the appended claims.

Claims

1. An image simulation system, characterized in that, include: Image capturing circuit, used to capture images; Memory is used to store multiple instructions and a fracture depth map; as well as The processor, connected to the image capturing circuit, accesses the instructions to perform the following operations: A four-connected graph and classification parameters are generated based on the image, wherein the classification parameters are related to the background and foreground in the image; The quadron graph is adjusted based on the classification parameters, and a guide mask is generated based on the adjusted quadron graph, wherein the guide mask is used to indicate the background and the foreground in the image; and Based on the image, the guide mask, and the fracture depth map, special effects images are generated, wherein the special effects images are used to simulate the weather corresponding to the fracture depth map in the image; The image capturing circuit is further used to capture another image from the previous frame. The operation of generating the special effects image based on the image, the guide mask, and the fragmentation depth map includes: A depth map is generated based on the image, and a point cloud map is generated based on the depth map; Identify the virtual ground position corresponding to the horizon in the image from the point cloud map, and adjust the guide mask according to the virtual ground position; Based on the image, the other image, the adjusted guide mask, and the depth map, temporal parameters and edge parameters are generated. The temporal parameters are used to resolve depth discontinuities between the image and the other image, and the edge parameters are used to enhance the edge enhancement of the depth of the image and the other image. The depth map is adjusted based on the timing parameters and the edge parameters, and the special effects image is generated based on the adjusted depth map and the fragmentation depth map.

2. The image simulation system according to claim 1, characterized in that, The operations for generating the four-connected graph and the classification parameters based on the image include: Semantic segmentation processing is performed on the image to generate a semantically segmented image, and multiple classification labels corresponding to multiple pixels of the image are generated based on the semantically segmented image; and A histogram corresponding to multiple numerical categories is generated based on the RGB values ​​of the pixels, and the classification parameter is generated based on the numerical categories and the classification label.

3. The image simulation system according to claim 2, characterized in that, The classification labels include foreground labels, background labels, and undetermined labels, wherein the operation of generating the classification parameter based on the numerical category and the classification labels includes: Select the minimum number from the number of foreground labels corresponding to each of the numerical categories and the number of undetermined labels; and The minimum quantity of each of the numerical categories is added together to generate a minimum sum value, and the classification parameter is generated based on the minimum sum value and the classification cost parameter.

4. The image simulation system according to claim 2, characterized in that, The operations for adjusting the four-connected graph based on the classification parameters include: A smoothing parameter is generated based on the RGB values ​​of each pixel and the RGB values ​​of the surrounding pixels of that pixel; and The quadconnected graph is adjusted based on the smoothing parameter and the classification parameter of each of the pixels.

5. The image simulation system according to claim 4, characterized in that, The operations for adjusting the four-connected graph based on the classification parameters include: Convert the image from the RGB domain to the HSV domain to generate the image's HSV values; Distance calculations are performed based on the HSV values ​​and shadow value range of the image to generate the shadow probability of the pixels in the image; and The classification parameter and the smoothing parameter are adjusted based on the shadow probability of the pixel, and the classification label and the four-connected graph are adjusted based on the adjusted smoothing parameter and the adjusted classification parameter.

6. The image simulation system according to claim 5, characterized in that, The operations of adjusting the classification labels and the four-connected graph based on the adjusted smoothing parameter and the adjusted classification parameter include: The adjusted smoothing parameter and the adjusted classification parameter are used to perform a maximum flow minimum segmentation operation on the four-connected graph to adjust the classification labels corresponding to the pixels of the image, and the guide mask is generated based on the adjusted classification labels and the four-connected graph.

7. The image simulation system according to claim 1, characterized in that, The operation of adjusting the depth map based on the time series parameter and the edge parameter includes: The depth map is adjusted by performing conjugate gradient descent based on the time series parameters and the edge parameters.

8. The image simulation system according to claim 1, characterized in that, The process of generating the special effects image based on the adjusted depth map and the fragmentation depth map includes: A sparse point cloud map is generated based on the fractured depth map, and the sparse point cloud map is combined with the image based on the adjusted depth map to produce the special effects image.

Images