Image correction method, device and system and storage medium

Abstract

The invention provides an image correction method, device and system and a storage medium, and belongs to the technical field of image processing and naked eye three-dimensional display. The image correction method comprises the steps of obtaining a to-be-corrected image, wherein the to-be-corrected image comprises depth information of each pixel point; performing initial correction on the to-be-corrected image by using the target correction model to obtain a first corrected image; comparing the first corrected image with the verification image, converting the first corrected image into a stereoscopic image under the condition that the first similarity between the first corrected image and the verification image meets a preset condition, and sending the stereoscopic image to a display for stereoscopic display; wherein the verification image is an image obtained by using a to-be-corrected image as an input sample and estimating by using a standard depth estimation large model; and in response to a correction operation of a user for the first corrected image, performing supplementary correction on the first corrected image to obtain a second corrected image, converting the second corrected image into a stereoscopic image, and sending the stereoscopic image to a display for stereoscopic display.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing and naked-eye stereoscopic display technology, and specifically relates to an image correction method, apparatus, system and storage medium. Background Technology

[0002] With the rapid development of 3D stereoscopic display technology, related products have been applied in fields such as medicine, education, and advertising. In particular, glasses-free 3D technology allows users to view 3D displays without the need for external devices such as 3D glasses or helmets. Its realistic depth and stereoscopic effect greatly enhance the visual impact and immersion of the viewing experience, making it the best display product for product promotion, public relations, and video playback.

[0003] Currently, the number of 3D videos is still very limited. Mainstream 3D video production falls into two categories: active and passive. Active 3D primarily utilizes shooting techniques, placing high demands on binocular cameras. It requires consistent lens, aperture, and color temperature, and synchronized signals from both channels, making stereoscopic shooting equipment extremely expensive. Passive 3D, on the other hand, is achieved through post-production 2D-to-3D conversion technology. This involves using computer vision and computer graphics techniques to estimate the binocular 3D video from a single-lens 2D video. This allows for two video inputs to the display device, resulting in a good 3D visual experience. Summary of the Invention

[0004] This disclosure aims to at least address one of the technical problems existing in the prior art, and to provide an image correction method, apparatus, system, and storage medium.

[0005] Firstly, the technical solution adopted to solve the technical problem of this disclosure is an image correction method, including:

[0006] Obtain the image to be corrected, which includes depth information of each pixel;

[0007] Using the target correction model, the image to be corrected is initially corrected to obtain the first corrected image;

[0008] The first corrected image and the verification image are compared. If the first similarity between the first corrected image and the verification image meets a preset condition, the first corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display. The verification image is an image estimated using the image to be corrected as an input sample and a standard depth estimation large model.

[0009] In response to the user's correction operation on the first correction image, the first correction image is supplemented and corrected to obtain a second correction image, and the second correction image is converted into a stereoscopic image and sent to the display for stereoscopic display.

[0010] In some embodiments, comparing the first corrected image and the verification image includes:

[0011] Based on the first corrected image and the verification image, a first loss function of the target correction model is determined. If the loss value of the first loss function is less than or equal to a preset threshold, the first similarity is determined to satisfy the preset condition. If the loss value corresponding to the first loss function is greater than the preset threshold, the first similarity is determined not to satisfy the preset condition.

[0012] In some embodiments, the image correction method further includes:

[0013] If the first similarity does not meet the preset condition, the target correction model is trained in reverse using the first loss function, with the same image to be corrected as the iterative sample, until the first similarity between the output of the target correction model and the verification image meets the preset condition.

[0014] In some embodiments, determining the first loss function of the target correction model based on the first corrected image and the verification image includes:

[0015] Compare the pixel information of corresponding pixels between the first corrected image and the verification image to determine the pixel loss between the first corrected image and the verification image;

[0016] Using an edge detection algorithm, the edge contour information of the first corrected image and the verification image are extracted respectively to obtain the edge loss between the first corrected image and the verification image;

[0017] The first loss function is determined based on the pixel loss and edge loss.

[0018] In some embodiments, the initial correction of the image to be corrected using a target correction model to obtain a first corrected image includes:

[0019] Using the target correction model, the error locations in the image to be corrected are identified and marked to obtain the marked location information;

[0020] A visual layer box is overlaid on the position indicated by the marked position information in the image to be corrected to synthesize a multi-layer corrected image, and a stereoscopic image based on the multi-layer corrected image is displayed;

[0021] In response to the correct depth information selected by the user for the visualization layer box, the depth information of the incorrect position in the image to be corrected is corrected based on the marker position information and the correct depth information to obtain the first corrected image.

[0022] In some embodiments, the step of supplementing the first correction image to obtain a second correction image in response to a user's correction operation on the first correction image includes:

[0023] In response to a user selecting a multimedia mode, the received multimedia information from the user is parsed, and the user's facial features and / or voice features are identified.

[0024] Based on the facial features and / or the voice features, the user's first correction information is determined;

[0025] Based on the first correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

[0026] In some embodiments, the step of supplementing the first correction image to obtain a second correction image in response to a user's correction operation on the first correction image includes:

[0027] In response to the user selecting a gesture mode, the user's gesture information is identified in response to the user's gesture operation on the first correction image;

[0028] Based on the gesture information, the user's second correction information is determined;

[0029] Based on the second correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

[0030] In some embodiments, the step of supplementing the first correction image to obtain a second correction image in response to a user's correction operation on the first correction image includes:

[0031] In response to the user selecting an editing mode, the third correction information is determined based on the received editing operation by the user on the first correction diagram;

[0032] Based on the third correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

[0033] In some embodiments, the step of supplementing the first correction image to obtain a second correction image in response to a user's correction operation on the first correction image includes:

[0034] In response to a user selecting a multimedia mode, the received multimedia information of the user is parsed, and the user's facial features and / or voice features are identified; based on the facial features and / or the voice features, the user's first correction information is determined;

[0035] In response to the user selecting a gesture mode, and in response to the user's gesture operation on the first correction image, the user's gesture information is identified; based on the gesture information, the user's second correction information is determined.

[0036] In response to the user selecting an editing mode, the third correction information is determined based on the received editing operation by the user on the first correction diagram;

[0037] Based on at least two of the first correction information, the second correction information, and the third correction information, the depth of the region to be corrected in the first correction image is adjusted to obtain the second correction image.

[0038] In some embodiments, before performing supplementary correction on the first correction image in response to a user's correction operation on the first correction image to obtain a second correction image, the method further includes:

[0039] In response to a user-initiated assistance request, at least one of the first correction information, second correction information, and third correction information stored in the memory during the historical phase is invoked to generate auxiliary correction information and display it in two dimensions.

[0040] In some embodiments, the image correction method further includes:

[0041] If there are multiple other viewpoint images in the image to be corrected, multiple viewpoint images are acquired, and the disparity information between each viewpoint image and the second corrected image is calculated.

[0042] Based on the disparity information corresponding to each viewpoint image, the position and depth information of the pixels in each viewpoint image, and the target correction information of the second correction image, fourth correction information corresponding to each viewpoint image is determined; the target correction information includes at least the correction information adjusted by the user for the image to be corrected, and the correction information adjusted using the target correction model.

[0043] Based on the fourth correction information, the corresponding viewpoint map is processed to obtain a corrected viewpoint map.

[0044] In some embodiments, the image correction method further includes:

[0045] In response to the user's correction operation on the first correction image, auxiliary training information is generated for the standard depth estimation large model;

[0046] The model parameters of the standard deep estimation large model are adjusted based on the auxiliary training information.

[0047] The first corrected image is used as a training sample for the standard depth estimation model, and the standard depth estimation model is continuously trained iteratively until it converges.

[0048] In some embodiments, the image correction method further includes:

[0049] The first corrected image is used as the training sample of the target correction model, and the target correction model is continuously iterated and trained until the second similarity between the output of the target correction model and the verification result of the standard depth estimation large model after training satisfies the preset condition.

[0050] In some embodiments, the image correction method further includes:

[0051] Obtain the multimedia file and determine its type;

[0052] If the multimedia file is a video, determine whether the resolution of the multimedia file exceeds a first preset resolution;

[0053] If the resolution exceeds the first preset resolution, the multimedia file is sent to a cloud server. The cloud server is equipped with the ffmpeg framework to decode the multimedia file and cut the decoded multimedia file into multiple consecutive image frames for display.

[0054] In response to a user selecting one of the viewpoint maps from a plurality of viewpoint maps in an image frame, the user-selected viewpoint map is converted as the original image into the image to be corrected having depth information.

[0055] In some embodiments, the image correction method further includes:

[0056] If the resolution of the multimedia file does not exceed the first preset resolution, the ffmpeg framework is used to decode the multimedia file to obtain multiple consecutive image frames, which are then displayed.

[0057] In response to a user selecting a viewpoint from a plurality of viewpoints in an image frame, the user-selected viewpoint is converted as the original image into the image to be corrected having depth information.

[0058] In some embodiments, the image correction method further includes:

[0059] If the multimedia file is an image, the multimedia file is converted into an image to be corrected with depth information.

[0060] In some embodiments, the image correction method further includes:

[0061] If the resolution of the multimedia file does not exceed the first preset resolution, the multimedia file is sent to the graphics card; the graphics card is equipped with the ffmpeg framework, so as to use the ffmpeg framework to decode the multimedia file, and cut the decoded multimedia file into multiple consecutive image frames for display;

[0062] In response to a user selecting a viewpoint from a plurality of viewpoints in an image frame, the user-selected viewpoint is converted as the original image into the image to be corrected having depth information.

[0063] Secondly, this disclosure also provides an image correction device, including an image acquisition module, a first correction module, a verification module, and a second correction module;

[0064] The image acquisition module is configured to acquire an image to be corrected, the image to be corrected including depth information of each pixel;

[0065] The first correction module is configured to perform initial correction on the image to be corrected using a target correction model to obtain a first corrected image;

[0066] The verification module is configured to compare the first corrected image and the verification image. If the first similarity between the first corrected image and the verification image meets a preset condition, the first corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display. The verification image is an image estimated using the image to be corrected as an input sample and a standard depth estimation large model.

[0067] The second correction module is configured to, in response to a user's correction operation on the first correction image, perform supplementary correction on the first correction image to obtain a second correction image, and convert the second correction image into a stereoscopic image and send it to the display for stereoscopic display.

[0068] Thirdly, embodiments of this disclosure also provide an image correction system, including a processor and a display;

[0069] The processor is configured to: acquire an image to be corrected, the image to be corrected including depth information of each pixel; perform initial correction on the image to be corrected using a target correction model to obtain a first corrected image; compare the first corrected image with a verification image, and if the first similarity between the first corrected image and the verification image meets a preset condition, convert the first corrected image into a stereoscopic image and send it to a display for stereoscopic display; wherein the verification image is an image estimated using a standard depth estimation model with the image to be corrected as input sample; and, in response to a user's correction operation on the first corrected image, perform supplementary correction on the first corrected image to obtain a second corrected image, and convert the second corrected image into a stereoscopic image and send it to the display for stereoscopic display.

[0070] The display is configured to receive a stereoscopic image converted from the first corrected image and enable a stereoscopic display mode for stereoscopic display; and to receive a stereoscopic image converted from the second corrected image and enable a stereoscopic display mode for stereoscopic display.

[0071] In some embodiments, the image correction system further includes a cloud server; the cloud server is deployed with the ffmpeg framework;

[0072] The processor is further configured to acquire a multimedia file and determine the type of the multimedia file; if the type of the multimedia file is video, determine whether the resolution of the multimedia file exceeds a first preset resolution; if the resolution exceeds the first preset resolution, send the multimedia file to a cloud server; and, in response to receiving multiple consecutive image frames, send them to the display; and in response to a user selecting one of the viewpoint images from multiple viewpoint images in one image frame, convert the user-selected viewpoint image as the original image into the image to be corrected with depth information.

[0073] The cloud server is configured to receive the multimedia file, decode the multimedia file using the ffmpeg framework, cut the decoded multimedia file into multiple consecutive image frames, and send the multiple consecutive image frames to the processor.

[0074] The display is also configured to receive multiple image frames and enable a two-dimensional display mode.

[0075] In some embodiments, the cloud server has 48 or more physical cores and 96GB or more memory capacity.

[0076] In some embodiments, the image correction system further includes a graphics card; the graphics card is deployed with the ffmpeg framework;

[0077] The processor is further configured to send the multimedia file to the graphics card if the resolution of the multimedia file does not exceed a first preset resolution; and to convert the user-selected viewpoint map as the original image into the image to be corrected with depth information in response to a user selecting a viewpoint map from a plurality of viewpoint maps in an image frame.

[0078] The graphics card is configured to use the ffmpeg framework to decode the multimedia file into multiple consecutive image frames and send them to the display for display.

[0079] Fourthly, embodiments of this disclosure also provide a computer non-transient readable storage medium, wherein a computer program is stored on the computer non-transient readable storage medium, and the computer program is executed by a processor to perform the steps of the image correction method as described in the first aspect. Attached Figure Description

[0080] Figure 1 A flowchart of the image correction method provided in the embodiments of this disclosure;

[0081] Figure 2 A schematic diagram of the optimized target correction model provided in the embodiments of this disclosure;

[0082] Figure 3 A schematic diagram illustrating the calculation process of the first loss function provided in an embodiment of this disclosure;

[0083] Figure 4 A schematic diagram of the target correction model provided in the embodiments of this disclosure;

[0084] Figure 5 This is a graphical schematic diagram of the target correction model processing procedure provided in the embodiments of this disclosure;

[0085] Figure 6 A schematic diagram of gesture capture provided in an embodiment of this disclosure;

[0086] Figure 7 A schematic diagram of multi-viewpoint correction provided in an embodiment of this disclosure;

[0087] Figure 8 This is an overall flowchart of the image correction method provided in the embodiments of this disclosure;

[0088] Figure 9 A schematic flowchart illustrating the process of determining the second correction pattern provided in an embodiment of this disclosure;

[0089] Figure 10 This is a schematic diagram of an image correction device provided in an embodiment of the present disclosure;

[0090] Figure 11 This is a schematic diagram of an image correction system provided in an embodiment of the present disclosure;

[0091] Figure 12 A schematic diagram of another image correction system provided in an embodiment of this disclosure;

[0092] Figure 13 This is a schematic diagram of another image correction system provided in an embodiment of the present disclosure. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0094] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “including,” “comprising,” or “containing,” and similar terms mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms “upper,” “lower,” “left,” and “right,” etc., are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0095] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0096] In related technologies, passive technology is often used to produce 3D videos in order to save product costs and increase product applicability. However, current 2D video to 3D conversion often suffers from the following problems: First, different videos have different image quality parameters; however, for users, when using the same glasses-free 3D display as the playback source, the image quality parameters of each video are the same for the visual experience. This differs from the user's actual viewing of real-life scenes, making it difficult for the brain to synthesize a sense of realism, resulting in inconsistencies in user cognition and perception, thus affecting the viewing experience. Second, the application of glasses-free 3D displays includes not only playing fixed videos but also real-time video calls. This requires real-time conversion of 3D videos, which current 2D video to 3D conversion technology cannot achieve. Third, artificial intelligence (AI) video conversion technology is currently in the development stage and faces some technical challenges and limitations. For example, ① layer disorder: AI may make errors when processing the hierarchical relationship between different elements in a video, leading to confusion between foreground and background, or incorrect occlusion relationships between objects, resulting in layer disorder. ② Inaccurate cutout: When merging one part of an image or video with another, edge mismatch may occur, which is often referred to as inaccurate cutout. ③ Unclear outlines: In AI-generated videos, the outlines of objects may not be clear enough, and edges may appear blurry or jagged, affecting the visual effect. ④ Image distortion: During video compositing, improper algorithm processing may cause image distortion, affecting the viewing experience. ⑤ Image jitter: AI-generated videos may exhibit flickering or jittering, which may be due to insufficient processing of the continuity between video frames by the algorithm.

[0097] In view of this, embodiments of this disclosure provide an image correction method, apparatus, system, and storage medium, which substantially eliminates one or more problems caused by limitations and defects in related technologies. Specifically, embodiments of this disclosure utilize a target correction model to perform initial correction on the image to be corrected, initially correcting more obvious errors in the image to be corrected, resulting in a first corrected image. Subsequently, some errors caused by insufficient accuracy of the target correction model are not corrected. At this time, the user can preview the stereoscopic image on the display and provide relatively simple correction operations for the first corrected image on the user operation page, such as manually selecting the error location and providing replacement depth information. In response to the user's correction operations for the first corrected image, the first corrected image is further corrected to obtain a second corrected image, which is then displayed in stereo. In addition, throughout the correction process, the display shows the processing results in real time, namely the first and second corrected images, sharing the correction results with the user in real time for further fine-tuning until a satisfactory effect is achieved. In addition, since the target correction model is not accurate enough, the output of the target correction model (the first corrected image) is supervised by a standard depth estimation big model during the real-time correction stage. Since the standard depth estimation big model is a relatively mature depth estimation model, its depth estimation accuracy is higher than that of the target correction model. Therefore, using the image to be corrected as the input sample and the verification image estimated by the standard depth estimation big model as the ground truth of the first corrected image can improve the correction accuracy of the target correction model and improve the instability and detail processing capability of image correction.

[0098] Firstly, for ease of understanding, the image correction method provided in the embodiments of this disclosure will be described. The image correction method provided in this disclosure can be executed by a processor with a certain computer processing capability. Figure 1 A flowchart of the image correction method provided in the embodiments of this disclosure is shown below. Figure 1 As shown, the image correction method includes steps S11 to S14.

[0099] S11. Obtain the image to be corrected.

[0100] The image to be corrected includes depth information for each pixel. This depth information is typically recorded as a grayscale image or a depth map, where the intensity value of each pixel represents the depth value of the corresponding point in the scene. Storing the depth information as a grayscale image facilitates subsequent processing and analysis. It should be noted that although the image to be corrected is a two-dimensional image with depth information, it needs to be converted to a color image, not a grayscale image, for easier viewing by the user when displaying it in two dimensions on a monitor. Optionally, before performing depth operations on the grayscale image, it needs to be normalized to ensure that the depth values ​​of all pixels are on the same order of magnitude, facilitating subsequent processing.

[0101] S12. Using the target correction model, perform initial correction on the image to be corrected to obtain the first corrected image.

[0102] Among them, the target correction model is an AI model capable of converting 2D images into 3D images; it can be understood as an AI model provided in related technologies. However, due to the limited number of 3D video samples, the AI ​​model faces some technical challenges and limitations. In response, for example... Figure 2 As shown, this disclosure combines a standard depth estimation model 21 to optimize a target correction model 22, resulting in the target correction model 22 outputting a first corrected image 201 with relatively accurate repair precision. The optimization process includes comparing the first corrected image 201 with the verification image 202 to determine the first similarity. If the first similarity does not meet a preset condition, the parameters of the target correction model 22 are continuously adjusted and optimized until the first similarity meets the preset condition. At this point, the target correction model 22 can be considered to output a first corrected image 201 with relatively accurate repair precision.

[0103] Here, the verification image 202 is an image estimated using the image to be corrected 203 as input sample and a standard depth estimation large model.

[0104] Optionally, such as Figure 2 As shown, the target correction model can be a monocular depth estimation model with strong generalization and robustness, an open-source model, or other large-scale models specifically designed for depth estimation. Standard large-scale depth estimation models can include monocular depth estimation models with strong generalization and robustness, open-source models, or other large-scale models specifically designed for depth estimation.

[0105] S13. Compare the first corrected image and the verification image. If the first similarity between the first corrected image and the verification image meets the preset conditions, convert the first corrected image into a stereoscopic image and send it to the display for stereoscopic display.

[0106] In this step, the first corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display, so that the user can further judge whether the first corrected image currently repaired by the model meets expectations based on the stereoscopic display effect.

[0107] S14. In response to the user's correction operation on the first correction image, perform supplementary correction on the first correction image to obtain a second correction image, and convert the second correction image into a stereoscopic image and send it to the display for stereoscopic display.

[0108] This step, based on the automatic correction using the target correction model, further introduces a correction operation based on the user's intuitive experience, further compensating for the shortcomings of the automatic correction using the target correction model, and obtaining a second corrected image, thereby improving the quality of 2D to 3D converted images.

[0109] The display disclosed herein can be a display capable of freely switching between two-dimensional and three-dimensional display modes. Alternatively, this disclosure provides two displays, such as a two-dimensional display and a glasses-free 3D display.

[0110] The two-dimensional display shows the first corrected image and the user operation page for user adjustment. The user operation page includes various controls that support image information adjustment, such as controls for adjusting depth values, controls for editing images (including scaling, rotation, and movement), controls for adding text, and controls for overlaying virtual frames. In addition to these controls, it also includes controls for previous, next, undo, and delete operations. Besides these controls, the user operation page also includes the corrected first and / or second corrected images. The displayed controls provide users with the ability to adjust the target image; the first corrected image is displayed to support further adjustments; and the second corrected image is displayed to support further adjustments.

[0111] It should be noted that this only describes the process of correcting the first corrected image to obtain the second corrected image. In actual operation, if the user perceives that the stereoscopic image corresponding to the second corrected image still contains errors, they can continue to adjust the second corrected image using the controls provided on the user operation page. Through continuous cycle correction, a corrected image that satisfies both cognitive and perceptual levels can be obtained and saved.

[0112] The user interface is supported by the correction tool (software tool) developed in this disclosure. The user interface (UI) supports human-computer interaction and operational logic. The correction tool supports the series of controls listed above for image correction. Optionally, the operating system of the correction tool can be a Windows platform.

[0113] A glasses-free 3D display is used to show users glasses-free 3D stereoscopic images for reference when converting 2D to 3D video effects. This disclosure also provides a real-time preview function, allowing one-click projection onto a large glasses-free 3D screen. Based on set screen parameters (such as line count and prism tilt angle), the correctness and accuracy of the 3D display can be previewed in real time. Here, line count represents the number of sub-pixels covered by the orthographic projection of a prism onto the surface of a sub-pixel in the glasses-free stereoscopic display 30. Prism tilt angle is the angle between the prism and the y-direction in the screen's two-dimensional coordinate system.

[0114] In some embodiments, the first corrected image is converted into a stereoscopic image and sent to a display for stereoscopic display. Specifically, an image segmentation algorithm can be used to segment the first corrected image into a first left image adapted to the left eye and a first right image adapted to the right eye. Then, a preset image interleaving algorithm is used to interleave the pixels of the first left image and the first right image to generate a stereoscopic image. Finally, the stereoscopic image corresponding to the first corrected image is sent to the display for display. Similarly, the second corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display. Specifically, an image segmentation algorithm can be used to segment the second corrected image into a second left image adapted to the left eye and a second right image adapted to the right eye. Then, a preset image interleaving algorithm is used to interleave the pixels of the second left image and the second right image to generate a stereoscopic image. Finally, the stereoscopic image corresponding to the second corrected image is sent to the display for display.

[0115] In some embodiments, step S13, comparing the first similarity between the first corrected image and the verification image, specifically includes steps S131 to S135.

[0116] S131. Based on the first correction image and the verification image, determine the first loss function of the target correction model.

[0117] Here, the calculation of the first loss function includes, but is not limited to: In one example, the pixel loss and edge loss between the first corrected image and the verification image can be calculated; based on the pixel loss and edge loss, the first loss function is determined. In another example, the pixel loss between the first corrected image and the verification image can also be calculated, and this pixel loss can be used as the first loss function. In yet another example, the edge loss between the first corrected image and the verification image can also be calculated, and this edge loss can be used as the first loss function.

[0118] It should be noted that the preset threshold can be set according to actual conditions and experience. The smaller the preset threshold, the closer the first corrected image output by the target correction model will be to the verification image.

[0119] S132. Determine whether the loss value of the first loss function is less than or equal to the preset threshold; if yes, execute S133 sequentially; otherwise, execute S134.

[0120] S133. If the loss value of the first loss function is less than or equal to a preset threshold, determine that the first similarity satisfies the preset condition.

[0121] S134. If the loss value corresponding to the first loss function is greater than the preset threshold, it is determined that the first similarity does not meet the preset condition.

[0122] Furthermore, if the first similarity does not meet the preset conditions, S135 is executed.

[0123] S135. Use the first loss function to back-train the target correction model, using the same image to be corrected as the iterative sample, until the first similarity between the output of the target correction model and the verification image meets the preset condition.

[0124] Here, the training process of the target correction model is a supervised training process.

[0125] This embodiment utilizes a first loss function to characterize the first similarity, which not only monitors the correction quality of the target correction model but also allows for reverse training of the target correction model using the first loss function when the loss value corresponding to the first loss function exceeds a preset threshold, thereby improving the correction accuracy of the target correction model. Here, the reverse training process, i.e., the process of adjusting the training parameters of the target correction model, is a technical means that is known to those skilled in the art and will not be described in detail here.

[0126] It should be noted that the technical means for calculating the first similarity mentioned above include, but are not limited to, the calculation of the loss function. Other algorithms that can calculate depth map similarity can also be selected, such as histogram method, structural similarity measure (SSIM), etc.

[0127] Figure 3 This is a schematic diagram illustrating the calculation process of the first loss function provided in an embodiment of this disclosure. In some embodiments, regarding S131 above, as... Figure 3 As shown, it specifically includes S1311 to S1313.

[0128] S1311. Compare the pixel information of corresponding pixels between the first corrected image and the verification image to determine the pixel loss between the first corrected image and the verification image.

[0129] Here, pixel information refers to the grayscale value corresponding to a pixel. This pixel loss can be characterized as the pixel difference between the first corrected image and the verification image.

[0130] S1312. Using an edge detection algorithm, extract the edge contour information of the first corrected image and the verification image respectively, and obtain the edge loss between the first corrected image and the verification image.

[0131] The edge contour information includes the coordinates of the edge contours of all objects with contours in the image and the grayscale values ​​of the corresponding pixels. Edge loss can be characterized as the difference between the edge contours of the object in the first corrected image and the edge contours of the same object in the verification image.

[0132] S1313. Determine the first loss function based on pixel loss and edge loss.

[0133] Specifically, pixel loss and edge loss can be weighted to obtain the first loss function, where the weighting method can be found in Formula 1.

[0134] Formula 1: Loss_1 = α × loss(outout, GT) + β × loss'(Canny(outout), Canny(GT)). Where Loss_1 represents the first loss function; loss(outout, GT) represents pixel loss; loss'(Canny(outout), Canny(GT)) represents edge loss; outout represents the first corrected image; GT represents the verification image; Canny(outout) represents the edge detection result of the first corrected image; and Canny(GT) represents the edge detection result of the verification image. The smaller the loss value corresponding to the first loss function, the closer the first corrected image and the verification image are, meaning the higher the first similarity and the more it meets the preset conditions.

[0135] Optionally, the edge loss loss'(Canny(outout),Canny(GT)) can be calculated using the cross-entropy loss function algorithm.

[0136] Figure 4 This is a schematic diagram of a target correction model provided in an embodiment of this disclosure. In some embodiments, such as Figure 4 As shown, the target correction model 22 includes an encoder 221 and a decoder 222. The encoder includes multiple layers of convolutional neural network layers for encoding, such as an input layer (Init Block), a first up convolutional layer (UpBlock1), a second up convolutional layer (Up Block2), and a third up convolutional layer (Up Block3). The decoder also includes multiple layers of convolutional neural network layers, such as a first down convolutional layer (Down Block1), a second down convolutional layer (Down Block2), a third down convolutional layer (Down Block3), and an output layer (Out Block). Here, "up" can be understood as upsampling, and "down" can be understood as downsampling.

[0137] The target correction model can identify the error locations in the image to be corrected and overlay a visual layer box on the error locations for the user to view during the display stage, so that the user can obtain the correct depth information based on the content selected by the visual layer box.

[0138] Figure 5 This is a graphical schematic diagram of the target correction model processing procedure provided in the embodiments of this disclosure. Specifically, S12 includes S121 to S123, as follows... Figure 5 As shown.

[0139] S121. Using the target correction model, identify and mark the error locations in the image 203 to be corrected, and obtain the marked location information.

[0140] S122. Overlay a visualization layer box 204 onto the position indicated by the marked position information in the image to be corrected 203, synthesize a multi-layer correction image 205, and display a stereoscopic image based on the transformation of the multi-layer correction image.

[0141] Optionally, the synthesized multi-layered corrected image 205 can also be displayed in two dimensions on a monitor. The visualization layer box 204 and the image to be corrected 203 belong to different layers. The visualization layer box 204 is mainly used to provide the user with a clearer location of errors during the display process.

[0142] S123, Responding to the correct depth information selected by the user for the visual layer box 204 (such as...) Figure 5 The depth value of the selected position in the virtual box 206 is used to correct the depth information of the incorrect position in the image to be corrected 203 based on the marked position information and the correct depth information, so as to obtain the first corrected image 201.

[0143] For example, based on the marked location information and the correct depth information, a depth map processing algorithm can be used to correct the depth information of the erroneous position in the image to be corrected 203, and a first corrected image 201 can be obtained.

[0144] Optionally, the correct depth information can be generated in the following ways: One possible implementation is to use the depth information of the user-selected location as the correct depth information in response to a user's selection operation of depth information at a certain position in the multi-layer correction image. Another possible implementation is to use a certain depth information input by the user for the multi-layer correction image as the correct depth information.

[0145] Optionally, the target correction model assigns the correct depth information provided by the user to the erroneous location in the image to be corrected, and outputs a new image, namely the first corrected image.

[0146] In some embodiments, the user operation page includes a multimedia mode, a gesture mode, and an editing mode. Regarding S14, in response to the user's correction operation on the first correction image, the first correction image is further corrected to obtain a second correction image. The user's correction operation on the first correction image can be a correction operation performed by the user via voice or facial expression in the multimedia mode. For example, a voice instruction to "increase the depth value of target A," or a clear voice instruction to "set the depth of target A to XX," or a facial expression instruction to "enhance the out-of-screen effect," etc. Alternatively, in the gesture mode, the user can select a specific area in the first correction image using gestures (including air selection or touch selection), as the area to be corrected. Alternatively, in the editing mode, the user can use an external device such as a mouse to click or select a specific area in the first correction image, as the area to be corrected; or actively edit text to generate an adjustment scheme, such as increasing the depth value of target A, or setting the depth value of target A to XX, etc.

[0147] In one possible implementation, the user's correction operation on the first correction image includes selection, facial expressions, and / or voice actions. Taking the user selecting a multimedia mode as an example, the specific steps include S14-1-1 to S14-1-3.

[0148] S14-1-1 In response to the user selecting a multimedia mode, the received multimedia information of the user is parsed, and the user's facial features and / or voice features are identified.

[0149] The multimedia information includes at least one of the following: images and videos of the user captured in real time, and audio recorded in real time. The image correction system also includes a camera, such as a monocular camera or a binocular camera, for capturing images or videos of the user in real time; and an audio acquisition device, such as a built-in microphone or a virtual microphone component, for acquiring the user's audio in real time.

[0150] Optionally, the Llama language model can be used to parse the received multimedia information from the user to identify the user's facial features and / or voice features. The Llama language model is deployed in the processor.

[0151] S14-1-2. Determine the user's first correction information based on facial features and / or voice features.

[0152] Optionally, for facial features, based on the specific meaning expressed by the facial features, such as capturing negative emotions like shaking one's head or frowning, it can be determined whether the user is dissatisfied with the current display; for voice features, based on the specific meaning expressed by the voice features, such as setting the depth of target A to XX (first correction information), it can further clarify the user's decision-making scheme for secondary correction. For example, after capturing the aforementioned facial features of negative emotions, secondary correction is required. For example, the algorithm is first notified to perform automatic correction, and then, combined with voice prompts, such as "enhance the out-of-screen effect," a decision-making scheme for enhancing the out-of-screen effect is generated, which is the first correction information, such as enhancing the out-of-screen effect of target A, with a depth value of XX.

[0153] S14-1-3. Based on the first correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

[0154] Based on the first correction information being "enhance the out-of-screen effect of target A, and the depth value is XX", the depth of the area to be corrected in the first correction image is adjusted to XX to obtain the second correction image.

[0155] In another possible implementation, the user's correction operation on the first correction image includes gesture operation; taking the user's selection of multimedia mode as an example, it specifically includes S14-2-1 to S14-2-3.

[0156] S14-2-1. In response to the user selecting a gesture mode, the user's gesture information is identified in response to the user's gesture operation on the first correction image.

[0157] The camera can capture images of the user's gestures. In response to the user's gestures on the first correction image, the gesture images are analyzed to identify the user's gesture information, which includes at least the coordinates of the area to be corrected in the first correction image manually selected by the user.

[0158] S14-2-2. Determine the user's second correction information based on gesture information.

[0159] Specifically, based on gesture information, the depth information corresponding to the area to be corrected can be retrieved to determine the user's second correction information. The second correction information includes at least the coordinates of the area to be corrected and the depth information within that area.

[0160] S14-2-3. Based on the second correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

[0161] Specifically, based on the coordinates of the region to be corrected and the depth information within that region, the depth information corresponding to the region to be corrected is replaced with the correct depth information to obtain the second corrected image. The correct depth information can be information entered by the user through gestures on the user interface, information recorded by clicking the capacity bar of the depth information control, the depth value of a specific location selected by the mouse in the first corrected image, or a specific depth value actively entered by the user using the keyboard.

[0162] In addition, algorithms such as interpolation, filtering, and pattern matching are applied during the depth assignment process to ensure a smooth transition and overall consistency of depth information between the region to be corrected and the uncorrected region in the first correction image.

[0163] Of course, there is more than one example of the above gesture information. It can also be other gesture operations that can be used, such as gesture operations that enhance the out-of-screen effect.

[0164] Optionally, the gesture image is analyzed to identify the user's gesture information. Specifically, the human eye coordinate system, finger coordinate system, and screen coordinate system form a three-level projection system transformation. Through two conversions and mappings, a comprehensive determination of the area to be corrected can be completed. Figure 6 A schematic diagram of gesture capture provided in an embodiment of this disclosure, such as... Figure 6As shown, firstly, assume the screen is a two-dimensional plane located on a horizontal plane, with its origin at the center of the screen, and establish a two-dimensional coordinate system xoy for the screen. The observer's position is fixed and their line of sight is perpendicular to the screen: For simplification, we assume the observer's (eye's) position is fixed and their line of sight is directly perpendicular to the center of the screen. This ignores the influence of changes in viewing angle, establishing an eye coordinate system. The direction the eye looks towards the center of the screen can be considered the positive z-axis of the eye coordinate system. However, in actual calculations, the eye position, as the observation point, involves more about the angle of sight than coordinate system transformation. The finger is located at a point between the eye and the screen: the finger does not touch the screen and lies on the line connecting the eye and the screen plane, meaning the finger, eye, and screen center are collinear. The finger coordinates describe the relative position of the finger in the direction of the eye's observation. In this context, this "coordinate" is more of a visual description of relative position than a strict mathematical coordinate. For example, if the eye position is fixed and the line of sight is perpendicular to the screen, then the projection of the finger onto the screen coordinate system is (0,0). This discussion is relatively basic and idealized; practical applications may involve more complex factors. This disclosure only provides a simple explanation of the principle of gesture capture. For example, the distance from the hand to the origin (0,0) of the 2D display screen is *a*, and the distance from the eye to the hand is *b*. During eye capture, the angle *c* between the eye and the z-axis, and the eye height *h*, can be obtained. The hand height *a×tan(c)* can be calculated using the formula: *a×tan(c) / h = r×d*. Here, *r* represents the radius of the gesture operation circle; *d* represents the field of view, which can be understood as the inscribed circle of the screen's visible area or the maximum visible radius of the screen's 3D display effect. Since *a×tan(c)*, *h*, and *d* are known, according to the formula *a×tan(c) / h = r×d*, *r* can be calculated as *a×tan(c) / (h×d)*. Next, to calculate the projected area *s*, we need to know the screen size or the distance between the screen and the eyes. Typically, the projected area can be calculated using the following formula: s = 2π × r. This is because the projected area is a circular region centered on the center of the screen, and its area is equal to the area of ​​a circle, which is calculated using the formula 2πr.

[0165] The second correction information also includes the shape, timestamp, correction depth information, and before-and-after information of the area to be corrected selected by the user.

[0166] In another possible implementation, the user's correction operation on the first correction image includes an editing operation. Taking the user selecting the editing mode as an example, the specific steps include S14-3-1 to S14-3-2.

[0167] S14-3-1. In response to the user selecting an editing mode, the third correction information is determined based on the received editing operation of the user on the first correction image.

[0168] Optionally, the editing operation can be a user-initiated operation provided by an external input device such as a keyboard or mouse. For example, the editing operation is to click or circle a specific area in the first correction image as the area to be corrected; the third correction information includes the coordinates of the area to be corrected and the depth information within the area to be corrected.

[0169] S14-3-2. Based on the third correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

[0170] Based on the coordinates of the region to be corrected and the depth information within that region, the depth information corresponding to the region to be corrected is replaced with the correct depth information to obtain the second corrected image. The correct depth information can be information entered by the user through gestures on the user interface, information recorded by clicking the capacity bar of the depth information control, the depth value of a specific location selected by the mouse in the first corrected image, or a specific depth value actively entered by the user using the keyboard.

[0171] Furthermore, during the depth assignment process, pixels in the first corrected image need to be remapped to ensure that the depth value of the area to be corrected is consistent with the surrounding environment, avoiding visual inconsistencies. Specifically, algorithms such as interpolation, filtering, and pattern matching can be applied to ensure a smooth transition and overall consistency of depth information between the area to be corrected and the uncorrected areas in the first corrected image.

[0172] Of course, there are more than one example of the above editing operations. It could also be other operations that can be performed using a mouse or keyboard, such as editing text to enhance the out-of-screen effect.

[0173] In another possible implementation, the example of selecting at least two of the above-mentioned multimedia mode, gesture mode and editing mode is described, specifically including S14-4-1 to S14-4-4.

[0174] S14-4-1. In response to the user selecting a multimedia mode, the received multimedia information of the user is parsed, and the user's facial features and / or voice features are identified; based on the facial features and / or voice features, the user's first correction information is determined.

[0175] S14-4-2. In response to the user selecting a gesture mode, in response to the user's gesture operation on the first correction image, the user's gesture information is identified; based on the gesture information, the user's second correction information is determined.

[0176] S14-4-3. In response to the user selecting an editing mode, the third correction information is determined based on the received editing operation of the user on the first correction image.

[0177] S14-4-4. Based on at least two of the first correction information, the second correction information, and the third correction information, adjust the depth of the region to be corrected in the first correction image to obtain the second correction image.

[0178] Optionally, the first corrected image is processed based on the first correction information and the second correction information to obtain the second corrected image. For example, the first corrected image can be processed based on the first correction information, and then the intermediate processing results of the first corrected image can be processed based on the second correction information to obtain the second corrected image.

[0179] Optionally, the first corrected image is processed based on the first correction information and the third correction information to obtain the second corrected image. For example, the first corrected image can be processed based on the first correction information, and then the intermediate processing results of the first corrected image can be processed based on the third correction information to obtain the second corrected image.

[0180] Optionally, the first corrected image is processed based on the second and third correction information to obtain the second corrected image. For example, the first corrected image can be processed based on the second correction information, and then the intermediate processing results of the first corrected image can be processed based on the third correction information to obtain the second corrected image.

[0181] Optionally, the first corrected image is processed based on the first correction information, the second correction information, and the third correction information to obtain the second corrected image. For example, the first corrected image can be processed based on the first correction information; then, based on the second correction information, the first intermediate processing result of the first corrected image can be processed to obtain the second intermediate processing result; then, based on the third correction information, the second intermediate processing result can be processed to obtain the second corrected image.

[0182] In some embodiments, the obtained first correction information, second correction information and third correction information are sent to a memory for storage, so as to be called in subsequent processing.

[0183] Optionally, in response to an assistance request initiated by the user, at least one of the first correction information, the second correction information, and the third correction information stored in the memory during the historical phase is invoked to generate auxiliary correction information and display it in two dimensions.

[0184] The auxiliary corrective information includes at least one of the following: historical data, first corrective information, second corrective information, and third corrective information.

[0185] In this embodiment, the image correction system also provides data recording and analysis functions, which can record the user's operation data during the interactive correction process and analyze and process this data. When the user subsequently initiates an assistance request, intelligent correction suggestions and assistance functions can be provided based on the user's historical operation data. The entire workflow of the image correction system provided in this embodiment is designed to minimize the difficulty of operation for users, allowing even users without a professional image processing background to easily complete the 2D to 3D conversion task.

[0186] Figure 7 This is a schematic diagram illustrating multi-viewpoint correction provided in an embodiment of this disclosure. In some embodiments, if the correction process involves multiple layers or multi-view images, image compositing is required to ensure that the depth information of all views is correctly corrected and aligned. Figure 7 As shown, the image correction method also includes S21 to S23.

[0187] S21. If there are multiple other viewpoint maps 71 in the image 203 to be corrected, acquire multiple viewpoint maps 71 and calculate the disparity information between each viewpoint map 71 and the second corrected image 72.

[0188] Because of the parallax between multi-view images, the correction effect needs to be adjusted according to the specific circumstances of each viewpoint to ensure visual consistency.

[0189] S22. Based on the disparity information corresponding to each viewpoint image 71, the position and depth information of the pixels of each viewpoint image 71, and the target correction information of the second correction image 72, determine the fourth correction information corresponding to each viewpoint image 71.

[0190] The target correction information includes at least the correction information adjusted by the user for the image 203 to be corrected, and the correction information adjusted using the target correction model. The correction information adjusted by the user for the image 203 to be corrected includes first correction information, second correction information, and third correction information. The correction information adjusted using the target correction model 22, i.e., the information automatically repaired by the target correction model 22, such as... Figure 2 The information shown is related to the automatically repaired part, such as coordinates and depth values.

[0191] S23. Based on the fourth correction information, the corresponding viewpoint map 71 is processed to obtain the corrected viewpoint map 73.

[0192] The fourth correction information includes target correction information and information about the area to be corrected in the viewpoint diagram.

[0193] Taking one viewpoint image from multiple viewpoint images, denoted as the target viewpoint image, as an example, given the disparity information between the target viewpoint image and the second corrected image, as well as the position coordinates and depth information of each pixel in the target viewpoint image, there is no need to actively identify the region to be corrected in the target viewpoint image. Based on the disparity information between the target viewpoint image and the second corrected image, the position coordinates of each pixel in the target viewpoint image and the position coordinates of each pixel in the second corrected image, the region to be corrected in the second corrected image is known, and thus the region to be corrected in the target viewpoint image can be obtained. Then, based on the target correction information in the second corrected image, the depth information of the region to be corrected in the target viewpoint image is corrected. The correction includes the information corrected during the transformation of the image to be corrected into the second corrected image, resulting in the corrected viewpoint image.

[0194] This disclosure discloses a correction process that is adjusted according to the specific circumstances of each viewpoint image to ensure visual consistency.

[0195] In some embodiments, the image correction method further includes steps S31 to S33.

[0196] S31. In response to the user's correction operation on the first correction image, generate auxiliary training information for a standard depth estimation large model.

[0197] In response to a user's correction operation on a first correction image, for example, in response to a user selecting a multimedia mode, the user's first correction information is determined and used as auxiliary training information for a standard depth estimation large model; and / or, in response to a user selecting a gesture mode, the user's second correction information is determined and used as auxiliary training information for a standard depth estimation large model; and / or, in response to a user selecting an edit mode, third correction information is determined and used as auxiliary training information for a standard depth estimation large model.

[0198] The auxiliary training information may include at least one of the first corrective information, the second corrective information, and the third corrective information.

[0199] S32. Adjust the model parameters of the standard deep estimation large model based on auxiliary training information.

[0200] The model parameters include, for example, weight values ​​and model structure.

[0201] S33. Use the first corrected image as the training sample of the standard depth estimation model, and continue to iterate the training of the standard depth estimation model until the standard depth estimation model converges.

[0202] The first corrected image is used as a training sample for a standard large-scale depth estimation model to obtain the output result. This output result is then compared with the ground truth value of the first corrected image (i.e., the corrected second corrected image) to calculate the third loss function. Further, the method for calculating the third loss function can be found in Formula 1; repeated parts will not be elaborated upon.

[0203] Then, the standard depth estimation model is trained in reverse using the third loss function. The output of the standard depth estimation model is used as the iterative sample to continuously train the standard depth estimation model until the output of the standard depth estimation model and the third loss function of the second correction map meet the convergence condition.

[0204] This embodiment improves video stability and detail processing capabilities by developing more advanced network structures and training strategies, and by introducing user feedback and iterative optimization. It enables the training of the target correction model and the standard depth estimation model, thereby solving the problems of hierarchical disorder and subtle image distortion in 3D videos.

[0205] In some embodiments, after the standard depth estimation large model is trained according to the above embodiments, the target correction model trained under supervision based on the standard depth estimation large model also needs to be adjusted synchronously.

[0206] In practice, the first corrected image can be used as the training sample of the target correction model, and the target correction model can be iteratively trained until the second similarity between the output of the target correction model and the verification result of the standard depth estimation large model after training meets the preset conditions.

[0207] Here, the data from repeated successful corrections of the target correction model can be used as the training set; the type of training set varies depending on the business type, such as human image training set, plant training set, animal training set, video call training set, etc.

[0208] The principle of further training the target correction model here is the same as that of training the target correction model described above. The main difference between the two is that the training samples are different, and the output results are naturally different as well. Therefore, for detailed explanations of training the target correction model, please refer to the above implementation example of training the target correction model. Repeated points will not be repeated.

[0209] The verification result is the output of a standard depth estimation model that has been trained, using the first corrected image as training samples. The similarity between this verification result and the fully corrected second image meets a preset condition. Furthermore, the second similarity between the output of the target correction model and the verification result output by the trained standard depth estimation model meets a preset condition. Therefore, it can be inferred that the similarity between the output of the target correction model and the fully corrected second image meets the preset condition; that is, the output of the trained target correction model is approximately the same as the target correction model before training and the final second corrected image after manual user adjustments.

[0210] This embodiment combines auxiliary training information from a standard depth estimation model to further train the standard depth estimation model and the target correction model, thereby improving the correction accuracy of the target correction model and enhancing the instability and detail processing capabilities of image correction. Furthermore, compared to the long pre-training production cycle of traditional AI models, this disclosure provides appropriate open parameters for subsequent secondary processing and training, which can effectively improve the delivery production cycle and accuracy, and reduce the waste of ineffective manpower and resources.

[0211] Furthermore, this disclosure analyzes and mitigates the impact of multi-model (algorithm) linkage on computing power: Since the goal of this disclosure is to manage and control calibration and subsequent secondary calibration, neither the correction model (target correction model) nor the auxiliary model (standard depth estimation large model) will experience parallelism or crosstalk. It can be clearly stated that the judgment and processing are sequential but do not crosstalk with each other. As for the collection of human body data such as face recognition and gesture recognition, these are byproducts generated during the implementation of human eye tracking and do not require additional collection to compensate for them.

[0212] In some embodiments, the image correction method further includes a process of generating an image to be corrected, specifically including steps S41 to S44.

[0213] S41. Obtain the multimedia file and determine its type.

[0214] Multimedia files can be of various types, such as video, audio, or images.

[0215] S42. If the multimedia file is a video, determine whether the resolution of the multimedia file exceeds the first preset resolution. If it does, execute S43 sequentially; otherwise, execute S45 or S47.

[0216] The resolution of the multimedia file refers to the resolution of a single frame in the video. Specifically, it can be the resolution of any single frame or the highest resolution among multiple frames. The first preset resolution can be set to a high resolution, such as 8K, 10K, 12K, and 16K. This disclosure uses a first preset resolution of 16K as an example for illustration.

[0217] S43. If the resolution exceeds the first preset resolution, send the multimedia file to the cloud server.

[0218] The cloud server is equipped with the ffmpeg framework to decode multimedia files and then segment the decoded multimedia files into multiple consecutive image frames for display. Optionally, the cloud server 40 has 48 or more physical cores and 96GB or more of memory.

[0219] S44. In response to the user selecting a viewpoint image from multiple viewpoint images in an image frame, convert the user-selected viewpoint image as the original image into an image to be corrected with depth information.

[0220] The image frame selected by the user includes multiple viewpoint images, such as 3×3 viewpoint images. One is the original image, and the other eight are images corresponding to other viewpoints of the original image. The user selects the original image from the multiple viewpoint images as the image to be corrected. The correction of the other viewpoint images can be adjusted based on the final correction parameters of the image to be corrected. The multiple viewpoint images are images at the same time point.

[0221] Here, the original image is a color image, and to obtain depth information, depth extraction is required. The method of depth extraction may vary depending on the source of the multimedia file.

[0222] For example, when the multimedia file is a real-time video of a real scene, specialized equipment can be deployed on-site to capture the depth information of the scene as depth information for image frames and store it in memory for the processor to access at any time. This specialized equipment could be, for example, a depth sensor, a stereo camera, or a LiDAR sensor.

[0223] For example, if the multimedia file is a pre-recorded video, then the method for extracting depth information from the video image frames needs to rely on depth extraction techniques. These techniques include, but are not limited to, deep learning algorithms, or implementations based on Field Programmable Gate Arrays (FPGAs) in the HSV color space. Using pre-trained or theoretically derived depth mapping functions, the V channel values ​​in HSV are mapped to estimated depth values, which serve as the depth information for the corresponding pixels. This depth information is then sent to memory for storage, allowing the processor to access it at any time.

[0224] This embodiment uses FFmpeg for video encoding and decoding. Since FFmpeg also supports GPU acceleration, this significantly improves video encoding and decoding efficiency. The FFmpeg framework is licensed under the LGPL or GPL, making it free to obtain and allowing users to freely use and modify the source code, thereby reducing system operating costs and increasing user control over encoding and decoding. Furthermore, this embodiment utilizes a powerful cloud CPU server (at least 48 cores and 96GB of RAM) to perform software encoding, which is particularly beneficial for high-resolution videos above 16K, optimizing compression efficiency and image quality.

[0225] In some embodiments, the processor is equipped with the ffmpeg framework. Therefore, in addition to S42 described above, it also includes S45 to S46.

[0226] S45. If the resolution of the multimedia file does not exceed the first preset resolution, use the ffmpeg framework to decode the multimedia file, obtain multiple consecutive image frames, and display them.

[0227] S46. In response to the user selecting a viewpoint from multiple viewpoints in an image frame, the user-selected viewpoint is converted into an image to be corrected with depth information as the original image.

[0228] Of course, following the above S42, if the resolution of the multimedia file does not exceed the first preset resolution, ordinary encoding and decoding algorithms can also be used to decode the multimedia file. Of course, this may result in some loss of computational efficiency and cost, which will not be listed in this disclosure.

[0229] In this embodiment, videos with resolutions below 16K are encoded using hardware, while videos with resolutions above 16K are encoded using software. Cloud server software encoding typically relies on software algorithms to encode video data, requiring significant cloud CPU computation. To avoid impacting development, this disclosure provides a cloud-based server with a CPU of more than 48 cores and 96GB for encoding and decoding 16K videos. The software encoder can process video data more flexibly, thus typically providing better video quality than hardware encoding at low bitrates, thereby optimizing compression efficiency and picture quality. Hardware encoding is usually integrated into the processor, enabling direct hardware-level processing of video data, thereby reducing the burden on the cloud CPU and providing a smoother playback experience. The correction tool in this disclosure uses the ffmpeg framework for encoding and decoding, supporting hardware-accelerated decoding of encoding formats such as H.264, further reducing the cloud CPU burden and providing a smooth playback experience.

[0230] In some embodiments, following step S41 above, if the multimedia file is an image, the multimedia file is converted into an image to be corrected with depth information. Here, the technical principle of converting an image into an image to be corrected with depth information is the same as the technical principle of converting a single image frame in a video into an image to be corrected with depth information; therefore, the repeated parts will not be described again.

[0231] In some embodiments, the graphics card deploys the ffmpeg framework. Therefore, in addition to S42 described above, it also includes S47 to S48.

[0232] S47. If the resolution of the multimedia file does not exceed the first preset resolution, send the multimedia file to the graphics card.

[0233] The graphics card is equipped with the ffmpeg framework to decode multimedia files and then cut the decoded multimedia files into multiple consecutive image frames for display.

[0234] S48. In response to the user selecting a viewpoint from multiple viewpoints in an image frame, the user-selected viewpoint is converted into an image to be corrected with depth information as the original image.

[0235] This embodiment integrates hardware encoding into the graphics card, enabling direct processing of video data at the hardware level, thereby reducing the burden on cloud CPUs and processors and providing a smoother playback experience.

[0236] The above is a complete description of the image correction method provided in the embodiments of this disclosure. Based on this, in order to further clarify the overall architecture of the image correction method of this disclosure, the overall workflow of the image correction method is described below.

[0237] Figure 8This is an overall flowchart of the image correction method provided in the embodiments of this disclosure, such as... Figure 8 As shown, S51: Read the multimedia file; S52: Determine the type of the multimedia file. If it is video, execute S53 sequentially; if it is an image, jump to S515; S53: If the multimedia file is video, decode the video into multiple consecutive image frames; S54: Select one viewpoint image from multiple viewpoint images in one image frame as the original image, and convert the original image into an image to be corrected with depth information; S55: Process the image to be corrected using the target correction model to obtain the first corrected image; S56: Generate a stereoscopic image using a preset image interleaving algorithm; S57: Display the stereoscopic image on the monitor; S58: Determine if the first corrected image is qualified. If it is not qualified, execute S515. 59. If qualified, proceed to S510; S59. Manually correct to obtain the second corrected image, and return to S56 until the second corrected image is qualified, then proceed to S510; S510. Save the first corrected image (or the second corrected image). If the multimedia file type is an image, proceed to S511; if the multimedia file type is an image, proceed to S514; S511. Determine if all consecutive image frames in the video have been corrected. If there are still uncorrected image frames, return to S54 to continue correction. If all consecutive image frames have been corrected, proceed to S512; S512. Generate the corrected 3D video according to the image frame sequence using encoding and decoding technology; S513. Export. Alternatively, if the multimedia file type is an image, no encoding or decoding is required, and proceed directly to S515; S515. Directly convert the image into an image to be corrected with depth information, and sequentially execute S55 until export.

[0238] In the overall workflow described above, the image to be corrected is processed to obtain the first corrected image. Specifically, this includes: identifying the error locations in the image to be corrected; marking the error locations; overlaying a visualization layer box; and, in response to the user-selected correct location information, assigning the depth information of the correct location information to the error location.

[0239] Figure 9 This is a schematic flowchart illustrating the process of determining the second correction pattern according to an embodiment of the present disclosure, such as... Figure 9 As shown, the manual correction process in the above overall workflow is based on the user correction page provided by the correction tool to realize human-computer interaction and obtain the second correction image. Specifically, it includes: responding to the user's selection of correction mode, including multimedia mode, gesture mode and editing mode; different correction modes correspond to different correction logic, which respectively correspond to obtaining the first correction information, the second correction information and the third correction information. One mode or multiple modes can be selected and combined to generate correction information for correcting the first correction image to obtain the second correction image.

[0240] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0241] Secondly, embodiments of this disclosure also provide an image correction device. Figure 10 This is a schematic diagram of the image correction device provided in the embodiments of this disclosure, such as... Figure 10 As shown, the head image correction device includes an image acquisition module 101, a first correction module 102, a verification module 103, and a second correction module 104.

[0242] The image acquisition module 101 is configured to acquire an image to be corrected, which includes depth information of each pixel.

[0243] It should be noted that the image acquisition module 101 in this embodiment is configured to perform step S11 in the above image correction method, and repeated parts will not be described again.

[0244] The first correction module 102 is configured to use the target correction model to perform initial correction on the image to be corrected, and obtain the first corrected image.

[0245] It should be noted that the first correction module 102 in this embodiment is configured to perform step S12 in the above image correction method, and the repeated parts will not be described again.

[0246] The verification module 103 is configured to compare the first corrected image and the verification image. If the first similarity between the first corrected image and the verification image meets the preset conditions, the first corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display. The verification image is an image estimated by using the image to be corrected as an input sample and a standard depth estimation large model.

[0247] It should be noted that the verification module 103 in this embodiment is configured to execute step S13 in the above image correction method, and the repeated parts will not be described again.

[0248] The second correction module 104 is configured to, in response to a user's correction operation on the first correction image, perform supplementary correction on the first correction image to obtain a second correction image, and convert the second correction image into a stereoscopic image and send it to the display for stereoscopic display.

[0249] It should be noted that the second correction module 104 in this embodiment is configured to perform step S14 in the above image correction method, and the repeated parts will not be described again.

[0250] For example, the image correction apparatus provided in this disclosure can be considered as a processor for performing an image correction method.

[0251] Thirdly, embodiments of this disclosure also provide an image correction system. Figure 11 This is a schematic diagram of an image correction system provided in an embodiment of the present disclosure, such as... Figure 11 As shown, the image correction system includes a processor 61 and a display 62.

[0252] The processor 61 is configured to acquire an image to be corrected, which includes depth information of each pixel; perform initial correction on the image to be corrected using a target correction model to obtain a first corrected image; compare the first corrected image with a verification image, and if the first similarity between the first corrected image and the verification image meets a preset condition, convert the first corrected image into a stereoscopic image and send it to the display 62 for stereoscopic display; wherein the verification image is an image estimated using a standard depth estimation model with the image to be corrected as input sample; and, in response to a user's correction operation on the first corrected image, perform supplementary correction on the first corrected image to obtain a second corrected image, and convert the second corrected image into a stereoscopic image and send it to the display 62 for stereoscopic display.

[0253] The display 62 is configured to receive a stereoscopic image converted from a first corrected image and enable stereoscopic display mode for stereoscopic display; and to receive a stereoscopic image converted from a second corrected image and enable stereoscopic display mode for stereoscopic display.

[0254] In some embodiments, the processor 61 compares the first similarity between the first corrected image and the verification image, specifically including: determining the first loss function of the target correction model based on the first corrected image and the verification image; determining that the first similarity satisfies the preset condition if the loss value of the first loss function is less than or equal to a preset threshold; and determining that the first similarity does not satisfy the preset condition if the loss value corresponding to the first loss function is greater than the preset threshold.

[0255] Here, the calculation of the first loss function includes, but is not limited to: In one example, the pixel loss and edge loss between the first corrected image and the verification image can be calculated; based on the pixel loss and edge loss, the first loss function is determined. In another example, the pixel loss between the first corrected image and the verification image can also be calculated, and this pixel loss can be used as the first loss function. In yet another example, the edge loss between the first corrected image and the verification image can also be calculated, and this edge loss can be used as the first loss function.

[0256] Furthermore, the processor 61 is also configured to, in the event that the first similarity does not meet the preset conditions, use the first loss function to back-train the target correction model, using the same image to be corrected as the iterative sample, until the similarity between the output of the target correction model and the verification image meets the preset conditions.

[0257] In some embodiments, the processor 61 determines a first loss function of a preset correction model based on the first corrected image and the verification image, such as... Figure 3 As shown, the specific steps include: comparing the pixel information of corresponding pixels between the first corrected image and the verification image to determine the pixel loss between the two images. Pixel information refers to the grayscale value corresponding to a pixel. This pixel loss can be characterized as the pixel difference between the first corrected image and the verification image. Using an edge detection algorithm, the edge contour information of the first corrected image and the verification image is extracted to obtain the edge loss between them. The edge contour information includes the edge contour coordinates of all objects with contours in the image and the grayscale value of the corresponding pixel. The edge loss can be characterized as the difference between the edge contour of an object in the first corrected image and the edge contour of the same object in the verification image. Based on the pixel loss and the edge loss, a first loss function is determined. Specifically, the pixel loss and the edge loss can be weighted to obtain the first loss function. The weighting method can be found in Formula 1.

[0258] In some embodiments, such as Figure 4 As shown, the target correction model 22 includes an encoder 221 and a decoder 222. The target correction model can identify the error locations in the image to be corrected and overlay a visual layer box on the error locations for display to provide users with correct depth information based on the content selected by the visual layer box.

[0259] In specific implementation, such as Figure 5 As shown, the processor 61 uses the target correction model to identify and mark the error locations in the image 203 to be corrected, obtaining the marked location information; it overlays a visualization layer box 204 onto the locations indicated by the marked location information in the image 203 to be corrected, and synthesizes a multi-layer correction image 205, which is then sent to the display 62 for display. The processor 61 responds to the user's selection of the correct depth information (such as...) for the visualization layer box 204. Figure 5 The depth value of the selected position (using the virtual bounding box 206) is used to correct the depth information of the incorrect position in the image to be corrected 203 based on the marked position information and the correct depth information, resulting in a first corrected image 201. The display 62 is also configured to display the received multi-layer corrected image in stereoscopic mode.

[0260] In some embodiments, the processor 61, in response to a user's correction operation on the first correction image, performs supplementary correction on the first correction image to obtain a second correction image. The user's correction operation on the first correction image can be a correction operation performed by the user via voice or facial expression in multimedia mode. For example, a voice instruction to "increase the depth value of target A," or a clear voice instruction to "set the depth of target A to XX," or a facial expression instruction to "enhance the out-of-screen effect," etc. Alternatively, in gesture mode, the user can select a specific area in the first correction image using gestures (including air selection or touch selection), as the area to be corrected. Alternatively, in edit mode, the user can use an external device such as a mouse to click or select a specific area in the first correction image, as the area to be corrected; or actively edit text to generate an adjustment scheme, such as increasing the depth value of target A, or setting the depth value of target A to XX, etc.

[0261] In one possible implementation, the example of a user selecting a multimedia mode is provided. The processor 61 determines the second correction image by: in response to the user selecting a multimedia mode, parsing the received multimedia information from the user and identifying the user's facial features and / or voice features; determining the user's first correction information based on the facial features and / or voice features; and adjusting the depth of the region to be corrected in the first correction image based on the first correction information to obtain the second correction image.

[0262] In another possible implementation, the user's selection of a gesture mode is used as an example. The user's correction operation on the first correction image includes a gesture operation. The processor 61 determines the second correction image, specifically including: in response to the user's selection of a gesture mode, in response to the user's gesture operation on the first correction image, recognizing the user's gesture information; based on the gesture information, determining the user's second correction information; and based on the second correction information, adjusting the depth of the area to be corrected in the first correction image to obtain the second correction image.

[0263] In another possible implementation, the user-selected editing mode is used as an example. The user's correction operation on the first correction image includes an editing operation. The processor 61 determines the second correction image by: in response to the user selecting the editing mode, determining third correction information based on the received editing operation of the user on the first correction image; and adjusting the depth of the area to be corrected in the first correction image based on the third correction information to obtain the second correction image.

[0264] In another possible implementation, the following example illustrates the selection of at least two of the aforementioned multimedia mode, gesture mode, and editing mode. The processor 61 determines the second correction image by: in response to the user selecting a multimedia mode, parsing the received multimedia information from the user and identifying the user's facial features and / or voice features; determining the user's first correction information based on the facial features and / or voice features; in response to the user selecting a gesture mode, identifying the user's gesture information in response to the user's gesture operation on the first correction image; determining the user's second correction information based on the gesture information; in response to the user selecting an editing mode, determining third correction information based on the received editing operation by the user on the first correction image; and adjusting the depth of the region to be corrected in the first correction image based on at least two of the first, second, and third correction information to obtain the second correction image.

[0265] In some embodiments, the image correction system further includes a memory 63; the processor 61 is also configured to send the obtained first correction information, second correction information and third correction information to the memory 63 for storage, for subsequent processing.

[0266] Optionally, the processor 61 is also configured to, in response to an assistance request initiated by the user, call at least one of the first correction information, the second correction information, and the third correction information stored in the memory 63 in the historical phase, generate assistance correction information, and send it to the display 62 for display.

[0267] In some embodiments, if the correction process involves multiple layers or multi-view images, image compositing is required to ensure that depth information from all views is correctly corrected and aligned. For example... Figure 7 As shown, the processor 61 is further configured to acquire multiple viewpoint maps 71 when there are other multiple viewpoint maps 71 in the image 203 to be corrected, and calculate the disparity information between each viewpoint map 71 and the second corrected image 72; based on the disparity information corresponding to each viewpoint map 71, the position and depth information of the pixels of each viewpoint map 71, and the target correction information of the second corrected image 72, determine the fourth correction information corresponding to each viewpoint map 71; and process the corresponding viewpoint map 71 based on the fourth correction information to obtain the corrected viewpoint map 73.

[0268] The target correction information includes at least the correction information adjusted by the user for the image 203 to be corrected, and the correction information adjusted using a preset correction model. The correction information adjusted by the user for the image 203 to be corrected includes first correction information, second correction information, and third correction information. The correction information adjusted using the preset correction model 22, i.e., the information automatically repaired by the preset correction model 22, such as... Figure 2The information shown is related to the automatically repaired part, such as coordinates and depth values.

[0269] In some embodiments, the processor 61 is further configured to, in response to a user's correction operation on the first correction image, generate auxiliary training information for a standard depth estimation big model; adjust the model parameters of the standard depth estimation big model based on the auxiliary training information; and continuously iterate and train the standard depth estimation big model using the first correction image as a training sample for the standard depth estimation big model until the standard depth estimation big model converges.

[0270] In some embodiments, after the standard depth estimation large model is trained according to the above embodiments, the target correction model trained under supervision based on the standard depth estimation large model also needs to be adjusted synchronously.

[0271] Specifically, the processor 61 is further configured to use the first correction image as a training sample for the target correction model, and continuously iterate the training of the target correction model until the second similarity between the output of the target correction model and the verification result of the standard depth estimation large model after training satisfies the preset condition.

[0272] In some embodiments, Figure 12 This is a schematic diagram of another image correction system provided in an embodiment of the present disclosure, as shown below. Figure 12 As shown, the image correction system also includes a cloud server 64. Optionally, the cloud server 64 is deployed with the ffmpeg framework. Optionally, the cloud server 64 has 48 or more physical cores and 96GB or more of memory.

[0273] The processor 61 is also configured to acquire a multimedia file and determine the type of the multimedia file; if the multimedia file is a video, determine whether the resolution of the multimedia file exceeds a first preset resolution, and if the resolution exceeds the first preset resolution, send the multimedia file to the cloud server 64.

[0274] The cloud server 64 is configured to receive a multimedia file, decode the multimedia file using the ffmpeg framework, cut the decoded multimedia file into multiple consecutive image frames, and send the multiple consecutive image frames to the processor 61.

[0275] The processor 61 is also configured to respond to receiving multiple consecutive image frames and send them to the display 62 for display.

[0276] The display 62 is also configured to receive multiple image frames and enable a two-dimensional display mode, allowing the user to select one frame from multiple consecutive image frames as the image to be corrected.

[0277] The processor 61 is also configured to convert the user-selected viewpoint as the original image into an image to be corrected with depth information in response to a viewpoint selected by the user from a plurality of viewpoints in an image frame.

[0278] This embodiment uses FFmpeg for video encoding and decoding. Since FFmpeg also supports GPU acceleration, this significantly improves video encoding and decoding efficiency. The FFmpeg framework is licensed under the LGPL or GPL, making it free to obtain and allowing users to freely use and modify the source code, thereby reducing system operating costs and increasing user control over encoding and decoding. Furthermore, this embodiment utilizes a powerful cloud CPU server (at least 48 cores and 96GB of RAM) to perform software encoding, which is particularly beneficial for high-resolution videos above 16K, optimizing compression efficiency and image quality.

[0279] Based on the previous embodiment, in one possible implementation, the processor 61 is deployed with an ffmpeg framework; the processor 61 is also configured to, when the resolution of the multimedia file does not exceed a first preset resolution, use the ffmpeg framework to decode the multimedia file to obtain multiple consecutive image frames, and send them to the display 62 for display; and, in response to a user selecting a viewpoint from multiple viewpoints in an image frame, convert the user-selected viewpoint as the original image into an image to be corrected with depth information.

[0280] In another possible implementation, processor 61 is further configured to send the multimedia file to cloud server 64 if the resolution of the multimedia file does not exceed a first preset resolution; cloud server 64 is configured to receive the multimedia file, decode the multimedia file using the ffmpeg framework, cut the decoded multimedia file into multiple consecutive image frames, and send the multiple consecutive image frames to processor 61; processor 61 is further configured to send the received multiple consecutive image frames to display 62 for two-dimensional display; display 62 is further configured to display the multiple consecutive image frames; processor 61 is further configured to convert the viewpoint selected by the user from multiple viewpoints in an image frame into an image to be corrected with depth information as the original image in response to the user selecting one viewpoint from multiple viewpoints in an image frame.

[0281] In some embodiments, the processor 61 is further configured to convert the multimedia file into an image to be corrected with depth information when the multimedia file is of image type.

[0282] In some embodiments, Figure 13 This is a schematic diagram of another image correction system provided in an embodiment of the present disclosure, as shown below. Figure 13As shown, the image correction system also includes a graphics card 65; the graphics card 65 is equipped with the ffmpeg framework. The processor 61 is further configured to send the multimedia file to the graphics card 65 if the resolution of the multimedia file does not exceed a first preset resolution. The graphics card 65 is configured to use the ffmpeg framework to decode the multimedia file into multiple consecutive image frames and send them to the display 62 for two-dimensional display. The display 62 is further configured to display the multiple consecutive image frames for the user to select one frame from the multiple consecutive image frames as the image to be corrected. The processor 61 is further configured to, in response to the user selecting a viewpoint from multiple viewpoints in an image frame, convert the user-selected viewpoint as the original image into the image to be corrected with depth information.

[0283] This embodiment integrates hardware encoding into the graphics card 65, enabling direct processing of video data at the hardware level, thereby reducing the burden on the cloud CPU and processor 61 and providing a smoother playback experience.

[0284] The above describes the architecture and functions of each module of the image correction system provided in the embodiments of this disclosure.

[0285] In some embodiments, such as Figures 11-13 As shown, memory 63 stores machine-readable instructions executable by processor 61. When the machine-readable instructions are executed by processor 61, processor 61 performs operations. The operations performed by processor 61 include the following steps: acquiring an image to be corrected, which includes depth information for each pixel; performing initial correction on the image to be corrected using a target correction model to obtain a first corrected image; comparing the first corrected image with a verification image, and if the first similarity between the first corrected image and the verification image meets a preset condition, converting the first corrected image into a stereoscopic image and sending it to display 62 for stereoscopic display; wherein the verification image is an image estimated using a standard depth estimation model with the image to be corrected as input; and, in response to a user's correction operation on the first corrected image, performing supplementary correction on the first corrected image to obtain a second corrected image, and converting the second corrected image into a stereoscopic image and sending it to display 62 for stereoscopic display.

[0286] In some embodiments, such as Figures 11-13 As shown, the image correction system also includes one or more I / O interfaces 66; the one or more I / O interfaces 66 are connected between the processor 61 and the memory 63 and configured to realize information interaction between the processor 61 and the memory 63.

[0287] In some embodiments, processor 61 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); memory 63 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); I / O interface 66 (read-write interface) is connected between processor 61 and memory 63, enabling information interaction between processor 61 and memory 63, including but not limited to a data bus (Bus).

[0288] In some embodiments, the processor 61, memory 63, and I / O interface 66 are interconnected via a data bus, and thus connected to other components of the computing device.

[0289] The above is a complete description of the image correction system provided in the embodiments of this disclosure.

[0290] Fourthly, embodiments of this disclosure also provide a computer non-transient readable storage medium. This computer non-transient readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image correction method as described in the above embodiments.

[0291] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined above in the system of this disclosure.

[0292] It should be noted that the computer-readable non-transient readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any non-transient readable computer storage medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the non-transient readable computer storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0293] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two adjacent blocks may actually represent substantially parallel execution, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0294] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. An image correction method, wherein, include: Obtain the image to be corrected, which includes depth information of each pixel; Using the target correction model, the image to be corrected is initially corrected to obtain the first corrected image; The first corrected image and the verification image are compared. If the first similarity between the first corrected image and the verification image meets a preset condition, the first corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display. The verification image is an image estimated using the image to be corrected as an input sample and a standard depth estimation large model. In response to the user's correction operation on the first correction image, the first correction image is supplemented and corrected to obtain a second correction image, and the second correction image is converted into a stereoscopic image and sent to the display for stereoscopic display.

2. The image correction method according to claim 1, wherein, The comparison of the first corrected image and the verification image includes: Based on the first corrected image and the verification image, a first loss function of the target correction model is determined. If the loss value of the first loss function is less than or equal to a preset threshold, the first similarity is determined to satisfy the preset condition. If the loss value corresponding to the first loss function is greater than the preset threshold, the first similarity is determined not to satisfy the preset condition.

3. The image correction method according to claim 2, wherein, The image correction method further includes: If the first similarity does not meet the preset condition, the target correction model is trained in reverse using the first loss function, with the same image to be corrected as the iterative sample, until the first similarity between the output of the target correction model and the verification image meets the preset condition.

4. The image correction method according to claim 2, wherein, The step of determining the first loss function of the target correction model based on the first corrected image and the verification image includes: Compare the pixel information of corresponding pixels between the first corrected image and the verification image to determine the pixel loss between the first corrected image and the verification image; Using an edge detection algorithm, the edge contour information of the first corrected image and the verification image are extracted respectively to obtain the edge loss between the first corrected image and the verification image; The first loss function is determined based on the pixel loss and edge loss.

5. The image correction method according to claim 1, wherein, The initial correction of the image to be corrected using the target correction model to obtain a first corrected image includes: Using the target correction model, the error locations in the image to be corrected are identified and marked to obtain the marked location information; A visual layer box is overlaid on the position indicated by the marked position information in the image to be corrected to synthesize a multi-layer corrected image, and a stereoscopic image based on the multi-layer corrected image is displayed; In response to the correct depth information selected by the user for the visualization layer box, the depth information of the incorrect position in the image to be corrected is corrected based on the marker position information and the correct depth information to obtain the first corrected image.

6. The image correction method according to claim 1, wherein, The step of responding to a user's correction operation on the first correction image by performing supplementary correction on the first correction image to obtain a second correction image includes: In response to a user selecting a multimedia mode, the received multimedia information from the user is parsed, and the user's facial features and / or voice features are identified. Based on the facial features and / or the voice features, the user's first correction information is determined; Based on the first correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

7. The image correction method according to claim 1, wherein, The step of responding to a user's correction operation on the first correction image by performing supplementary correction on the first correction image to obtain a second correction image includes: In response to the user selecting a gesture mode, the user's gesture information is identified in response to the user's gesture operation on the first correction image; Based on the gesture information, the user's second correction information is determined; Based on the second correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

8. The image correction method according to claim 1, wherein, The step of responding to a user's correction operation on the first correction image by performing supplementary correction on the first correction image to obtain a second correction image includes: In response to the user selecting an editing mode, the third correction information is determined based on the received editing operation by the user on the first correction diagram; Based on the third correction information, the depth of the area to be corrected in the first correction image is adjusted to obtain the second correction image.

9. The image correction method according to claim 1, wherein, The step of responding to a user's correction operation on the first correction image by performing supplementary correction on the first correction image to obtain a second correction image includes: In response to a user selecting a multimedia mode, the received multimedia information of the user is parsed, and the user's facial features and / or voice features are identified; based on the facial features and / or the voice features, the user's first correction information is determined; In response to the user selecting a gesture mode, and in response to the user's gesture operation on the first correction image, the user's gesture information is identified; based on the gesture information, the user's second correction information is determined. In response to the user selecting an editing mode, the third correction information is determined based on the received editing operation by the user on the first correction diagram; Based on at least two of the first correction information, the second correction information, and the third correction information, the depth of the region to be corrected in the first correction image is adjusted to obtain the second correction image.

10. The image correction method according to any one of claims 1 to 9, wherein, Before performing supplementary correction on the first correction image in response to the user's correction operation on the first correction image to obtain the second correction image, the method further includes: In response to a user-initiated assistance request, at least one of the first correction information, second correction information, and third correction information stored in the memory during the historical phase is invoked to generate auxiliary correction information and display it in two dimensions.

11. The image correction method according to any one of claims 1 to 9, wherein, The image correction method further includes: If there are multiple other viewpoint images in the image to be corrected, multiple viewpoint images are acquired, and the disparity information between each viewpoint image and the second corrected image is calculated. Based on the disparity information corresponding to each viewpoint image, the position and depth information of the pixels in each viewpoint image, and the target correction information of the second correction image, fourth correction information corresponding to each viewpoint image is determined; the target correction information includes at least the correction information adjusted by the user for the image to be corrected, and the correction information adjusted using the target correction model. Based on the fourth correction information, the corresponding viewpoint map is processed to obtain a corrected viewpoint map.

12. The image correction method according to any one of claims 1 to 9, wherein, The image correction method further includes: In response to the user's correction operation on the first correction image, auxiliary training information is generated for the standard depth estimation large model; The model parameters of the standard deep estimation large model are adjusted based on the auxiliary training information. The first corrected image is used as a training sample for the standard depth estimation model, and the standard depth estimation model is continuously trained iteratively until it converges.

13. The image correction method according to claim 12, wherein, The image correction method further includes: The first corrected image is used as the training sample of the target correction model, and the target correction model is continuously iterated and trained until the second similarity between the output of the target correction model and the verification result of the standard depth estimation large model after training satisfies the preset condition.

14. The image correction method according to claim 1, wherein, The image correction method further includes: Obtain the multimedia file and determine its type; If the multimedia file is a video, determine whether the resolution of the multimedia file exceeds a first preset resolution; If the resolution exceeds the first preset resolution, the multimedia file is sent to a cloud server. The cloud server is equipped with the ffmpeg framework to decode the multimedia file and cut the decoded multimedia file into multiple consecutive image frames for display. In response to a user selecting one of the viewpoint maps from a plurality of viewpoint maps in an image frame, the user-selected viewpoint map is converted as the original image into the image to be corrected having depth information.

15. The image correction method according to claim 14, wherein, The image correction method further includes: If the resolution of the multimedia file does not exceed the first preset resolution, the ffmpeg framework is used to decode the multimedia file to obtain multiple consecutive image frames, which are then displayed. In response to a user selecting a viewpoint from a plurality of viewpoints in an image frame, the user-selected viewpoint is converted as the original image into the image to be corrected having depth information.

16. The image correction method according to claim 14, wherein, The image correction method further includes: If the multimedia file is an image, the multimedia file is converted into an image to be corrected with depth information.

17. The image correction method according to claim 14, wherein, The image correction method further includes: If the resolution of the multimedia file does not exceed the first preset resolution, the multimedia file is sent to the graphics card; the graphics card is equipped with the ffmpeg framework, so as to use the ffmpeg framework to decode the multimedia file, and cut the decoded multimedia file into multiple consecutive image frames for display; In response to a user selecting a viewpoint from a plurality of viewpoints in an image frame, the user-selected viewpoint is converted as the original image into the image to be corrected having depth information.

18. An image correction device, comprising an image acquisition module, a first correction module, a verification module, and a second correction module; The image acquisition module is configured to acquire an image to be corrected, the image to be corrected including depth information of each pixel; The first correction module is configured to perform initial correction on the image to be corrected using a target correction model to obtain a first corrected image; The verification module is configured to compare the first corrected image and the verification image, and if the first similarity between the first corrected image and the verification image meets a preset condition, convert the first corrected image into a stereoscopic image and send it to the display for stereoscopic display; wherein, The verification image is an image estimated using the image to be corrected as an input sample and a standard depth estimation large model. The second correction module is configured to, in response to a user's correction operation on the first correction image, perform supplementary correction on the first correction image to obtain a second correction image, and convert the second correction image into a stereoscopic image and send it to the display for stereoscopic display.

19. An image correction system comprising a processor and a display; The processor is configured to acquire an image to be corrected, the image including depth information of each pixel; perform initial correction on the image to be corrected using a target correction model to obtain a first corrected image; compare the first corrected image with a verification image, and if the first similarity between the first corrected image and the verification image meets a preset condition, convert the first corrected image into a stereoscopic image and send it to a display for stereoscopic display; wherein... The verification image is an image estimated using a standard depth estimation model, with the image to be corrected as the input sample. In response to the user's correction operation on the first corrected image, the first corrected image is supplemented and corrected to obtain a second corrected image, and the second corrected image is converted into a stereoscopic image and sent to the display for stereoscopic display. The display is configured to receive a stereoscopic image converted from the first corrected image and enable a stereoscopic display mode for stereoscopic display; and to receive a stereoscopic image converted from the second corrected image and enable a stereoscopic display mode for stereoscopic display.

20. The image correction system according to claim 19, wherein, The image correction system also includes a cloud server; the cloud server is deployed with the ffmpeg framework. The processor is further configured to acquire a multimedia file and determine the type of the multimedia file; if the type of the multimedia file is video, determine whether the resolution of the multimedia file exceeds a first preset resolution. If the resolution exceeds the first preset resolution, the multimedia file is sent to the cloud server; And, in response to receiving multiple consecutive image frames, send them to the display; In response to a user selecting one of the viewpoint maps from multiple viewpoint maps in an image frame, the user-selected viewpoint map is converted as the original image into the image to be corrected with depth information. The cloud server is configured to receive the multimedia file, decode the multimedia file using the ffmpeg framework, cut the decoded multimedia file into multiple consecutive image frames, and send the multiple consecutive image frames to the processor. The display is also configured to receive multiple image frames and enable a two-dimensional display mode.

21. The image correction system according to claim 20, wherein, The cloud server has 48 or more physical cores and 96GB or more of memory.

22. The image correction system according to claim 20, wherein, The image correction system also includes a graphics card; the graphics card is deployed with the ffmpeg framework. The processor is also configured to send the multimedia file to the graphics card if the resolution of the multimedia file does not exceed a first preset resolution; And, in response to a user selecting a viewpoint from a plurality of viewpoints in an image frame, the user-selected viewpoint is converted as the original image into the image to be corrected having depth information; The graphics card is configured to use the ffmpeg framework to decode the multimedia file into multiple consecutive image frames and send them to the display for display.

23. A computer-defined non-transient readable storage medium, wherein, The computer non-transient readable storage medium stores a computer program that, when executed by a processor, performs the steps of the image correction method as described in any one of claims 1 to 17.

Images