A method for image restoration and completion

Through the image repair and completion method, the problem of errors in depth data edge identification in structured light and ToF technology is solved. Replacement pixel points through alignment and threshold filtering, the quality and usability of image data are improved, and it is suitable for a variety of visual applications.

CN117541510BActive Publication Date: 2024-11-01WAVUE PHOTONICS (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311634058.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-01
Publication Date
2024-11-01
Estimated Expiration
2043-12-01

AI Technical Summary

Technical Problem

When existing structured light and ToF technologies acquire depth data, the edge identification of objects is incorrect or unclear, especially for some bright black objects and complex materials, resulting in missing or inaccurate depth values, and the edge problems are exacerbated when using multiple depth sensors. .

Method used

By using the image repair and completion method, by obtaining the image data of the target scene, segmenting the object outline data is aligned with the depth value, setting threshold values ​​is set according to the difference between the pixel points and surrounding points, and replacing unqualified pixel points, using the segmentation algorithm and threshold values ​​are adjusted. To improve the quality of image data.

Benefits of technology

It improves the quality of image data, reduces noise and distortion, ensures that the object profile is consistent with depth information, enhances the usability and credibility of images, and is suitable for computer vision, virtual reality and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117541510B_ABST
    Figure CN117541510B_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of image data processing, and specifically discloses an image restoration and completion method, comprising: acquiring image data of a target scene; segmenting the image data to obtain contour data of each object in the target scene on the image data; aligning all contour data with depth values ​​on the image data where they are located; for each pixel point on the image data, judging whether the current pixel point is qualified according to the difference relationship between the current pixel point and surrounding pixels; replacing all unqualified pixels according to the association relationship between unqualified pixels and pixels on the image data where they are located; and having the following advantages: the segmentation algorithm is trained based on a large number of data sets, and maintains good robustness and generalization for different image segmentation effects, and at the same time, the thresholds of the filtering and completion algorithms can be adjusted, so that the algorithm has a wider range of application scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to an image restoration and completion method. Background Art

[0002] Structured light technology uses a projector and a depth sensor to obtain depth information, and ToF (Time of flight) technology uses the propagation time of light to measure the distance from the object to the sensor. When existing structured light and ToF obtain depth data, there are recognition errors or unclear edges of objects, and some special materials such as some bright black objects cannot be recognized in some areas. When multiple depth sensors are used at the same time, the edge problems will be further magnified after projection, such as errors at the edges of complex objects and missing or inaccurate depth values ​​of some objects (such as black objects).

[0003] Therefore, an image restoration and completion method is proposed to solve the above-mentioned problems. Summary of the invention

[0004] The present invention aims to provide an image restoration and completion method to solve or improve at least one of the above-mentioned technical problems.

[0005] In view of this, a first aspect of the present invention is to provide an image restoration and completion method.

[0006] A second aspect of the present invention is to provide a repair completion system.

[0007] A third aspect of the present invention is to provide an electronic device.

[0008] A fourth aspect of the present invention is to provide a computer-readable storage medium.

[0009] The first aspect of the present invention provides an image restoration and completion method, comprising the following steps: acquiring image data of a target scene; segmenting the image data to obtain contour data of each object in the target scene in the image data; aligning all the contour data with their depth values ​​on the image data; acquiring all pixels of the aligned image data in the current frame; judging whether the current pixel is qualified based on the difference relationship between the current pixel and its surrounding pixels; and replacing all unqualified pixels based on the association relationship between the unqualified pixels and the pixels on the image data where they are located.

[0010] Further, the difference relationship includes a first difference relationship and a second difference relationship, and the step of judging whether the current pixel point is qualified based on the difference relationship between the current pixel point and the surrounding pixel points specifically includes: setting a first threshold, taking the difference between the current pixel point and the pixel points in the contour data as the first difference relationship, obtaining the first difference relationship and judging whether the current pixel point is qualified through the first threshold; setting a second threshold, taking the difference between the current pixel point and the adjacent pixel points along a certain direction as the second difference relationship, obtaining the second difference relationship and judging whether the current pixel point is qualified through the second threshold.

[0011] Furthermore, the step of obtaining the first difference relationship and judging whether the current pixel point is qualified through the first threshold specifically includes: obtaining the depth value of the current pixel point and the depth values ​​of all pixels in the contour data where the current pixel point is located; calculating the overall depth value of the contour data according to the depth values ​​of all pixels in the contour data where the current pixel point is located; calculating the first difference between the overall depth value and the depth value of the current pixel point; judging whether the first difference is greater than the first threshold, and if so, judging the current pixel point as unqualified, and otherwise, judging the current pixel point as qualified.

[0012] Furthermore, the step of obtaining the second difference relationship and judging whether the current pixel point is qualified by the second threshold value specifically includes: obtaining the horizontal and vertical directions of the image data where the current pixel point is located; taking the current pixel point as the center, obtaining the adjacent pixel points of the current pixel point along the horizontal and vertical directions respectively; obtaining the first pixel value of the current pixel point and the second pixel value of the adjacent pixel point respectively, and calculating the second difference between the first pixel value and the second pixel value; judging whether the second difference is greater than the second threshold value, and if so, judging that the current pixel point is unqualified, and otherwise, judging that the current pixel point is qualified.

[0013] Furthermore, the second threshold is used to determine the pixel points that are determined to be qualified by the first threshold; wherein the values ​​of the first threshold and the second threshold are both set according to the type of the target scene.

[0014] Furthermore, the image data is segmented by a segmentation algorithm, and before the step of segmenting the image data, it also includes: obtaining a training set containing image data through a database; defining the type of the target scene, and dividing the training set according to the type of the target scene to obtain multiple training subsets; training the segmentation algorithm respectively through the multiple training subsets, and setting the values ​​of the first threshold and the second threshold for each training subset respectively.

[0015] Furthermore, the step of replacing all the unqualified pixel points specifically includes: traversing the unqualified pixel points on each contour data; for the current unqualified pixel point, obtaining the pixel point closest to it in the contour data; and using the depth value of the nearest pixel point as the depth value of the current unqualified pixel point to complete the replacement.

[0016] The second aspect of the present invention is to provide a repair and completion system, an image input module, which is used to input image data acquired by a sensor; an image segmentation module, which is used to divide the image data into multiple contour data according to the objects of the target scene corresponding to the image data; a data preprocessing module, which is used to align the contour data and the depth value, remove the pixels that are noise and filter outliers in the contour data, and perform filtering operations and correct the coordinate system of the data; an image completion module, which is used to complete the removed pixels according to the association relationship; a repair and optimization module, which is used to adjust the contour data, including operations such as removing artifacts, smoothing images, correcting alignment errors, color correction, and removing duplicate pixels; an image output module, which generates image data according to the adjusted contour data and outputs it; a user interface editing module, which includes a user interface, which is used to display and edit image data, adjust parameters, and monitor the processing progress in real time; a parameter setting and automation module, which is used to adjust the parameters of the image data processing to adjust the image processing process according to the type of the target scene, and adjust the processing parameters according to the matching relationship between the preset parameters and the type of the target scene.

[0017] A third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the above method when executed by a processor.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] It mainly repairs the problems existing in the depth map data obtained by the current structured light and ToF tests, such as the lack of some depth values ​​and poor edge contours of objects. By screening and replacing unqualified pixels, the quality of image data can be improved, and noise and distortion can be reduced; Accurate contour data: Depth value alignment ensures that the contour data of the object is consistent with the depth information, which helps to accurately capture the shape and position of the object; repairing unqualified pixels can increase the usability of the image, making it more suitable for analysis, recognition and visual applications; it can be used in applications that require accurate object contour data and depth information, such as computer vision, image analysis, virtual reality and augmented reality. Through depth value alignment and pixel screening, the usability and credibility of image data can be enhanced.

[0021] The segmentation algorithm is trained based on a large number of data sets, maintaining good robustness and generalization for different image segmentation effects. At the same time, the thresholds of the filtering and completion algorithms are adjustable, making the algorithm have a wider range of application scenarios.

[0022] Additional aspects and advantages of embodiments according to the present invention will become apparent in the following description or may be learned through practice of embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0024] Figure 1 is a flow chart of the method steps of the present invention;

[0025] Figure 2 It is a schematic diagram of image segmentation of the present invention;

[0026] Figure 3 It is a structural block diagram of the repair and completion system of the present invention;

[0027] Figure 4 The figure is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0028] See also Figure 1-4 , an image restoration and completion method according to some embodiments of the present invention is described below.

[0029] As described in the background technology, structured light technology uses a projector and a depth sensor to obtain depth information. Its working principle is as follows: Projector: The projector is usually a laser light source or LED emitter, which emits a special light pattern or spot (usually a grid or stripe pattern). Depth sensor: The depth sensor is usually a camera that is used to capture the structured light pattern projected onto the surface of an object. Calculating depth: When the structured light pattern is irradiated onto the surface of an object, the shape and surface features of the object will distort these light patterns. The depth sensor captures these distorted patterns and calculates the depth information of the object by analyzing the changes in the pattern. Usually, calculating depth requires the use of triangulation or other complex algorithms to measure the displacement or deformation of the light spot. The advantages of structured light technology include high accuracy and applicability to a variety of environments and materials, but there are also some limitations, such as poor performance in strong light or transparent objects.

[0030] ToF technology uses the propagation time of light to measure the distance of an object from the sensor. It works as follows: Transmit a light pulse: The ToF sensor transmits a very short light pulse, usually laser or infrared light. The pulsed light shines on the target object. Receive reflected light: Once the light pulse hits the surface of the object and reflects back to the sensor, the sensor records the return time of the light pulse. Calculate depth: The ToF sensor calculates the distance of the object based on the speed of light propagation (usually the speed of light) and the round-trip time of the light pulse. The depth is equal to the speed of light multiplied by half the round-trip time. The advantages of ToF technology include fast speed, adaptability to different lighting conditions and applicability to a variety of materials. It has a wide range of uses in many applications, such as virtual reality, self-driving cars and robot navigation.

[0031] The depth data obtained by structured light and ToF may have incorrect or unclear recognition of the edges of objects, and some special materials, such as some bright black objects, may not be recognized in some areas. When multiple depth sensors are used at the same time, the edge problem will be further magnified after projection.

[0032] In view of the above-mentioned technical problems, the embodiments of the first aspect of the present invention provide an image restoration and completion method. In some embodiments of the present invention, Figure 1-2 As shown, the method includes:

[0033] S101, acquiring image data of a target scene.

[0034] Here, the target scene refers to the real scene, which has a diagram in the original document and can be a scene captured by a camera of a sweeping robot.

[0035] Specifically, color images are acquired through structured light or ToF.

[0036] Here, RGB images and depth maps are used. RGB is the color image captured by an ordinary camera, and the depth map and RGB image have a one-to-one coordinate correspondence.

[0037] Among them, structured light uses emission structured light: using a light source to emit structured light, usually generated by a grating or projector. This structured light can be a spot or stripe with a specific pattern; irradiation target: the structured light is irradiated on the target object for which depth information is to be obtained; camera capture image: using a color camera to capture the image of the structured light and ambient light reflected by the target object. The image usually includes color information; structured light decoding: analyzing and decoding the captured image in a computer to identify the changes and shapes of the structured light, thereby obtaining depth information; depth information superposition: combining depth information and color images to generate an image with color and depth information.

[0038] ToF acquires color images: Emitting time-of-flight pulses: Using the built-in light source of the ToF camera to emit light pulses; Light pulse propagation: After a period of time, the emitted light pulses are reflected back to the ToF camera. Since light is blocked and reflected by objects during propagation, each pixel records the flight time of light; Depth map generation: By measuring the flight time of light, the ToF camera can calculate the distance from each pixel to the object, thereby generating a depth image; Capturing color images: Using a color camera at the same time to capture a color image of the target object; Depth and color image superposition: Combining depth information and color images to generate an image with color and depth information.

[0039] Here, the image restoration and completion method further includes: obtaining a training set containing image data through a database.

[0040] The target scene is classified into a certain type, and the training set is divided according to the type of the target scene to obtain a plurality of training subsets.

[0041] The segmentation algorithm is trained respectively through a plurality of the training subsets, and the values ​​of the first threshold and the second threshold are set for each of the training subsets.

[0042] As can be seen from the above, a training set containing image data is collected from the database. These images can include various types of target scenes to represent multiple categories of interest; the target scene category to which each image belongs is defined and marked. This will become the basis for dividing the training set; the entire training set is divided into multiple subsets, each corresponding to a target scene category. For example, a subset can be created for indoor scenes, another subset for outdoor scenes, and so on; an appropriate segmentation algorithm is selected for each training subset. Different scenarios may require different algorithms. Then, each algorithm is trained using the corresponding training subset; for each training subset, appropriate values ​​of the first threshold and the second threshold are set based on experiments and performance evaluation. These thresholds are usually related to specific parameters of the algorithm or segmentation tasks; the performance of the algorithm after training each training subset is evaluated. This may involve using a validation data set or cross-validation to measure performance indicators such as accuracy, recall, F1 score, etc. of the algorithm; compare algorithms trained on different training subsets and select the model or algorithm with the best performance. This may involve weighing the performance requirements of different scene categories.

[0043] In summary, by dividing the target scenes into different categories, the characteristics of different scenarios can be taken into account, which helps the algorithm better adapt to various situations; setting different thresholds can optimize the performance of the algorithm according to different scenario requirements, so that it can perform well in each scenario; the thresholds of different scenarios can be adjusted as needed to adapt to different application scenarios and improve the flexibility of the algorithm; it is easier to evaluate the performance of the algorithm in different scenarios, so as to make more informed decisions. Dividing the training set according to different target scene types, and training and setting thresholds for each subset separately, can make the segmentation algorithm better adapt to diverse scenarios, thereby improving its performance and applicability.

[0044] S102, segmenting the image data to obtain contour data of each object in the target scene in the image data.

[0045] Specifically, the segmentation algorithm is first used to segment the color image obtained by structured light or ToF to segment the individual contour pixel coordinates of each object. The contour data is the area segmented by the segmentation algorithm. For example, if there is a stool in the image, the segmentation algorithm will segment all the pixels of the stool as a contour, that is, the contour of the stool. Subsequent filtering and completion are processed in this contour, and all contours are processed in sequence.

[0046] As can be seen above, first, color images are acquired using structured light or ToF technology. These images may include depth information about objects in the scene, which is usually presented in the form of grayscale values ​​or depth images; at the initial stage of the processing flow, the color image is processed using a segmentation algorithm to segment the contours of each object. The task of the segmentation algorithm is to divide the pixels in the image into different objects or object contour areas. This is usually based on the similarity and connectivity of the pixel values ​​so that pixels of the same object are assigned to the same area; once the segmentation algorithm is completed, the contour area of ​​each object will be marked and assigned. Then, the pixel coordinates of the contour are extracted from the contour area of ​​each object. These coordinates represent the location of the object's contour in the image; for each segmented object contour, the next step is to align the contour pixel coordinates with the detected depth value. This means ensuring that each pixel on the contour has a corresponding depth value. This can be achieved in the following ways; if the depth information is provided in the form of a depth image, then the depth image needs to be registered with the color image to ensure that the depth information is aligned with the object contour in the color image; the depth value is mapped to the pixel coordinates of the color image to obtain the depth value corresponding to the contour pixel coordinates. This usually requires some geometric and projective transformations; the final result is that the pixel coordinates of each object outline are aligned with their corresponding depth values. This means that you can know the depth information of each pixel on each object outline, which is very important for subsequent 3D reconstruction, object recognition and depth analysis.

[0047] In summary, processing in the segmentation stage is a key step in fields such as depth perception, computer vision, and 3D reconstruction, which helps to extract the 3D information of objects for more accurate scene understanding and analysis.

[0048] Furthermore, the specific steps of the segmentation algorithm are: input the image into the encoder, encode the embedded vector data, then train it by combining the segmentation label and the box coordinates, and finally output the segmentation result. The algorithm is trained on a large amount of data and maintains good robustness and generalization in different scenarios.

[0049] As can be seen above, the input of the algorithm is a color image. These images may contain different objects or objects that need to be segmented in order to separate or mark them out; first, the image is fed into the encoder network, which converts the image into embedded vector data. The encoder is usually a deep neural network that can capture high-level features and semantic information in the image; in order to train the model, labeled data is required. These labels usually include segmentation labels, which are used to indicate the object or category to which each pixel in the image belongs, and box coordinates, which are used to identify the location and bounding box of each object; in the training phase, the model uses labeled data to learn how to perform image segmentation by comparing its output and the true label. This is achieved through optimization techniques such as backpropagation and gradient descent. The goal of the training process is to minimize the segmentation error of the model so that it can accurately segment the image; in order to maintain robustness and generalization in different scenarios, the model needs to be trained in a large number of different data sets and scenarios. This includes different lighting conditions, angles, object types, etc. The diversity of training data helps the model better adapt to various situations; after training, the model can be applied to new images to achieve image segmentation. The model will generate segmentation results that separate different objects in the image or assign corresponding category labels to each pixel.

[0050] In summary, style algorithms are often used for image segmentation tasks, such as semantic segmentation or instance segmentation, where the goal is to separate and label different objects or categories in an image. Through large-scale training and diverse data, the model can learn to generalize to different scenarios, thereby showing robustness in practical applications and being able to handle different situations and image conditions. This is very important for application areas such as computer vision, autonomous driving, and object recognition.

[0051] Furthermore, for each input image, the segmentation algorithm detects the contour information of each object in the image, labels each contour, and obtains the coordinate position of each contour in the image.

[0052] As can be seen above, the input of the algorithm is a color or grayscale image, usually containing one or more objects that need to be segmented and labeled; first, the algorithm will apply image processing and computer vision techniques to detect the outline of each object in the image. This usually includes the following steps:

[0053] Edges in an image usually represent the outline of an object. The algorithm uses edge detection techniques (such as Sobel, Canny operator, etc.) to find edges in an image.

[0054] After finding the edge, the algorithm analyzes the connectivity between pixels to combine adjacent edge pixels into a complete contour.

[0055] Once the connectivity analysis is completed, the algorithm obtains the outline information of each object, which is represented by a sequence of pixel coordinates.

[0056] For each detected contour, the algorithm assigns it an appropriate label. Labels are usually integer values, with each different label corresponding to a different object or region. This can be achieved through different methods, such as region growing, region segmentation, or by assigning it based on the contour's properties.

[0057] For each detected contour, the algorithm also obtains its coordinate position in the image. This usually includes information such as the geometric center of the contour, the coordinates of the bounding box, the pixel coordinates of the contour, etc.

[0058] Finally, the algorithm outputs a data structure containing the coordinate information of each object's outline and the corresponding label. The data structure can be used for further image processing, object recognition, tracking, measurement and other tasks.

[0059] In summary, this image segmentation algorithm is widely used in computer vision, image processing, autonomous driving, medical image analysis and other fields. It is used to extract information about different objects or areas from images for subsequent analysis and decision-making.

[0060] S103: Align all the contour data with their depth values ​​on the image data.

[0061] Specifically, first, it is necessary to obtain the contour data in the image. This can be achieved through image segmentation algorithms, edge detection or other related technologies. The contour data usually includes the boundary coordinates or pixel positions of the object; it is necessary to obtain depth data associated with the image. It can be obtained through structured light, ToF or other depth sensors. The depth data represents the distance or depth value from each pixel to the surface of the object; align the coordinates of the contour data with the coordinates of the depth data to ensure that they refer to the same object. This usually involves coordinate transformation and registration operations to ensure the spatial alignment of the contour data and the depth data; after alignment, the contour data can be associated with the depth data, and each contour point or boundary can be associated with its corresponding depth value in the depth image; using the associated depth value, the corresponding three-dimensional coordinates can be created for each contour point or boundary point, thereby obtaining the three-dimensional information of the object. This enables the geometric shape, size and position of the object to be understood.

[0062] As can be seen above, aligning and associating contour data with depth data enables obtaining 3D information of an object, including its shape and depth information; combining depth information can enhance image segmentation and object recognition tasks. It can help identify the spatial relationship between multiple objects and help segment objects more accurately; 3D information can be used for 3D visualization, such as rendering a 3D model of an object to better understand its appearance and position; aligning contour data and depth data is critical for many machine vision applications, such as target tracking, robot navigation, virtual reality, etc.

[0063] In summary, aligning contour data with depth data is an important operation that expands the information obtained from images and provides 3D geometric information of objects, which is of great significance for many vision and computer vision applications.

[0064] S104, obtaining all pixels of the aligned image data in the current frame; judging whether the current pixel is qualified according to the difference relationship between the current pixel and its surrounding pixels.

[0065] Specifically, the difference relationship includes a first difference relationship and a second difference relationship, and the step of judging whether the current pixel point is qualified according to the difference relationship between the current pixel point and the surrounding pixel points specifically includes:

[0066] A first threshold is set, and the difference between the current pixel point and the pixel points in the contour data is taken as a first difference relationship. The first difference relationship is obtained and whether the current pixel point is qualified is determined by the first threshold.

[0067] A second threshold is set, and a difference between the current pixel point and the adjacent pixel point along a certain direction is taken as a second difference relationship. The second difference relationship is obtained and whether the current pixel point is qualified is determined by the second threshold.

[0068] Here, first of all, for the first threshold, a first threshold needs to be set, and this threshold will be used to determine whether the pixel is qualified. The selection of the threshold is usually based on the specific application requirements and image characteristics; for each pixel, calculate the difference between it and the pixel in the contour data. This can be the difference between pixel values, color differences, depth differences, etc., depending on the application; the calculated difference value is used as the first difference relationship; the first difference relationship is used to compare with the set first threshold. If the first difference relationship is less than the first threshold, the current pixel is judged to be qualified; otherwise, it is judged to be unqualified.

[0069] First, for the second threshold, a second threshold needs to be set, which is used to determine what degree of difference is considered unqualified. The choice of threshold is usually based on specific application requirements and image characteristics; adjacent pixels are selected along a certain direction, which can be horizontal, vertical or diagonal, etc. The selected direction depends on the needs of the application; for each pixel, calculate its difference with the adjacent pixels along the selected direction. This can be the difference between pixel values, color differences, depth differences, etc., depending on the application; the calculated difference value is used as the second difference relationship; the second difference relationship is used to compare with the set second threshold. If the second difference relationship is less than the second threshold, the current pixel is judged to be qualified; otherwise, it is judged to be unqualified.

[0070] As can be seen from the above, by setting the first threshold, those pixels that differ too much from the pixels in the contour data can be filtered out, thereby filtering out the required pixels; adjusting the threshold can make this method adaptable according to different applications or image conditions. The threshold can be changed according to actual needs to meet different standards; the difference relationship and threshold can help reduce the noise in the image, especially when the difference caused by noise is small; this method helps to select areas or objects of interest and ignore irrelevant pixels for subsequent analysis and processing; filtering unqualified pixels can improve the quality of the image and make it more in line with the requirements of the application.

[0071] By setting the second threshold and calculating the difference between adjacent pixels, it can be used to detect features such as edges, textures, lines, etc. in the image, which helps to analyze the image; by comparing the difference between a pixel and its adjacent pixels, it can help reduce the noise in the image, especially when the difference caused by the noise is small; this method helps to select areas or objects of interest and ignore irrelevant pixels for subsequent analysis and processing; by adjusting the second threshold, the strictness of the eligibility judgment can be controlled to adapt to different application requirements; this method helps to improve the data quality of image processing and segmentation applications, especially when high-quality image information is required.

[0072] In summary, by setting a threshold and comparing the difference relationship between pixels, and by setting a threshold and comparing the difference between pixels and adjacent pixels, the eligibility of pixels can be determined, which helps to improve the accuracy and controllability of applications such as image processing, segmentation and analysis.

[0073] S1041, obtaining the depth value of the current pixel and the depth values ​​of all pixels in the contour data where the current pixel is located.

[0074] The overall depth value of the contour data is calculated based on the depth values ​​of all pixels in the contour data where the current pixel is located.

[0075] A first difference between the overall depth value and the current depth value of the pixel is calculated.

[0076] It is determined whether the first difference is greater than the first threshold value, if so, the current pixel point is determined to be unqualified, otherwise, the current pixel point is determined to be qualified.

[0077] Here, get the depth value of the current pixel: first, you need to get the depth value of the current pixel, usually from a depth sensor or depth image; get the depth values ​​of all pixels in the contour data where the current pixel is located: access all adjacent pixels in the contour data where the current pixel is located, and get their depth values; count the depth values ​​of all pixels in the contour, and calculate the overall depth value of the contour data, such as the average depth or median depth; subtract the depth value of the current pixel from the overall depth value to get the first difference; set a first threshold, which is used to determine what degree of difference is considered unqualified. The choice of threshold is usually based on specific application requirements and image characteristics; if the first difference is greater than the set first threshold, then the current pixel is judged to be unqualified; otherwise, it is judged to be qualified.

[0078] From the above, it can be seen that by calculating the difference between the overall depth value and the pixel depth value, the distribution of depth data within the entire contour can be considered, rather than just the depth of a single pixel; adjusting the first threshold can make this method adaptable according to different application requirements and image conditions.

[0079] S1042, obtaining the horizontal and vertical directions of the image data where the current pixel point is located.

[0080] Taking the current pixel point as the center, adjacent pixel points of the current pixel point are obtained along the horizontal direction and the vertical direction respectively.

[0081] The first pixel value of the current pixel point and the second pixel value of the adjacent pixel point are respectively obtained, and a second difference between the first pixel value and the second pixel value is calculated and obtained.

[0082] Determine whether the second difference is greater than the second threshold value, if so, determine that the current pixel point is unqualified, otherwise, determine that the current pixel point is qualified.

[0083] Here, first, it is necessary to determine the horizontal and vertical directions, which usually refer to the horizontal and vertical directions in the image. This will be used to determine the position of adjacent pixels; with the current pixel as the center, adjacent pixels are obtained in the horizontal and vertical directions respectively. These adjacent pixels are usually pixels around the current pixel; for the current pixel, its pixel value (first pixel value) is obtained, and the pixel value of the adjacent pixel (second pixel value) is obtained. Then, the difference between the first pixel value and the second pixel value is calculated to obtain the second difference; a second threshold is set, which is used to determine what degree of difference is considered unqualified. The selection of the threshold is usually based on specific application requirements and image characteristics; the second difference is compared with the set second threshold. If the second difference is greater than the set second threshold, the current pixel is judged to be unqualified; otherwise, it is judged to be qualified, and the pixel value is the image value; for multiple adjacent pixels of a single pixel, when the obtained second difference is greater than the second threshold, the pixel value is newly grouped, and finally sorted according to the number of each, retaining the group with the largest number, and filtering out the other groups.

[0084] It can be seen from the above that by adjusting the second threshold, the strictness of the qualification determination can be controlled to adapt to different application requirements.

[0085] Furthermore, the second threshold is used to determine the pixel points that are determined to be qualified by the first threshold.

[0086] The values ​​of the first threshold and the second threshold are both set according to the type of the target scene.

[0087] Here, first, it is necessary to set the first threshold value according to the type of target scene. Different scenes may require different threshold values ​​to meet specific application requirements; the first threshold value is used to determine whether the pixel is qualified. By comparing the first difference with the first threshold value, the pixels that are qualified by the first threshold value are screened out; then the second threshold value is set to further screen the pixels that are qualified by the first threshold value. The setting of the second threshold value is usually also based on the type of target scene; the second threshold value is used to determine which pixels that are qualified by the first threshold value also meet the conditions of the second threshold value.

[0088] From the above, it can be seen that by setting different thresholds according to different target scene types, the eligibility judgment standard can be customized according to the characteristics of the scene; the use of the first threshold and the second threshold allows for more refined screening and control of pixels to meet the needs of specific applications; the use of the first threshold can help preliminarily screen out qualified pixels, thereby reducing the amount of calculation for subsequent processing and improving algorithm performance; different scenes and applications may require different threshold settings, so this method has good adaptability and is suitable for a variety of different situations; through multi-level threshold judgment, the accuracy of the final selected pixel points can be improved and errors can be reduced.

[0089] In summary, by setting the first and second thresholds according to the type of the target scene, multi-level screening of pixels can be achieved, which helps to improve the accuracy and performance of image processing and segmentation applications.

[0090] S105: Replace all the unqualified pixels according to the association relationship between the unqualified pixels and the pixels on the image data to which they belong.

[0091] Specifically, unqualified pixel points are traversed on each contour data.

[0092] For the currently unqualified pixel point, obtain the pixel point closest to it in the contour data.

[0093] The depth value of the nearest pixel is used as the depth value of the currently unqualified pixel to complete the replacement.

[0094] Here, unqualified pixels are extracted from the contour data, which may be inaccurate due to errors in the depth sensor or other factors; for each unqualified pixel, its nearest neighbor pixel in the contour data is found. Usually, a distance metric such as Euclidean distance or Manhattan distance is used to determine the nearest pixel; the depth value of the nearest neighbor pixel is obtained. This is usually the original depth value of the pixel; the depth value of the nearest neighbor pixel is used as the depth value of the current unqualified pixel, thereby completing the replacement; repeat the above steps until all unqualified pixels are replaced.

[0095] As can be seen from the above, this method allows repairing unqualified pixels in depth images, thereby improving the quality of depth data. This is very important for applications that require accurate depth information, such as 3D reconstruction and virtual reality; replacing unqualified pixels can reduce noise in depth images, making depth data smoother and more reliable; repairing unqualified pixels helps improve the visual effect of depth images, making them more attractive and clear; the method helps to fill in missing data in depth images and enhance the integrity of the data; repairing depth data can improve the accuracy of subsequent computer vision and machine learning tasks, such as object detection, segmentation and recognition.

[0096] In summary, by replacing the depth values ​​of unqualified pixels with the depth values ​​of their nearest neighbors, the quality and accuracy of the depth image can be improved, thereby providing more reliable depth information for various visual applications.

[0097] The image restoration and completion method provided by the present invention obtains image data: first, it is necessary to obtain the image data of the target scene, which can be collected using a camera or a sensor; image segmentation: using image segmentation technology, such as semantic segmentation or edge detection, to segment different objects or regions in the image to obtain the contour data of each object; depth value alignment: obtaining depth information to ensure that the contour data of each object corresponds to its depth value on the image. This can be achieved through methods such as depth sensors or structured light scanning; pixel point screening: analyzing each pixel point on the image data, and judging whether the current pixel point is qualified based on the difference relationship between the current pixel point and the surrounding pixels. This can include differences in color, brightness, texture, etc.; unqualified pixel point replacement: for pixels that are judged to be unqualified, the pixel values ​​are replaced or repaired based on their positions and associations in the image. This can be achieved using interpolation, patching or other image processing techniques.

[0098] By screening and replacing unqualified pixels, the quality of image data can be improved and noise and distortion can be reduced.

[0099] Depth value alignment ensures that the object's outline data is consistent with the depth information, helping to accurately capture the object's shape and position.

[0100] Enhance image usability: Repairing bad pixels can increase the usability of images, making them more suitable for analysis, recognition, and vision applications.

[0101] It can be used in applications that require accurate object contour data and depth information, such as computer vision, image analysis, virtual reality, and augmented reality. Through depth value alignment and pixel point screening, the availability and credibility of image data can be enhanced.

[0102] Furthermore, the first threshold of the filtering algorithm is between 0 and 0.4, and the second threshold of the completion algorithm is between 5 and 20. The smaller the value, the stronger the filtering and completion capabilities.

[0103] Embodiments of the second aspect of the present invention provide a repair and completion system. In some embodiments of the present invention, such as Figure 3 As shown, the system includes:

[0104] The image input module 201 is used to input image data acquired by the sensor.

[0105] The image segmentation module 203 is used to divide the image data into a plurality of contour data according to the object of the target scene corresponding to the image data.

[0106] The data preprocessing module 202 is used to align the contour data and the depth value, remove the pixels that are noise and outliers in the contour data, perform filtering operations and correct the coordinate system of the data.

[0107] An image completion module 204, used to complete the removed pixel points according to the association relationship;

[0108] The repair and optimization module 205 is used to adjust the contour data, including removing artifacts, smoothing images, correcting alignment errors, color correction, and removing duplicate pixels.

[0109] The image output module 206 generates and outputs image data according to the adjusted contour data.

[0110] The user interface editing module 207 includes a user interface for displaying and editing image data, adjusting parameters, and monitoring the processing progress in real time.

[0111] The parameter setting and automation module 208 is used to adjust the parameters of the image data processing to adjust the image processing process according to the type of the target scene, and adjust the processing parameters according to the matching relationship between the preset parameters and the type of the target scene.

[0112] The image input module of the image restoration and completion system provided by the present invention is responsible for receiving image data obtained from the sensor, which usually contains depth information. This allows the system to start processing data from the real world; the image data is divided into multiple contour data according to the objects in the image data. This helps the system to better understand different objects in the scene and provide object-level information for subsequent processing; the contour data is aligned with the depth value, noise is removed and outliers are filtered, filtering operations are performed to smooth the data, and the coordinate system of the data is corrected. This helps to improve the quality and accuracy of the image data and reduce errors and noise; the deleted pixels are completed according to the association relationship. This helps to restore the removed data and make the image more complete; the contour data is further processed, including operations such as removing artifacts, smoothing the image, correcting alignment errors, color correction, and removing duplicate pixels. This helps to further improve the quality and fidelity of the image data; image data is generated and output according to the adjusted contour data. This means that the system can provide the final result required by the user, such as generating a processed image or reconstructing a scene image; and a user interface is provided to allow the user to display and edit the image data, adjust parameters, and monitor the processing progress in real time. This provides a user-friendly way to interact with the system and control the processing; responsible for adjusting the parameters of image data processing according to the type of target scene. Through automation, the system can adjust parameters according to the characteristics of different scenes, thereby improving the efficiency and accuracy of processing.

[0113] Improve image data quality: Through operations such as data preprocessing, completion, repair and optimization, the system can improve the quality and accuracy of image data and reduce noise and errors.

[0114] Improved automation: The system’s parameter settings and automation modules allow for automatic adjustment of processing parameters based on the type of target scenario, thus reducing manual intervention by the user.

[0115] More precise image data: Through correction and processing, the system can provide more accurate, precise and realistic image data, which helps various applications such as virtual reality, 3D modeling and machine vision.

[0116] User-Friendly: The user interface editing module makes the system easier to use and control, enabling users to monitor and adjust the processing process.

Claims

1. A method for image restoration and completion, It is characterized in that The steps include: Acquire image data of the target scene; Segmenting the image data to obtain contour data of each object in the target scene in the image data; Aligning all the contour data with their depth values ​​on the image data; Acquire all pixels of the aligned image data in the current frame; the difference relationship between the current pixel and the surrounding pixels includes a first difference relationship and a second difference relationship; set a first threshold, take the difference between the current pixel and the pixels in the contour data as the first difference relationship, acquire the first difference relationship, and judge whether the current pixel is qualified by using the first threshold; set a second threshold, take the difference between the current pixel and the adjacent pixels along a certain direction as the second difference relationship, acquire the second difference relationship, and judge whether the current pixel is qualified by using the second threshold; All the unqualified pixel points are replaced according to the association relationship between the unqualified pixel points and the pixel points on the image data to which they belong.

2. The image restoration method according to claim 1, It is characterized in that The step of obtaining the first difference relationship and judging whether the current pixel point is qualified by using the first threshold specifically includes: Obtaining the depth value of the current pixel and the depth values ​​of all pixels in the contour data where the current pixel is located; Calculate the overall depth value of the contour data according to the depth values ​​of all pixels in the contour data where the current pixel is located; Calculate a first difference between the overall depth value and the depth value of the current pixel point; It is determined whether the first difference is greater than the first threshold value, if so, the current pixel point is determined to be unqualified, otherwise, the current pixel point is determined to be qualified.

3. The image restoration method according to claim 1, It is characterized in that The step of obtaining the second difference relationship and judging whether the current pixel point is qualified by using the second threshold specifically includes: Obtain the horizontal and vertical directions of the image data where the current pixel point is located; Taking the current pixel point as the center, respectively obtain the adjacent pixel points of the current pixel point along the horizontal direction and the vertical direction; Obtaining respectively a first pixel value of the current pixel point and a second pixel value of an adjacent pixel point, and calculating and obtaining a second difference between the first pixel value and the second pixel value; Determine whether the second difference is greater than the second threshold value, if so, determine that the current pixel point is unqualified, otherwise, determine that the current pixel point is qualified.

4. The image restoration method according to claim 1, It is characterized in that The second threshold is used to determine the pixel points that are determined to be qualified by the first threshold; The values ​​of the first threshold and the second threshold are both set according to the type of the target scene.

5. The image restoration method according to claim 4, It is characterized in that The image data is segmented by a segmentation algorithm, and before the step of segmenting the image data, the step further includes: Obtain a training set containing image data from a database; Defining the type of the target scene, and dividing the training set according to the type of the target scene to obtain a plurality of training subsets; The segmentation algorithm is trained respectively through a plurality of the training subsets, and the values ​​of the first threshold and the second threshold are set for each of the training subsets respectively.

6. The image restoration method according to claim 1, It is characterized in that The step of replacing all unqualified pixels specifically includes: Traversing the unqualified pixel points on each contour data; For the currently unqualified pixel point, obtain the pixel point closest to it in the contour data; The depth value of the nearest pixel is used as the depth value of the currently unqualified pixel to complete the replacement.

7. A restoration and completion system for implementing the image restoration and completion method according to any one of claims 1 to 6, It is characterized in that include: An image input module, used for inputting image data acquired by a sensor; An image segmentation module, used for dividing the image data into a plurality of contour data according to the object of the target scene corresponding to the image data; A data preprocessing module, used for aligning the contour data and the depth value, removing the pixel points that are noise and outliers in the contour data, performing filtering operations and correcting the coordinate system of the data; An image completion module, used for completing the removed pixel points according to the association relationship; A repair and optimization module, used for adjusting the contour data, including removing artifacts, smoothing images, correcting alignment errors, color correction, and removing duplicate pixels; An image output module generates and outputs image data according to the adjusted contour data; A user interface editing module, which includes a user interface for displaying and editing image data, adjusting parameters, and monitoring processing progress in real time; The parameter setting and automation module is used to adjust the parameters of the image data processing to adjust the image processing process according to the type of the target scene, and adjust the processing parameters according to the matching relationship between the preset parameters and the type of the target scene.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, It is characterized in that When the processor executes the computer program, the steps of the image restoration and completion method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the image restoration and completion method according to any one of claims 1 to 6 are implemented.