Image object tagging method and circuitry

Abstract

An image object labeling method and circuit system for implementing the method, in which an image is first obtained, the image is divided into one or more blocks by object classification, each block is classified into a category and is assigned a category label, and depth information of each pixel in the image is estimated by a depth estimation method. Then, according to the category of each block, it is determined whether the depth information of each pixel in the block matches the category of the block to which the pixel belongs. When the depth information of any pixel in each block matches the category of the block to which the pixel belongs, weights are set according to the category labels of each block, and a post-processing procedure is performed on the image according to the weights of each block. Otherwise, if the depth information of any pixel does not match the category of the block to which the pixel belongs, it is considered as noise and the post-processing procedure is not used.

Description

Technical Field

[0001] The specification discloses a method for labeling objects in an image, and in particular a method and circuit system for classifying objects in an image and matching pixel depths to perform image object labeling. Background Technology

[0002] When an imaging device generates an image, it performs a series of image enhancement procedures, such as adjusting overly bright or dark areas of the image, enhancing contrast and saturation, and other processing such as sharpening, spatial noise reduction, and color adjustment. These image enhancement steps depend on the results of classifying objects in the image.

[0003] A common approach is to segment objects within an image. The results of object segmentation can be used for fine-tuning of image quality. However, incorrect classification results may lead to enhancement of objects that should not have their image quality enhanced. Therefore, improving and avoiding erroneous classification results has always been a challenge in image calibration.

[0004] In the existing technology, a random condition field method is proposed, which is a statistical modeling method for recognizing patterns in images. It can be used for post-processing of the results of image segmentation by object classification. However, such post-processing algorithms mostly require a lot of computing time and cost. Summary of the Invention

[0005] To improve and avoid problems caused by incorrect classification results in image post-processing, the publication proposes an image object labeling method and circuit system, one of the purposes of which is to improve the results of image classification and avoid the problem of image enhancement post-processing on parts that should not be enhanced.

[0006] According to an embodiment of the image object labeling method, an image is first acquired, and the image is divided into one or more blocks using an object classification method. Each block is classified into a category and assigned a corresponding category label. Depth information is estimated for each pixel of the image using a depth estimation method. Then, based on the classification of each block of the image, it is determined whether the depth information of each pixel in each block matches the category of the block to which the pixel belongs. If the depth information of any pixel in each block matches the category of the block to which the pixel belongs, a weight can be set according to the category label of each block, and then a post-processing process is performed on the image according to the weight of each block. Conversely, if the depth information of any pixel does not match the category of the block to which the pixel belongs, it is considered noise, and no post-processing process can be performed, or a lower level of processing can be applied.

[0007] In this process, pixels considered as noise can be assigned a lower weight. The circuit system can perform a post-processing procedure based on the weight of each block, enhancing pixels in blocks with higher weights in the image, while leaving unprocessed or processing pixels judged as noise to a low degree.

[0008] Preferably, each block in the image, classified according to object type, represents an object in the image and is assigned a classification label. Furthermore, the classification label assigned to each block can correspond to a depth range; if the depth information of any pixel in a block exceeds the depth range of its corresponding block, the pixel is determined not to match the classification of its block.

[0009] In one implementation, in each block, a depth difference of the block can be obtained based on the individual depth information of multiple pixels in the block, and pixels in the block whose depth difference is greater than a depth difference threshold are regarded as noise.

[0010] In another implementation, a relative positional relationship is established between multiple blocks with different classification labels in the image. By comparing the depth information of pixels in multiple blocks, pixels that do not conform to the relative positional relationship are regarded as noise.

[0011] Preferably, the object classification method employs an image classification neural network model; while the depth estimation method may employ a depth estimation neural network model.

[0012] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0013] Figure 1 A block diagram showing an embodiment of the circuit system for performing the image object tagging method;

[0014] Figure 2 Flowchart of an embodiment of the method for displaying image object tags;

[0015] Figure 3 Flowchart of an embodiment of the image segmentation method in the method for displaying image object tags;

[0016] Figure 4 A flowchart illustrating an embodiment of the method for estimating depth in the display image object tagging method;

[0017] Figure 5A and Figure 5B One of the embodiment diagrams showing image segmentation;

[0018] Figure 6A and Figure 6B Figure 2 shows an embodiment of image segmentation;

[0019] Figure 7A and Figure 7B One of the embodiment diagrams showing depth estimation; and Figure 8A and Figure 8B Figure 2 shows an example of depth estimation.

[0020] Symbol Explanation

[0021] 100: Input Image

[0022] 10: Circuit System

[0023] 101: Image Segmentation Unit

[0024] 103: Depth Estimation Unit

[0025] 105: Category Tags

[0026] 107: In-depth Information

[0027] 109: Judgment Unit

[0028] 111: Image Object Classification Tag Unit

[0029] 113: Post-processing

[0030] 115: Output Image

[0031] 501: Item One

[0032] 502: Item Two

[0033] 503: Item Three

[0034] 504: Item Four

[0035] 501': Object-1 Division Area

[0036] 502': Object Divided Area

[0037] 503': Three-part division of an object

[0038] 504': Object divided into four sections

[0039] 601: Item One

[0040] 602: Item Two

[0041] 603: Item Three

[0042] 604: Item Four

[0043] 605: Item Five

[0044] 601': Object-1 Division Area

[0045] 602': Object Divided Area

[0046] 603': Three-part division of an object

[0047] 604': Object quadrilateral

[0048] 605': Five-section division of an object

[0049] 701: Vision

[0050] 702: Medium Shot

[0051] 703: Close-up

[0052] 701': Depth of Vision

[0053] 702': Medium Depth of Shot

[0054] 703': Depth of field

[0055] 801: Vision

[0056] 802: Medium Shot

[0057] 803: Close-up

[0058] 801': Depth of Vision

[0059] 802': Medium Depth of Shot

[0060] 803': Depth of field

[0061] The flowchart of the image object tagging method in steps S201 to S213

[0062] The process of image segmentation in the image object tagging method in steps S301 to S307

[0063] The process of estimating depth in the image object tagging method, steps S401 to S411 Detailed Implementation

[0064] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0065] It should be understood that while terms such as "first," "second," and "third" may be used in this document to describe various components or signals, these components or signals should not be limited by these terms. These terms are primarily used to distinguish one component from another, or one signal from another. Furthermore, the term "or" as used herein should, as appropriate, include any combination of one or more of the associated listed items.

[0066] In order to more accurately distinguish objects at different depths in an image so as to perform different levels of image processing on different blocks, this disclosure proposes an image object labeling method and circuit. One of the main purposes is to first classify the image into multiple blocks according to the objects in it, and estimate the depth of the pixels in the image, thereby confirming whether the depth information of the pixels matches the classification of the block to which they belong. This can obtain accurate object classification and eliminate noise, so as to accurately perform a post-processing process on the image.

[0067] The image object tagging method proposed in the open book, besides being implemented through software, can also be applied to... Figure 1 In the circuit system 10 shown, the circuit system 10 can be an integrated circuit used in an electronic device, or a computer system that performs related methods. The integrated circuit can also be an application-specific integrated circuit (ASIC) or a system on a chip (SoC).

[0068] according to Figure 1 The illustrated block diagram of the circuit system embodiment shows that circuit system 10 implements multiple functional units according to firmware and hardware collaboration, wherein the image object tagging method in operation can also be referenced. Figure 2 The flowchart of the shown embodiment.

[0069] Initially, the circuit system 10 receives an input image 100, which is a raw image (step S201). The circuit system 10 performs image object recognition and classification, pixel depth estimation and noise removal through various functional units, and then labels the image objects. After that, weights can be set and image post-processing can be performed.

[0070] According to an embodiment, the image segmentation unit 101 segments the acquired input image 100 into one or more blocks, wherein an object classification method can be run. The object classification method can identify and classify objects based on object features in a database using an image recognition technology. In another embodiment, the object classification method can employ a neural network model for image classification. This model is trained using neural network technology on a large amount of image data to derive the object classification model, thereby performing image object identification and classification. Then, based on the resulting object segmentation, the image is divided into one (no object identified) or multiple blocks (multiple objects identified). Each block is classified into a category and assigned a classification label 105 (step S203).

[0071] It is worth mentioning that in the process of image object recognition and classification, the relevant algorithms are based on the characteristics of various image objects, such as shape, color, depth, position, and / or the relative relationship between foreground and background. This allows the image object labeling method proposed in the publication to exclude misclassified pixels or blocks based on these criteria. In the image segmentation process, segmenting the image refers to adding a label to each pixel in the digital image, subdividing it into multiple image sub-blocks. This process ensures that pixels with common image features have the same label, aiming to simplify or change the image representation, making the image easier to analyze and understand. Common image segmentation algorithms include: thresholding, clustering (e.g., K-means algorithm), histogram (e.g., Otsu's method), watershed, level set method, wavelet-based image segmentation, or other deep learning algorithms based on neural networks, such as semantic segmentation and instance segmentation.

[0072] On the other hand, the circuit system 10 uses the depth estimation unit 103 to perform depth estimation on each pixel in the input image 100 (excluding pixels selected through a sampling procedure) using a depth estimation method to obtain the pixel's depth information 107 (step S205). According to one embodiment, the step of estimating the depth information of multiple pixels in an image can be performed by taking images of the same location from different positions. That is, taking two (at least two) images from different perspectives of the same target object, obtaining two pixels of the same location of the target object in the two (or more) images, and then calculating the disparity of the two or more pixels corresponding to each location in the two (or more) images, thus obtaining the depth information of each location. In another embodiment, the depth estimation method can employ a depth estimation neural network model, wherein a model for estimating the depth information of each pixel in the image can be obtained by first training a large amount of image data using neural network technology, and then this neural network model for estimating pixel depth information can be applied to estimate the depth information of pixels in various images.

[0073] It's worth mentioning that the depth estimation method proposed in the publication refers to the process of inputting one or more digital images and calculating the distance information between the viewpoint and the surface of scene objects. Besides the aforementioned neural network-based deep learning algorithm, other common depth estimation algorithms include stereomatching, which first inputs a pair of left and right images captured simultaneously and subjected to extreme correction. Scene depth information is obtained through steps such as matching cost calculation, cost aggregation, disparity calculation, and disparity refinement. Related descriptions can be found in [reference needed]. Figure 4 .

[0074] After obtaining the classification labels 105 and pixel depth information 107 of each block in the image, the circuit system 10 uses the judgment unit 109 to determine whether the depth information of each pixel in each block matches the classification of the block to which the pixel belongs (step S207). In this judgment step S207, if the depth information of any pixel does not match the classification of the block to which the pixel belongs (no), it is considered noise (step S211), or the image object segmentation labeling unit 111 sets a classification label for the successfully matched pixel. In subsequent steps, when the depth information of any pixel in each block matches the classification of the block to which the pixel belongs (yes), the circuit system 10 can perform post-processing 113 on the matched block. In this process, a weight can be set according to the classification label of each block, and then the image is post-processed according to the weight of each block (step S209), finally generating the post-processed output image 115 (step S213).

[0075] According to the embodiment, during the process of determining whether a pixel matches the depth of its block, the determination unit 109 determines that the pixel does not match the block's classification based on the depth range corresponding to the classification label of each block. Within each block, a depth difference is calculated based on the individual depth information of multiple pixels, representing the depth difference between pixels with greater depth and pixels with less depth within the block. In this determination, pixels with a depth difference greater than a threshold are considered noise. Furthermore, a relative positional relationship is established between multiple blocks with different classification labels. By comparing the depth information of pixels in multiple blocks, pixels that do not conform to the relative positional relationship are considered noise. Moreover, since there may be small, categorized blocks in the image, the pixel depths within these small blocks should be within a consistent depth range; pixels with abnormal depths can also be classified as noise.

[0076] For example, each image may include objects with absolute positions, such as the sky, clouds, and mountains that must appear in the far distance of the scene. Therefore, the depth information of pixels in such object blocks (with a classification label) should have a certain depth. When the depth of pixels in such object blocks is shallow, the method can correct the classification label of this pixel to noise.

[0077] In another example, for objects in small blocks, such as faces or balls, these objects are not prone to significant depth differences in the scene. However, if the depth information of pixels within such object blocks differs greatly, the classification label of this pixel is corrected to noise.

[0078] In another example, the positions of multiple objects in a scene inevitably have a spatial relationship; for instance, the sky will be in front of the plants, and a person in the foreground should be in front of a building in the background. If a pixel in the foreground has excessive depth information, its classification label should also be corrected to noise. Then, appropriate weight values ​​can be assigned based on the classification labels of each block. For example, if you want to increase the sharpness of the grass while decreasing the sharpness of the sky, you can assign a larger weight to the grass and a smaller weight to the sky, performing post-processing sharpening on the image.

[0079] Based on the judgment result of the judgment unit 109, when the estimated depth information of a pixel does not match the depth that its block should have, such pixels considered as noise can be ignored, or a lower weight value can be assigned to such pixels considered as noise. The circuit system performs a post-processing procedure according to the weight of each block, that is, it strengthens the pixels in the blocks with higher weights in the image, while not processing or processing the pixels judged as noise to a low degree. When each block of the entire image has its own weight, a weight matrix is ​​formed. The circuit system will apply this weight matrix to the image and execute the post-processing procedure.

[0080] In the above-described object classification method that segments and labels images, one embodiment can be referred to. Figure 3 The flowchart shown.

[0081] When the original image is obtained, the image pixel information is first obtained (step S301), then the similarity between adjacent pixels is calculated (step S303) so that multiple pixels in the image can be distinguished into one or more blocks according to the pixel similarity and the segmentation threshold preset in the circuit system (step S305), and then each block is labeled (step S307).

[0082] When using pixel similarity as the basis for image differentiation, the applied similarity can be defined on different features, such as the grayscale or color value of pixels, the texture or geometric structure of small pixel blocks, and the movement or deformation of blocks. Corresponding segmentation algorithms can include threshold-based segmentation, region-based segmentation, and motion-based segmentation.

[0083] Further implementation methods for pixel depth estimation described above can be found in [reference needed]. Figure 4 The flowchart of the embodiment shown is shown.

[0084] In this process, at least two images with parallax are obtained by shooting from different angles (step S401), and then limit correction is performed (step S403). This step is used to correct the image obtained by shooting the target object with the camera so that the correct image with parallax can be obtained.

[0085] Next, matching cost computation is performed (step S405). Matching cost computation calculates the depth information of each pixel in the original image. This is done by matching the pixel in the image with all disparity possibilities and calculating the cost. The calculated cost can be stored in a three-dimensional array, usually called a disparity space image (DSI). Based on this, noise that does not need to be matched can be eliminated, improving the accuracy of depth information calculation.

[0086] Next, cost aggregation (step S407) is performed to aggregate the above matching cost calculation results to obtain the cumulative cost of image disparity at each position on the image. By matching cost aggregation, the influence of noise (pixels with abnormal depth) is reduced and the matching accuracy is improved.

[0087] The next step is disparity computation (step S409). After the matching cost calculation and cost aggregation calculation described above, this step selects the pixel with the optimal cumulative cost within a disparity search range as the basis for disparity computation. Finally, accurate depth estimation can be performed through disparity refinement (step S411). In the calculation of pixel depth values ​​by calculating disparity, the above steps can obtain at least two images of the same location to calculate disparity. However, this disparity value may still have some problems, such as noise or errors caused by matching errors. Therefore, the disparity map needs to be optimized. Optimization methods include interpolation, subpixel enhancement, refinement, and image filtering.

[0088] Figure 5A and Figure 5B The following diagram illustrates an example of image segmentation, demonstrating how object classification works in the image object labeling method.

[0089] Figure 5A The image displays an original image. In a simple classification process, the background sky can be roughly considered object 1 (501), the Eiffel Tower (502), the trees on the left and right sides (503), and the vehicles in the foreground (504). Then, based on the pixel features and similarity of each block in the image, after image classification processing, as shown... Figure 5BAs shown, after noise in the image is eliminated through the above embodiment, the image can be divided into object segmentation area 501', which corresponds to the sky block; object segmentation area 502', which corresponds to the tower block; object segmentation area 503', which corresponds to the trees on the left and right sides of the image; and object segmentation area 504', which corresponds to the vehicles in the foreground.

[0090] Figure 6A and Figure 6B This diagram shows another example of image segmentation.

[0091] Figure 6A The image shows objects that can be roughly distinguished as follows: object 1 (sky), object 2 (distant mountains and trees), object 3 (a row of houses), object 4 (a boat in the lake), and object 5 (the lake). Figure 6B The display shows a schematic diagram of the image classification process after the object classification method is running, based on the characteristics and similarity of each pixel in the image. That is, the blocks distinguished by pixel similarity can include object segmentation area 601' (sky), object segmentation area 602' (distant mountains and trees), object segmentation area 603' (a row of houses), object segmentation area 604' (boats in the lake), and object segmentation area 605' (lake).

[0092] In the image object labeling method, according to the above Figure 5B and Figure 6B The displayed segmented regions are assigned individual classification labels, and weight values ​​are set as needed during post-processing. For example, regarding the improved object classification results in the image described in the above embodiments, when image enhancement processing is to be performed on the sky in the image, a higher weight value can be set for the block corresponding to the sky's classification label. When performing a certain post-processing (such as saturation, contrast, sharpening, color temperature, noise reduction, and / or super-resolution imaging), the sky block will be enhanced.

[0093] Furthermore, when classifying based on the depth information of each pixel, the image can be divided into blocks with different depth ranges, such as a distant view, a medium view, and a close-up view. An implementation example can be found here. Figure 7A and Figure 7B An example diagram showing depth estimation is displayed.

[0094] exist Figure 7A In the displayed image diagram, a forest is shown as the background (701), the mid-ground (702) appears as a lawn, and the foreground (703) shows a garden. The background (701), mid-ground (702), and foreground (703) each have different depth information and different depth ranges. Based on this depth information, the image is processed to display as follows: Figure 7BThe diagram shows three different depth blocks: far-field depth 701', mid-field depth 702', and near-field depth 703'.

[0095] Figure 8A and Figure 8B This diagram shows another example of depth estimation.

[0096] Figure 8A The sky is shown as a distant view (801), the castle as a medium view (802), and the lawn and path in the foreground as a close-up (803). After categorization, the diagram is displayed as follows: Figure 8B The image is divided into three blocks with different depths: a distant depth of 801', a mid-range depth of 802', and a near depth of 803'.

[0097] As can be seen from the above diagram, after classifying objects in the image, we can further eliminate noise caused by misjudgment based on the depth information of pixels in the classification blocks. The characteristics of this can be: the depth difference reflected by the front-to-back relationship of objects (if the pixel depth does not match, it is considered noise); pixels within an object block should have consistent depth information (if the pixel depth does not match, it is considered noise); and the pixel depth difference within a small range should not be large (pixels with excessive depth differences are considered noise).

[0098] After segmenting the image into blocks and matching based on pixel depth information, different classification labels can be accurately assigned to different objects in the image. This also allows for more accurate processing of different objects to varying degrees during post-processing. Furthermore, the image can be optimized using spatial and temporal filters. The purpose of filtering is to smooth out the resulting blocks of the object classification and avoid side effects.

[0099] In summary, the image object tagging method and circuit system described in the above embodiments can use an intelligent model to estimate pixel depth information in the image and classify objects in the image. Then, it can determine whether the pixel depth matches the classified blocks to confirm that the objects in the image conform to the characteristics of the physical world. It can also provide feedback to correct the intelligent model. One of its purposes is to accurately classify and partition objects in the image so that they can be post-processed to meet user needs.

[0100] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the claims of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of the claims of the present invention.

Claims

1. A method for tagging image objects, executed in a circuit system, comprising: Obtain an image; The image is divided into one or more blocks using an object classification method. Each block is classified into a category and assigned a category label. The category label of each block corresponds to a depth range. The depth information of each pixel in the image is estimated using a depth estimation method. Based on the classification labels of each block of the image, the depth information of each pixel in each block is compared with the depth range corresponding to the classification label of the block to determine whether the depth information of each pixel in each block matches the classification of the block to which the pixel belongs. as well as When the depth information of any pixel in each block exceeds the depth range of its block, it is determined that the pixel does not match the classification of its block; when the depth information of any pixel in each block matches the classification of its block, the circuit system sets a weight according to the classification label of each block, and then performs a post-processing process on the image according to the weight of each block. If the depth information of any pixel does not match the classification of the block to which that pixel belongs, it is considered noise.

2. The image object labeling method as described in claim 1, wherein each block in the image, derived according to object classification, represents an object in the image, and is assigned a classification label.

3. The method of claim 1, wherein, In this circuit system, in each block, a depth difference of the block is obtained based on the individual depth information of multiple pixels in the block, and pixels in the block whose depth difference is greater than a depth difference threshold are regarded as noise.

4. The image object labeling method as described in claim 1, wherein a relative positional relationship is provided between the plurality of blocks with different classification labels, and pixels that do not conform to the relative positional relationship are regarded as noise by comparing the depth information of the pixels in the plurality of blocks.

5. The method of claim 1, wherein, In this object classification method, the similarity between adjacent pixels in a plurality of pixels of the image is calculated, and then the plurality of pixels are classified according to the similarity between adjacent pixels and a segmentation threshold to distinguish the image as one or more blocks.

6. The image object tagging method as described in claim 1, wherein the step of estimating the depth information of multiple pixels in the image includes: For two images of the same target object, obtain two pixels at the same location of the target object in the two images; as well as Calculate the disparity between the two pixels corresponding to each position in the two images to obtain the depth information for each position.

7. The image object label method according to any one of claims 1 to 6, wherein The circuit system assigns a lower weight to pixels considered as noise. Based on the weight of each block, the post-processing process is performed to enhance pixels in blocks with higher weights in the image, while pixels judged as noise are not processed or are processed to a low degree.

8. A circuit system for operating an image object tagging method, comprising: Obtain an image; The image is divided into one or more blocks using an object classification method. Each block is classified into a category and assigned a category label. The category label of each block corresponds to a depth range. The depth information of each pixel in the image is estimated using a depth estimation method. Based on the classification labels of each block of the image, the depth information of each pixel in each block is compared with the depth range corresponding to the classification label of the block to determine whether the depth information of each pixel in each block matches the classification of the block to which the pixel belongs. as well as When the depth information of any pixel in each block exceeds the depth range of its block, it is determined that the pixel does not match the classification of its block; when the depth information of any pixel in each block matches the classification of its block, the circuit system sets a weight according to the classification label of each block, and then performs a post-processing process on the image according to the weight of each block. If the depth information of any pixel does not match the classification of the block to which that pixel belongs, it is considered noise.

9. The circuitry of claim 8, wherein, Each block is assigned a classification label corresponding to a depth range. When the depth information of any pixel in a block exceeds the depth range of its block, the pixel is determined not to match the classification of its block. In each block, a depth difference is obtained based on the individual depth information of multiple pixels in the block. Pixels in the block whose depth difference is greater than a depth difference threshold are considered noise. Furthermore, a relative positional relationship is established between the multiple blocks with different classification labels. By comparing the depth information of the pixels in the multiple blocks, pixels that do not conform to the relative positional relationship are regarded as noise.

Images