Image processing and neural network construction method, device and storage medium

Abstract

The embodiments of the present invention provide an image processing method, an apparatus, and a computer-readable storage medium, as well as a neural network construction method, an apparatus, and a computer-readable storage medium. The image processing method according to the embodiments of the present invention includes: obtaining an input image; processing the input image to enhance the structural features of the input image and obtain a structural image; processing the input image and the structural image using a neural network to extract common structural features of the input image and the structural image; and segmenting the input image according to the structure using the neural network at least based on the common structural features to obtain a segmentation result of the input image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device and computer-readable storage medium, as well as a neural network construction method, device and computer-readable storage medium. Background Art

[0002] Segmenting images according to their structure, and further extracting contours and detecting defects are important research directions in the field of image processing. In the current image segmentation and defect detection process, a large amount of labeled data is usually used to train the image segmentation network, so as to achieve image segmentation according to structure and defect detection based on the segmentation results through the trained image segmentation network.

[0003] However, factors such as the color of objects in the image may introduce noise into the image, thereby affecting the representation of the structural features of the image, which will lead to a decrease in the accuracy of image segmentation according to the structure and further affect the results of defect detection.

[0004] Therefore, there is a need for an image processing method and device that can effectively utilize the structural features of an image to perform image segmentation, thereby improving the accuracy of the image segmentation result. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, there is provided an image processing method, comprising: obtaining an input image; processing the input image to enhance the structural features of the input image and obtain a structural image; processing the input image and the structural image to extract common structural features of the input image and the structural image; and segmenting the input image according to the structure at least based on the common structural features to obtain a segmentation result of the input image.

[0006] According to another aspect of the present invention, there is provided an image processing method, comprising: obtaining an input image; segmenting the input image according to structure using a neural network; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image and obtain a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least based on the common structural features and obtaining a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with the actual segmentation result of the input training image to adjust the parameters of the neural network.

[0007] According to another aspect of the present invention, there is provided a method for constructing a neural network for image processing, comprising: constructing a neural network comprising an encoder, a decoder and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; configuring the encoder to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image; configuring the common feature extraction layer to process the input image and the structural image to extract common structural features of the input image and the structural image; configuring the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

[0008] According to another aspect of the present invention, there is provided an image processing device, comprising: an acquisition unit, configured to acquire an input image; an enhancement unit, configured to process the input image to enhance the structural features of the input image and acquire a structural image; an extraction unit, configured to process the input image and the structural image and extract common structural features of the input image and the structural image; and a segmentation unit, configured to segment the input image according to structure at least based on the common structural features and acquire a segmentation result of the input image.

[0009] According to another aspect of the present invention, there is provided an image processing device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor performs the following steps: acquiring an input image; processing the input image to enhance structural features of the input image and acquiring a structural image; processing the input image and the structural image to extract common structural features of the input image and the structural image; and segmenting the input image according to structure at least based on the common structural features to obtain a segmentation result of the input image.

[0010] According to another aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring an input image; processing the input image to enhance structural features of the input image and acquiring a structural image; processing the input image and the structural image to extract common structural features of the input image and the structural image; and structurally segmenting the input image at least based on the common structural features to obtain a segmentation result of the input image.

[0011] According to another aspect of the present invention, there is provided an image processing device, comprising: an acquisition unit, configured to acquire an input image; a processing unit, configured to use a neural network to segment the input image according to structure; wherein the neural network used by the processing unit is trained in the following manner: acquiring an input training image; processing the input training image to enhance the structural features of the input training image and acquire a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least according to the common structural features and acquiring a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with the true segmentation result of the input training image to adjust the parameters of the neural network.

[0012] According to another aspect of the present invention, there is provided an image processing device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: obtaining an input image; segmenting the input image according to structure using a neural network; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image and obtaining a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least according to the common structural features and obtaining a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with a true segmentation result of the input training image to adjust the parameters of the neural network.

[0013] According to another aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining an input image; segmenting the input image according to structure using a neural network; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image and obtaining a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least according to the common structural features and obtaining a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with a true segmentation result of the input training image to adjust the parameters of the neural network.

[0014] According to another aspect of the present invention, there is provided a device for constructing a neural network for image processing, comprising: a construction unit, configured to construct a neural network including an encoder, a decoder and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; an encoding configuration unit, configured to configure the encoder to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image; an extraction layer configuration unit, configured to configure the common feature extraction layer to process the input image and the structural image to extract common structural features of the input image and the structural image; and a decoding configuration unit, configured to configure the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

[0015] According to another aspect of the present invention, there is provided a device for constructing a neural network for image processing, comprising: a processor; and a memory, in which computer program instructions are stored, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: constructing a neural network comprising an encoder, a decoder and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; configuring the encoder to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image; configuring the common feature extraction layer to process the input image and the structural image to extract common structural features of the input image and the structural image; configuring the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: constructing a neural network including an encoder, a decoder and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; configuring the encoder to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image; configuring the common feature extraction layer to process the input image and the structural image to extract common structural features of the input image and the structural image; configuring the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

[0017] According to the above-mentioned image processing method, neural network construction method, device and computer-readable storage medium of the present invention, it is possible to obtain a structural image by enhancing the structural features of the input image, and to segment the input image according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0019] Figure 1 An example showing two images of the same structure but different colors;

[0020] Figure 2 A flowchart showing an image processing method according to an embodiment of the present invention is shown;

[0021] Figure 3 An example of using a neural network to remove color from an input image to obtain a structural image according to an embodiment of the present invention is shown;

[0022] Figure 4 An example of segmenting an input image and a structure image using a neural network according to an embodiment of the present invention is shown;

[0023] Figure 5 An example of extracting common features from an input image and a structure image according to an embodiment of the present invention is shown;

[0024] Figure 6 An example of fusing the segmentation results of the input image and the structure image according to one embodiment of the present invention is shown;

[0025] Figure 7 An example of performing contour extraction and defect detection on the segmentation result of an input image according to an embodiment of the present invention is shown;

[0026] Figure 8 A flowchart showing an image processing method according to another embodiment of the present invention;

[0027] Fig. 9 A flowchart showing a method for constructing a neural network for image processing according to an embodiment of the present invention;

[0028] Fig.10 A block diagram showing an image processing apparatus according to an embodiment of the present invention;

[0029] Fig.11 A block diagram showing an image processing apparatus according to another embodiment of the present invention;

[0030] Fig.12 A block diagram showing an image processing apparatus according to another embodiment of the present invention;

[0031] Fig.13 A block diagram showing an image processing apparatus according to another embodiment of the present invention;

[0032] Fig.14 A block diagram showing a device for constructing a neural network for image processing according to an embodiment of the present invention;

[0033] Fig.15 A block diagram showing a device for constructing a neural network for image processing according to another embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will describe an image processing method, device and computer-readable storage medium according to an embodiment of the present invention with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0035] In the process of structural segmentation and defect detection for the acquired input image, a large amount of labeled data is usually required to train the image segmentation network, so as to achieve image segmentation according to structure and defect detection based on the segmentation results through the trained image segmentation network. However, in the actual detection process, the complex color and pattern distribution of objects in the image will have a significant impact on the image segmentation according to structure, thereby increasing the difficulty of structure-based image segmentation and reducing the accuracy of defect detection. Figure 1 An example of two images having the same image segmentation region in structure but having different colors respectively is shown. Figure 1The uppers shown in the left and right images are of the same specifications and fabric material, the only difference being that the upper in the left image has a lighter single color, while the upper in the right image has a darker complex pattern. Figure 1 In the example, the difficulty of image segmentation according to the structure of the left and right images will deviate greatly. The segmentation difficulty of the image with dark complex patterns will be significantly greater than that of the image with a single light color. In addition, the segmentation accuracy of the image with dark complex patterns may also be greatly reduced relative to the segmentation accuracy of the image with a single light color.

[0036] Therefore, it is desirable to provide an image processing method and apparatus that can effectively remove the influence of noise generated by non-structural features (such as color, pattern, etc.) in the input image on the image segmentation result according to the structure segmentation, thereby improving the accuracy of the image segmentation result. In addition, the image processing method, apparatus, and computer-readable storage medium according to the embodiments of the present invention can be applied to a variety of application scenarios, such as image segmentation and defect detection in various situations such as fabric texture, road surface, and chip surface.

[0037] The following will refer to Figure 2 An image processing method according to an embodiment of the present invention is described. Figure 2 A flow chart of the image processing method 200 is shown.

[0038] like Figure 2 As shown, in step S201, an input image is obtained.

[0039] In this step, the input image may be an input image acquired by an image acquisition device such as a camera or a video camera, for example, a two-dimensional image or a frame of a two-dimensional image captured from a video. In addition, the input image may also be an output image of any layer in one or more neural networks, etc., which is not limited here.

[0040] In step S202, the input image is processed to enhance the structural features of the input image and obtain a structural image.

[0041] In this step, the specific operation of enhancing the structural features of the input image may include: performing at least one of color removal, edge enhancement, texture enhancement, contrast enhancement, noise removal, and resolution adjustment on the input image. Optionally, various image processing methods may be used to perform at least one of the above operations, for example, a corresponding neural network may be selected to perform at least one of the above operations. Figure 3 An example of using a neural network to remove color from an input image to obtain a structural image according to an embodiment of the present invention is shown. Figure 3As shown, an input image A with complex color patterns can be input into a neural network, such as a generative network, and a structural image B that embodies the structural features of the enhanced input image can be obtained through the illustrated encoding and decoding process. In the encoding process, the processing of multiple groups of convolution, normalization and pooling processes with different parameters in the generative network can be used to extract structural features of different scales. First, the input image can be convolved to obtain a convolution map. Then, the convolution map can be normalized using a linear correction unit and a batch normalization method to obtain a normalized convolution map. Subsequently, the normalized convolution map can be subjected to maximum pooling or average pooling. In order to obtain rich, multi-scale structural features, various relevant parameters can be adjusted during the processing process, and the above process can be repeated multiple times to extract multi-scale structural feature maps respectively. Finally, the resolution of the feature map can be restored by methods such as convolution, normalization and upsampling during the decoding process, and a fused structural image can be obtained, which has enhanced structural features of the input image.

[0042] Of course, the input image may also be processed by edge enhancement combined with noise removal, contrast enhancement, texture enhancement combined with resolution adjustment, etc. The above-mentioned processing of the input image to enhance the structural features is only an example. In practical applications, any one or more methods that enhance the structural features of the input image may be used for processing, and no limitation is made here.

[0043] In step S203, the input image and the structural image are processed to extract common structural features of the input image and the structural image.

[0044] Specifically, the input image and the structural image can be processed by a neural network, and extracting the common structural features of the input image and the structural image can include: inputting the input image and the structural image into the neural network respectively; using the neural network, encoding the input image and the structural image at multiple different step lengths respectively to obtain the encoding features of the input image and the structural image at each different step length; according to the encoding features of the input image and the structural image at each different step length, obtaining the common structural features of the input image and the structural image. In the above steps, the structural features of the input image and the structural image can be forced to be focused on, while ignoring other features such as color and pattern, so that the common structural features of the input image and the structural image obtained can be more robust to the situation of having the same structure and different color patterns.

[0045] In step S204, the input image is segmented according to the structure at least based on the common structural features to obtain a segmentation result of the input image.

[0046] In this step, the neural network can be used to segment the input image and the structural image respectively according to at least the common structural features, and then the results of the respective segmentations are fused based on a predetermined rule as a segmentation result of the input image.

[0047] Figure 4 An example of segmenting an input image and a structure image using a neural network according to an embodiment of the present invention is shown. Figure 4 In Figure 3 The input image A and the obtained structural image B are respectively input into a neural network, such as a segmentation network, and the features of the input image A and the structural image B are respectively encoded during the encoding process. Subsequently, after encoding, the common structural features of the input image and the structural image can be extracted from the encoded features. After extracting the common structural features, the encoded features and the extracted common structural features can be combined to obtain combined features, and the obtained combined features can be decoded during the decoding process, and the segmentation results of the input image A and the structural image B are respectively output according to the decoded features, that is, Figure 4 A' and B' in.

[0048] Figure 5 FIG. 2 shows an example of extracting common features from an input image and a structural image according to an embodiment of the present invention. Figure 5 As shown, features of different receptive fields can be extracted from the input image A and the structural image B using convolution blocks of multiple different step sizes, thereby obtaining encoding features of the input image A and the structural image B at multiple different step sizes. Subsequently, the features of the input image A and the structural image B at corresponding step sizes can be connected, and the above process can be repeated multiple times until the common structural features of the input image and the structural image are obtained. The common structural features of the input image and the structural image may include the common structural features obtained at each different step size, or may include the common structural features at some of the step sizes, which are not limited here.

[0049] Figure 6 FIG. 4 shows an example of fusing the segmentation results A' and B' of the input image and the structure image according to an embodiment of the present invention. Figure 6 As shown, a neural network, such as a fusion network, can be used to fuse the received segmentation results A' and B'. For example, the segmentation results A' and B' can be aggregated first, and feature extraction can be performed to obtain an extracted feature map; then, the extracted feature map can be fused based on a predetermined rule as a segmentation result C of the input image.

[0050] In one embodiment of the present invention, the contour of the input image can be further extracted according to the segmentation result of the input image, so as to be used for various purposes such as contour display, image recognition and analysis of the object contained in the input image. In addition, according to another embodiment of the present invention, the contour of the input image can be further compared with the standard contour to perform defect detection on the input image.

[0051] Figure 7 An example of performing contour extraction and defect detection on the segmentation result C of the input image according to an embodiment of the present invention is shown. Figure 7 In the method, the final segmentation result C of the input image according to the structure can be first obtained, and the contour and / or feature point information of the segmentation result C can be extracted to obtain the contour of the input image. Subsequently, the pre-acquired standard contour can be used to compare with the contour of the input image, and the difference between the standard contour and the contour of the input image can be calculated, for example, the distance between the standard contour and the contour of the input image. After the difference between the two is obtained, defects can be detected from the contour of the input image according to the difference, and the detected defect position and the degree of contour deviation (or confidence) can be further output.

[0052] The above-mentioned image processing method according to the embodiment of the present invention can obtain a structural image by enhancing the structural features of the input image, and segment the input image according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection.

[0053] Refer to the following Figure 8 An image processing method according to an embodiment of the present invention is described. Figure 8 A flow chart of the image processing method 800 is shown.

[0054] In step S801, an input image is acquired.

[0055] In this step, the input image may be an input image acquired by an image acquisition device such as a camera or a video camera, for example, a two-dimensional image or a frame of a two-dimensional image captured from a video. In addition, the input image may also be an output image of any layer in one or more neural networks, etc., which is not limited here.

[0056] In step S802, the input image is segmented according to structure using a neural network; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image and obtaining a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least based on the common structural features to obtain a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with a true segmentation result of the input training image to adjust the parameters of the neural network.

[0057] In this step, the input image can be segmented according to the structure using a neural network, thereby obtaining the segmentation result of the input image. The training method of the neural network here is the same as Figure 2 The process described in is similar, that is, the labeled input training images for training can be used Figure 2 The process shown is to perform segmentation according to the structure, obtain the segmentation result of the input training image, and compare it with the real segmentation result of the input training image to adjust the parameters of the neural network. The training of the neural network in the embodiment of the present invention can be performed by inputting a large number of input training images for training to update and iterate the parameters of the neural network multiple times, so as to minimize the difference between the segmentation result of the input training image obtained by training and the real segmentation result of the marked input training image. For specific operation methods, see Figure 2 The above is not repeated here.

[0058] According to one embodiment of the present invention, processing the input training image and the structure training image to extract common structural features of the input training image and the structure training image may include: respectively acquiring structural features of the input training image and the structure training image at different scales; and acquiring common structural features of the input training image and the structure training image based on the structural features of the input training image and the structure training image at different scales.

[0059] According to another embodiment of the present invention, the input training image is processed to enhance the structural features of the input training image, and obtaining the structural training image may include: reverse processing the obtained structural training image to obtain a supervised training image, comparing the obtained supervised training image with the input training image, supervising the acquisition of the structural training image according to the comparison result, and adjusting the acquisition result of the structural training image, wherein the reverse processing is an inverse processing for enhancing the structural features of the input training image. Optionally, another trained generative network may be used to transform the structural training image as much as possible in the direction of the input training image so that the obtained supervised training image is as close to the input training image as possible. Thus, by comparing the structure and style of the generated supervised training image with the input training image, the training parameters of the neural network are adjusted.

[0060] According to another embodiment of the present invention, the input training image is processed to enhance the structural features of the input training image, and obtaining the structural training image may also include: comparing the obtained structural training image with at least one real structural image, using the comparison result to supervise the acquisition of the structural training image, and adjusting the acquisition result of the structural training image. Specifically, the structural training image obtained through training can be compared with one or more real structural images. If the comparison result between the trained structural training image and the real structural image is judged to be true, it can be considered that the structural training image is close enough to the real structural image, and therefore is real enough, which also indicates that the parameters of the neural network are sufficiently converged.

[0061] In the neural network training process of the embodiment of the present invention, all prediction and supervision results generated by the neural network training can be used to calculate the loss of the neural network, so as to continuously update the parameters of the neural network by minimizing the loss. For example, in the neural network training process, the losses in each process such as the generation of the structure training image, the comparison result of the structure training image and the real structure image, the results of the segmentation of the input training image and the structure training image, and the segmentation result of the input training image can be calculated respectively, and the total loss of the neural network is calculated by the weighted sum of these losses, so as to update the neural network parameters and determine the neural network model finally used by minimizing the total loss of the neural network.

[0062] The above-mentioned image processing method according to the embodiment of the present invention can obtain a structural image by enhancing the structural features of the input image, and segment the input image according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection.

[0063] Refer to the following Fig. 9 A method for constructing a neural network for image processing according to an embodiment of the present invention is described. Fig. 9 A flowchart of the method 900 for constructing the neural network is shown. In an embodiment of the present invention, Fig. 9 The neural network constructed by the neural network construction method shown can be used in the processing flow of the aforementioned image processing method, and similar neural network structures and specific operation steps will not be repeated here.

[0064] In step S901, a neural network including an encoder, a decoder and a common feature extraction layer is constructed, wherein the common feature extraction layer is cascaded between the encoder and the decoder.

[0065] In step S902, the encoder is configured to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image.

[0066] In step S903, the common feature extraction layer is configured to process the input image and the structural image to extract common structural features of the input image and the structural image.

[0067] In step S904, the decoder is configured to segment the input image according to the structure based on the encoding result of the encoder and the common structural features, so as to obtain a segmentation result of the input image.

[0068] The neural network construction method according to the embodiment of the present invention can construct a neural network, so that when the neural network is used for image processing, a structural image is obtained by enhancing the structural features of the input image, and the input image is segmented according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection.

[0069] Below, refer to Fig.10 An image processing apparatus according to an embodiment of the present invention will be described. Fig.10 FIG. 1 is a block diagram of an image processing apparatus 1000 according to an embodiment of the present invention. Fig.10 As shown in FIG. 1 , the image processing apparatus 1000 includes an acquisition unit 1010, an enhancement unit 1020, an extraction unit 1030, and a segmentation unit 1040. In addition to these units, the image processing apparatus 1000 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the image processing apparatus 1000 according to the embodiment of the present invention are the same as those described above with reference to FIG. Figure 2The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0070] Fig.10 The acquisition unit 1010 of the image processing device 1000 acquires an input image.

[0071] The input image acquired by the acquisition unit 1010 may be an input image acquired by an image acquisition device such as a camera or a video camera, for example, a two-dimensional image or a frame of a two-dimensional image captured from a video. In addition, the input image may also be an output image of any layer in one or more neural networks, etc., which is not limited here.

[0072] The enhancement unit 1020 processes the input image to enhance the structural features of the input image and obtain a structural image.

[0073] The specific operation of enhancing the structural features of the input image by the enhancement unit 1020 may include: performing at least one of color removal, edge enhancement, texture enhancement, contrast enhancement, noise removal, and resolution adjustment on the input image. Optionally, various image processing methods may be used to perform at least one of the above operations, for example, a corresponding neural network may be selected to perform at least one of the above operations. Figure 3 An example of using a neural network to remove color from an input image to obtain a structural image according to an embodiment of the present invention is shown. Figure 3 As shown, an input image A with complex color patterns can be input into a neural network, such as a generative network, and a structural image B that embodies the structural features of the enhanced input image can be obtained through the illustrated encoding and decoding process. In the encoding process, the processing of multiple groups of convolution, normalization and pooling processes with different parameters in the generative network can be used to extract structural features of different scales. First, the input image can be convolved to obtain a convolution map. Then, the convolution map can be normalized using a linear correction unit and a batch normalization method to obtain a normalized convolution map. Subsequently, the normalized convolution map can be subjected to maximum pooling or average pooling. In order to obtain rich, multi-scale structural features, various relevant parameters can be adjusted during the processing process, and the above process can be repeated multiple times to extract multi-scale structural feature maps respectively. Finally, the resolution of the feature map can be restored by methods such as convolution, normalization and upsampling during the decoding process, and a fused structural image can be obtained, which has enhanced structural features of the input image.

[0074] Of course, the input image may also be processed by edge enhancement combined with noise removal, contrast enhancement, texture enhancement combined with resolution adjustment, etc. The above-mentioned processing of the input image to enhance the structural features is only an example. In practical applications, any one or more methods that enhance the structural features of the input image may be used for processing, and no limitation is made here.

[0075] The extraction unit 1030 processes the input image and the structural image to extract common structural features of the input image and the structural image.

[0076] Specifically, the extraction unit 1030 can use a neural network to process the input image and the structural image, and extracting the common structural features of the input image and the structural image can include: inputting the input image and the structural image to the neural network respectively; using the neural network, encoding the input image and the structural image at multiple different step lengths respectively to obtain the encoding features of the input image and the structural image at each different step length; according to the encoding features of the input image and the structural image at each different step length, obtaining the common structural features of the input image and the structural image. In the above steps, the structural features of the input image and the structural image can be forced to be focused on, while ignoring other features such as color and pattern, so that the common structural features of the input image and the structural image obtained can be more robust to the situation of having the same structure and different color patterns.

[0077] The segmentation unit 1040 segments the input image according to the structure at least based on the common structural features to obtain a segmentation result of the input image.

[0078] The segmentation unit 1040 can use the neural network to segment the input image and the structural image respectively according to at least the common structural features, and then fuse the results of the respective segmentations based on a predetermined rule as the segmentation result of the input image.

[0079] Figure 4 An example of segmenting an input image and a structure image using a neural network according to an embodiment of the present invention is shown. Figure 4 In Figure 3The input image A and the obtained structural image B are respectively input into a neural network, such as a segmentation network, and the features of the input image A and the structural image B are respectively encoded during the encoding process. Subsequently, after encoding, the common structural features of the input image and the structural image can be extracted from the encoded features. After extracting the common structural features, the encoded features and the extracted common structural features can be combined to obtain combined features, and the obtained combined features can be decoded during the decoding process, and the segmentation results of the input image A and the structural image B are respectively output according to the decoded features, that is, Figure 4 A' and B' in.

[0080] Figure 5 FIG. 2 shows an example of extracting common features from an input image and a structural image according to an embodiment of the present invention. Figure 5 As shown, features of different receptive fields can be extracted from the input image A and the structural image B using convolution blocks of multiple different step sizes, thereby obtaining encoding features of the input image A and the structural image B at multiple different step sizes. Subsequently, the features of the input image A and the structural image B at corresponding step sizes can be connected, and the above process can be repeated multiple times until the common structural features of the input image and the structural image are obtained. The common structural features of the input image and the structural image may include the common structural features obtained at each different step size, or may include the common structural features at some of the step sizes, which are not limited here.

[0081] Figure 6 FIG. 4 shows an example of fusing the segmentation results A' and B' of the input image and the structure image according to an embodiment of the present invention. Figure 6 As shown, a neural network, such as a fusion network, can be used to fuse the received segmentation results A' and B'. For example, the segmentation results A' and B' can be aggregated first, and feature extraction can be performed to obtain an extracted feature map; then, the extracted feature map can be fused based on a predetermined rule as a segmentation result C of the input image.

[0082] In one embodiment of the present invention, the contour of the input image can be further extracted according to the segmentation result of the input image, so as to be used for various purposes such as contour display, image recognition and analysis of the object contained in the input image. In addition, according to another embodiment of the present invention, the contour of the input image can be further compared with the standard contour to perform defect detection on the input image.

[0083] Figure 7 An example of performing contour extraction and defect detection on the segmentation result C of the input image according to an embodiment of the present invention is shown. Figure 7In the method, the final segmentation result C of the input image according to the structure can be first obtained, and the contour and / or feature point information of the segmentation result C can be extracted to obtain the contour of the input image. Subsequently, the pre-acquired standard contour can be used to compare with the contour of the input image, and the difference between the standard contour and the contour of the input image can be calculated, for example, the distance between the standard contour and the contour of the input image. After the difference between the two is obtained, defects can be detected from the contour of the input image according to the difference, and the detected defect position and the degree of contour deviation (or confidence) can be further output.

[0084] The above-mentioned image processing device according to the embodiment of the present invention can obtain a structural image by enhancing the structural features of the input image, and segment the input image according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection.

[0085] Below, refer to Fig.11 An image processing apparatus according to an embodiment of the present invention will be described. Fig.11 FIG. 1 is a block diagram of an image processing apparatus 1100 according to an embodiment of the present invention. Fig.11 As shown, the device 1100 may be a computer or a server.

[0086] like Fig.11 As shown, the image processing device 1100 includes one or more processors 1110 and a memory 1120. Of course, in addition to this, the image processing device 1100 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Figure 8 The components and structure of the image processing device 1100 shown are merely exemplary and not restrictive. The image processing device 1100 may also have other components and structures as required.

[0087] The processor 1110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1120 to perform desired functions, which may include: acquiring an input image; processing the input image to enhance the structural features of the input image and acquire a structural image; processing the input image and the structural image to extract common structural features of the input image and the structural image; and structurally segmenting the input image at least based on the common structural features to acquire a segmentation result of the input image.

[0088] The memory 1120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1110 may run the program instructions to implement the functions of the image processing device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the image processing method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0089] Below, a computer-readable storage medium according to an embodiment of the present invention is described, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring an input image; processing the input image to enhance structural features of the input image and acquiring a structural image; processing the input image and the structural image to extract common structural features of the input image and the structural image; and structurally segmenting the input image at least based on the common structural features to obtain a segmentation result of the input image.

[0090] Below, refer to Fig.12 An image processing apparatus according to an embodiment of the present invention will be described. Fig.12 FIG. 1 is a block diagram of an image processing apparatus 1200 according to an embodiment of the present invention. Fig.12 As shown in FIG. 1 , the image processing device 1200 includes an acquisition unit 1210 and a processing unit 1220. In addition to these units, the image processing device 1200 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the image processing device 1200 according to the embodiment of the present invention are the same as those described above with reference to FIG. Figure 8 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0091] Fig.12 The acquisition unit 1210 of the image processing device 1200 acquires an input image.

[0092] The input image acquired by the acquisition unit 1210 may be an input image acquired by an image acquisition device such as a camera or a video camera, for example, a two-dimensional image or a frame of a two-dimensional image captured from a video. In addition, the input image may also be an output image of any layer in one or more neural networks, etc., which is not limited here.

[0093] The processing unit 1220 uses a neural network to segment the input image according to its structure; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image and obtaining a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to its structure at least based on the common structural features to obtain a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with a true segmentation result of the input training image to adjust the parameters of the neural network.

[0094] The processing unit 1220 can segment the input image according to the structure using a neural network, thereby obtaining a segmentation result of the input image. The training method of the neural network here is the same as Figure 2 The process described in is similar, that is, the labeled input training images for training can be used Figure 2 The process shown is to perform segmentation according to the structure, obtain the segmentation result of the input training image, and compare it with the real segmentation result of the input training image to adjust the parameters of the neural network. The training of the neural network in the embodiment of the present invention can be performed by inputting a large number of input training images for training to update and iterate the parameters of the neural network multiple times, so as to minimize the difference between the segmentation result of the input training image obtained by training and the real segmentation result of the marked input training image. For specific operation methods, see Figure 2 The above is not repeated here.

[0095] According to one embodiment of the present invention, processing the input training image and the structure training image to extract common structural features of the input training image and the structure training image may include: respectively acquiring structural features of the input training image and the structure training image at different scales; and acquiring common structural features of the input training image and the structure training image based on the structural features of the input training image and the structure training image at different scales.

[0096] According to another embodiment of the present invention, the input training image is processed to enhance the structural features of the input training image, and obtaining the structural training image may include: reverse processing the obtained structural training image to obtain a supervised training image, comparing the obtained supervised training image with the input training image, supervising the acquisition of the structural training image according to the comparison result, and adjusting the acquisition result of the structural training image, wherein the reverse processing is an inverse processing for enhancing the structural features of the input training image. Optionally, another trained generative network may be used to transform the structural training image as much as possible in the direction of the input training image so that the obtained supervised training image is as close to the input training image as possible. Thus, by comparing the structure and style of the generated supervised training image with the input training image, the training parameters of the neural network are adjusted.

[0097] According to another embodiment of the present invention, the input training image is processed to enhance the structural features of the input training image, and obtaining the structural training image may also include: comparing the obtained structural training image with at least one real structural image, using the comparison result to supervise the acquisition of the structural training image, and adjusting the acquisition result of the structural training image. Specifically, the structural training image obtained through training can be compared with one or more real structural images. If the comparison result between the trained structural training image and the real structural image is judged to be true, it can be considered that the structural training image is close enough to the real structural image, and therefore is real enough, which also indicates that the parameters of the neural network are sufficiently converged.

[0098] In the neural network training process of the embodiment of the present invention, all prediction and supervision results generated by the neural network training can be used to calculate the loss of the neural network, so as to continuously update the parameters of the neural network by minimizing the loss. For example, in the neural network training process, the losses in each process such as the generation of the structure training image, the comparison result of the structure training image and the real structure image, the results of the segmentation of the input training image and the structure training image, and the segmentation result of the input training image can be calculated respectively, and the total loss of the neural network is calculated by the weighted sum of these losses, so as to update the neural network parameters and determine the neural network model finally used by minimizing the total loss of the neural network.

[0099] The above-mentioned image processing device according to the embodiment of the present invention can obtain a structural image by enhancing the structural features of the input image, and segment the input image according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection.

[0100] Below, refer to Fig.13 An image processing apparatus according to an embodiment of the present invention will be described. Fig.13 FIG. 1 is a block diagram of an image processing apparatus 1300 according to an embodiment of the present invention. Fig.13 As shown, the device 1300 may be a computer or a server.

[0101] like Fig.13 As shown, the image processing device 1300 includes one or more processors 1310 and a memory 1320. Of course, in addition to this, the image processing device 1300 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.13 The components and structure of the image processing device 1300 shown are merely exemplary and non-limiting. The image processing device 1300 may also have other components and structures as required.

[0102] The processor 1310 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1020 to perform desired functions, which may include: obtaining an input image; segmenting the input image according to structure using a neural network; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image and obtain a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least based on the common structural features and obtaining a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with the actual segmentation result of the input training image to adjust the parameters of the neural network.

[0103] The memory 1320 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1310 may run the program instructions to implement the functions of the image processing device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the image processing method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0104] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: obtaining an input image; segmenting the input image according to structure using a neural network; wherein the neural network is trained in the following manner: obtaining an input training image; processing the input training image to enhance the structural features of the input training image to obtain a structural training image; processing the input training image and the structural training image to extract common structural features of the input training image and the structural training image; segmenting the input training image according to structure at least based on the common structural features to obtain a segmentation result of the input training image; comparing the acquired segmentation result of the input training image with a true segmentation result of the input training image to adjust the parameters of the neural network.

[0105] Below, refer to Fig.14 To describe a device for constructing a neural network for image processing according to an embodiment of the present invention. Fig.14 FIG. 1 shows a block diagram of a construction device 1400 according to an embodiment of the present invention. Fig.14 As shown, the construction device 1400 includes a construction unit 1410, an encoding configuration unit 1420, an extraction layer configuration unit 1430, and a decoding configuration unit 1440. In addition to these units, the construction device 1400 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the construction device 1400 according to the embodiment of the present invention are the same as those described above with reference to Fig. 9 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0106] Fig.14 The construction unit 1410 of the construction device 1400 constructs a neural network including an encoder, a decoder and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder.

[0107] The encoding configuration unit 1420 configures the encoder to encode the input image and the structure image respectively, wherein the structure image is obtained by processing the input image to enhance the structure feature of the input image.

[0108] The extraction layer configuration unit 1430 configures the common feature extraction layer to process the input image and the structure image to extract common structural features of the input image and the structure image.

[0109] The decoding configuration unit 1440 configures the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural feature, so as to obtain a segmentation result of the input image.

[0110] The neural network construction device according to the embodiment of the present invention is capable of constructing a neural network, so that when the neural network is used for image processing, a structural image is obtained by enhancing the structural features of the input image, and the input image is segmented according to the structure based on the common structural features of the input image and the structural image, thereby effectively removing the influence of non-structural features in the input image on the segmentation result, and improving the accuracy of the image segmentation result and subsequent defect detection.

[0111] Below, refer to Fig.15 To describe a device for constructing a neural network for image processing according to an embodiment of the present invention. Fig.15 FIG. 1 shows a block diagram of a construction device 1500 according to an embodiment of the present invention. Fig.15 As shown, the device 1500 may be a computer or a server.

[0112] like Fig.15 As shown, the construction device 1500 includes one or more processors 1510 and a memory 1520. Of course, in addition to this, the construction device 1500 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.15 The components and structures of the construction device 1500 shown are merely exemplary and non-limiting. The construction device 1500 may also have other components and structures as required.

[0113] The processor 1510 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1020 to perform desired functions, which may include: constructing a neural network comprising an encoder, a decoder, and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; configuring the encoder to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image; configuring the common feature extraction layer to process the input image and the structural image to extract common structural features of the input image and the structural image; configuring the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

[0114] The memory 1520 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1510 may run the program instructions to implement the functions of the construction device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the construction method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0115] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions implement the following steps when executed by a processor: constructing a neural network including an encoder, a decoder, and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; configuring the encoder to encode an input image and a structural image respectively, wherein the structural image is obtained by processing the input image to enhance the structural features of the input image; configuring the common feature extraction layer to process the input image and the structural image to extract common structural features of the input image and the structural image; configuring the decoder to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

[0116] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0117] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0118] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0119] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "a", "an", or "the", is not to be construed as limiting the element to the singular.

[0120] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0121] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0122] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0123] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0124] The method invented herein includes one or more actions for implementing the described method. The method and / or action can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0125] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0126] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0127] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0128] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0129] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0130] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0131] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0132] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An image processing method, comprising: Get the input image; Processing the input image to enhance the structural features of the input image and obtain a structural image; Processing the input image and the structural image to extract common structural features of the input image and the structural image; At least according to the common structural features, the input image is segmented according to the structure to obtain a segmentation result of the input image, which includes: The input image and the structural image are segmented respectively according to at least the common structural features by using a neural network, and the segmentation results are fused based on a predetermined rule as a segmentation result of the input image.

2. The method according to claim 1, in, Processing the input image to enhance the structural features of the input image includes: At least one operation of color removal, edge enhancement, texture enhancement, contrast enhancement, noise removal, and resolution adjustment is performed on the input image.

3. The method according to claim 1, in, Processing the input image and the structural image to extract common structural features of the input image and the structural image includes: Inputting the input image and the structure image into the neural network respectively; Using the neural network, respectively encoding the input image and the structural image at a plurality of different step lengths to obtain encoding features of the input image and the structural image at each different step length; According to the encoding features of the input image and the structural image at different step lengths, the common structural features of the input image and the structural image are obtained.

4. The method according to claim 1, in, The method further comprises: The contour of the input image is extracted according to the segmentation result of the input image.

5. The method according to claim 4, in, The method further comprises: The contour of the input image is compared with a standard contour to perform defect detection on the input image.

6. An image processing method, include: Get the input image; Using a neural network to segment the input image according to structure; Wherein, the neural network is trained in the following manner: Get input training image; Processing the input training image to enhance the structural features of the input training image and obtain a structural training image; Processing the input training image and the structure training image to extract common structural features of the input training image and the structure training image; Segmenting the input training image according to the structure at least according to the common structural features to obtain a segmentation result of the input training image; The acquired segmentation result of the input training image is compared with the actual segmentation result of the input training image to adjust the parameters of the neural network.

7. The method according to claim 6, in, Processing the input training image and the structure training image to extract common structural features of the input training image and the structure training image includes: Respectively obtaining structural features of the input training image and the structural training image at different scales; According to the structural features of the input training image and the structural training image at different scales, the common structural features of the input training image and the structural training image are obtained.

8. The method according to claim 6, in, Processing the input training image to enhance the structural features of the input training image, obtaining the structural training image includes: Performing reverse processing on the acquired structural training image to acquire a supervised training image, comparing the acquired supervised training image with the input training image, supervising the acquisition of the structural training image according to the comparison result, and adjusting the acquisition result of the structural training image, wherein the reverse processing is an inverse processing for enhancing the structural features of the input training image; or The acquired structure training image is compared with at least one real structure image, and the acquisition of the structure training image is supervised by using the comparison result, so as to adjust the acquisition result of the structure training image.

9. A method for constructing a neural network for image processing, include: Constructing a neural network comprising an encoder, a decoder, and a common feature extraction layer, wherein the common feature extraction layer is cascaded between the encoder and the decoder; The encoder is configured to encode an input image and a structure image respectively, wherein the structure image is obtained by processing the input image to enhance the structure feature of the input image; The common feature extraction layer is configured to process the input image and the structure image to extract common structural features of the input image and the structure image; The decoder is configured to segment the input image according to the structure based on the encoding result of the encoder and the common structural features to obtain a segmentation result of the input image.

Images