Image processing method and system

Abstract

The invention relates to an image processing method and system. The method comprises the following steps: determining a target object enhanced image of an input image by using a segmentation algorithm, wherein the target object enhanced image comprises an image of which each pixel classified as a target object is enhanced and displayed; and applying an integrogram algorithm to the target object enhanced image to determine a target object positioning image.

Description

[0001] This application is a divisional application based on the invention patent application with application number 202180078471.2, application date December 7, 2021, applicant being Contemporary Amperex Technology Co., Ltd., and invention name being "Image processing method and system". Technical Field

[0002] The present application relates to computer technology, and in particular to image processing technology. Background Art

[0003] Image processing using computers is widely used in various fields. Image processing can be used to improve the visual quality of images, extract the features of specific objects in images, and store and transmit images. To extract the features of specific objects in images, it is necessary to identify and locate the specific objects.

[0004] Therefore, there is a need for an improved technique that can accurately locate a specific object in an image. Summary of the Invention

[0005] In view of the above problems, the present application provides an image processing method and system that can improve the accuracy of locating and segmenting specific targets in an image.

[0006] In a first aspect, the present application provides an image processing method, comprising: using a segmentation algorithm to determine a target object enhanced image of an input image, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced and displayed; and applying an integral image algorithm to the target object enhanced image to determine a target object positioning image.

[0007] In the technical solution of the embodiment of the present application, target object positioning is performed on the input image based on the classification of each pixel in the image as either a target object or a non-target object using a segmentation algorithm. The segmentation and positioning of the target object are combined, and the segmentation algorithm is combined with the integral image algorithm, which can improve the accuracy of target object positioning.

[0008] In some embodiments, determining a target object enhanced image of an input image using a segmentation algorithm further includes: performing feature extraction on the input image to determine a pixel feature map; performing feature extraction on the input image to determine a context feature map; determining contextual association information for each pixel based on the pixel feature map and the context feature map; determining the target object enhanced image based on the contextual association information and the input image, wherein the pixels of the target object enhanced image include weight information, and the weight information is related to whether the pixel belongs to the target object. The segmentation algorithm in this application not only considers pixel-level classification information, but also takes into account the classification information of the context surrounding the target pixel, and determines the final classification result of the target pixel based on the correlation between the target pixel and its context. By incorporating contextual information into the classification algorithm to further improve the accuracy of the classification of the target pixel, a more accurate segmentation of the target object is provided. The target object enhanced image is generated by changing the weight applied to each pixel that is ultimately classified as the target object, so that the target object is enhanced and displayed, thereby providing a more accurate basis for subsequent further positioning processing, which can further improve the accuracy of positioning the target object. The weight can be user-configurable. The change of the weight setting may affect the enhancement effect of the target object in the target object enhanced image, so that the desired target object enhancement effect may be achieved through user setting.

[0009] In some embodiments, applying the integral map algorithm to the target object enhanced image to determine the target object localization image further includes: determining an integral map based on the target object enhanced image; and determining the target object localization image using the integral map. Applying the integral map algorithm to the target object enhanced image in which the target object has been enhanced can further improve the accuracy of locating the target object.

[0010] In some embodiments, determining the integral image based on the target object enhanced image further includes applying a scaling factor to the target object enhanced image. Applying the scaling factor can adjust the amount of data to be processed, thereby accelerating the calculation process and / or improving the accuracy of the integral image according to actual needs.

[0011] In some embodiments, the method further includes: calculating a loss ratio between the target object enhanced image and the input image using a loss function; and feeding the calculated loss ratio back to the segmentation algorithm. The loss ratio between the target object enhanced image output by the segmentation algorithm and the labeled production line image reflects the similarity between the target object enhanced image output by the segmentation algorithm and the original input image. Feeding this loss ratio back to the segmentation algorithm to perform supervised learning training on the segmentation algorithm can improve the accuracy of the segmentation algorithm through continuous training and learning while achieving training fit regression.

[0012] In some embodiments, the method further includes updating the segmentation algorithm based on the loss rate, labeled production line images, or a combination of the two. The segmentation algorithm in this application uses the calculated loss rate, labeled production line images, or a combination of the two as training data to continuously improve the accuracy of the segmentation algorithm in segmenting target objects through supervised learning. Furthermore, because the training data comes from real production lines, it can cover actual needs and can be truly implemented and promoted on production lines.

[0013] In some embodiments, the segmentation algorithm is implemented by a deep convolutional neural network HRNet18. HRNet18 keeps the features at high resolution throughout the entire segmentation algorithm process, which helps to accurately segment the target object. In addition, different branches of the HRNet18 network produce features of different resolutions, and these features interact to obtain information, thereby obtaining high-resolution features containing multi-channel information. In addition, in the case of limited training data, the HRNet18 model is selected to avoid the risk of overfitting, and its smaller structure can speed up the operation of the entire segmentation algorithm.

[0014] In a second aspect, the present application provides an image processing system, comprising: a segmentation module configured to determine a target object enhanced image of an input image using a segmentation algorithm, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced and displayed; a positioning image generation module configured to apply an integral image algorithm to the target object enhanced image to determine a target object positioning image.

[0015] In the technical solution of the embodiment of the present application, target object positioning is performed on the input image based on the classification of each pixel in the image as either a target object or a non-target object using a segmentation algorithm. The segmentation and positioning of the target object are combined, and the segmentation algorithm is combined with the integral image algorithm, which can improve the accuracy of target object positioning.

[0016] In some embodiments, the segmentation module further includes: a feature extraction component configured to perform feature extraction on the input image to determine a pixel feature map and to perform feature extraction on the input image to determine a context feature map; a context component configured to determine contextual association information for each pixel based on the pixel feature map and the context feature map; and an enhanced image generation component configured to determine the target object enhanced image based on the contextual association information and the input image, wherein the pixels of the target object enhanced image include weight information, and the weight information is related to whether the pixel belongs to the target object. The segmentation algorithm in the present application not only considers pixel-level classification information, but also takes into account the classification information of the context surrounding the target pixel, and determines the final classification result of the target pixel based on the correlation between the target pixel and its context. By incorporating contextual information into the classification algorithm to further improve the accuracy of the classification of the target pixel, a more accurate segmentation of the target object is provided. The target object enhanced image is generated by changing the weight applied to each pixel that is finally classified as the target object, so that the target object is enhanced and displayed, thereby providing a more accurate basis for subsequent further positioning processing, which can further improve the accuracy of positioning the target object. The weight can be user-configurable. The change of the weight setting may affect the enhancement effect of the target object in the target object enhanced image, so that the desired target object enhancement effect may be achieved through user setting.

[0017] In some embodiments, the positioning image generation module is further configured to: determine an integral image based on the target object enhanced image; and determine the target object positioning image using the integral image. Applying the integral image algorithm to the target object enhanced image in which the target object is enhanced can further improve the accuracy of locating the target object.

[0018] In some embodiments, the positioning image generation module is further configured to apply a scaling factor to the target object enhanced image. Applying the scaling factor can adjust the amount of data to be processed, thereby accelerating the calculation process and / or improving the accuracy of the integral image according to actual needs.

[0019] In some embodiments, the system further includes a loss rate module configured to: calculate a loss rate between the target object enhanced image and the input image using a loss function; and feed the calculated loss rate back to the segmentation module. The loss rate between the target object enhanced image output by the segmentation algorithm and the labeled production line image reflects the similarity between the target object enhanced image output by the segmentation algorithm and the original input image. Feeding this loss rate back to the segmentation algorithm allows for supervised learning training of the segmentation algorithm, thereby achieving training fit regression while improving the accuracy of the segmentation algorithm through continuous training and learning.

[0020] In some embodiments, the segmentation module is further configured to update the segmentation module based on the loss rate, labeled production line images, or a combination of the two. The segmentation algorithm in this application uses the calculated loss rate, labeled production line images, or a combination of the two as training data, enabling a supervised learning approach to continuously improve the accuracy of the segmentation algorithm in segmenting target objects. Furthermore, because the training data is derived from real production lines, it can address actual needs and be truly implemented and promoted on production lines.

[0021] In a third aspect, the present application provides an image processing system, comprising: a memory storing computer-executable instructions; and a processor coupled to the memory, wherein the computer-executable instructions, when executed by the processor, cause the system to perform the following operations: determine a target object enhanced image of an input image using a segmentation algorithm, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced and displayed; and apply an integral image algorithm to the target object enhanced image to determine a target object positioning image.

[0022] In the technical solution of the embodiment of the present application, target object positioning is performed on the input image based on the classification of each pixel in the image as either a target object or a non-target object using a segmentation algorithm. The segmentation and positioning of the target object are combined, and the segmentation algorithm is combined with the integral image algorithm, which can improve the accuracy of target object positioning.

[0023] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference numerals are used throughout the drawings to represent the same components. In the drawings:

[0025] Figure 1 is a flowchart of an image processing method according to some embodiments of the present application;

[0026] Figure 2 is a flowchart of a method for determining a target object of an input image using a segmentation algorithm according to some embodiments of the present application;

[0027] Figure 3is an effect diagram showing the steps of segmenting a target object according to some embodiments of the present application;

[0028] Figure 4 is a rendering showing the steps of locating a target object according to some embodiments of the present application;

[0029] Figure 5 is a diagram of a network model architecture of a segmentation algorithm for implementing the image processing method of some embodiments of the present application;

[0030] Figure 6 is a functional block diagram of an image processing system according to some embodiments of the present application;

[0031] Figure 7 is a functional block diagram of a segmentation module according to some embodiments of the present application; and

[0032] Figure 8 It is a structural block diagram of a computer system suitable for implementing an image processing system according to some embodiments of the present application. DETAILED DESCRIPTION

[0033] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0035] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0036] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0038] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0039] Image processing using computers is widely used in various fields. Image processing can be used to improve the visual quality of images, extract features of specific targets in images, store and transmit images, and more. To extract features of specific targets in images, it is necessary to identify and locate the specific targets. Extracting specific targets can be used to detect defects in specific targets. For example, in the case of power lithium batteries, by capturing images of lithium batteries produced on a production line and locating target objects such as tabs, it is possible to effectively detect defects such as folded tabs.

[0040] Defects are inevitable during the production of power lithium batteries due to process and equipment limitations. Detecting whether the battery's tabs are folded is crucial throughout every step of the production line. The effectiveness of these inspections ensures the safety of batteries before they leave the factory. However, because tabs comprise only a very small percentage of the entire lithium battery, detecting folded tabs places significant demands on image resolution and accurate tab positioning.

[0041] Some image processing methods include performing double Gaussian difference on the input image, annotating the processed image, building a neural network and model for training and learning, and finally performing data inference based on the model. In this type of technology, the first step is often to input the image data into the model for feature extraction. Therefore, the quality of the input image data (such as resolution, signal-to-noise ratio, etc.) will directly affect the accuracy of the trained model. In the case where the target object is small in size, for example, the lithium battery tab, the double Gaussian difference method cannot effectively locate the target object with a very small volume and extremely high resolution requirements. The image background (non-target object) has a large interference on the target object, resulting in lower target object positioning accuracy and ultimately making it difficult to accurately detect defects in the target object (for example, whether the tab is folded). Therefore, there is a need for an improved technology that can accurately locate target objects that occupy a small area in the image and require high resolution.

[0042] To solve the above problems, the present application provides a technology capable of accurately locating a target object with a small proportion in an image and requiring high resolution. The scheme of the present application can include segmentation of the target object and location of the target object. In the segmentation stage, the present application determines a target object enhanced image of the input image using a segmentation algorithm, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced and displayed. In the location stage, the present application generates an integral image according to the target object enhanced image and generates a target object location image using an integral image algorithm.

[0043] In the technical scheme of the embodiments of the present application, the target object location is performed on the input image based on the classification of each pixel in the image as a target object or a non-target object using the segmentation algorithm, the segmentation and location of the target object are combined, and the segmentation algorithm and the integral image algorithm are combined, which can improve the accuracy of the location of the target object.

[0044] The technical scheme of the embodiments of the present application is suitable for segmentation and location of a target object with a small proportion in an image and requiring high resolution, including but not limited to defect detection of a tab in a lithium battery, identification and annotation of a species observed in the wild, detection and interpretation of human facial micro-expressions, and the like. In the case of a species observed in the wild, the identification of the species is often based on the annotation of a specific pattern or design of the face or a part of the body, and an infrared camera for observation in the wild often cannot provide a clear image with high resolution. Therefore, improving the segmentation and location of the specific pattern or design by the improved segmentation and location algorithm of the present application helps the identification and annotation of the species. Similarly, human face recognition by image capture has been widely applied, and on this basis, interpretation of the micro-expressions of the recognized human face also has wide application. A slight upward tilt of the corners of the mouth, a slight frown on the forehead, and a short spasm of a certain facial muscle are often small in the entire image and difficult to be recognized. Improving the recognition and location of the micro-expressions by the improved segmentation and location algorithm of the present application can improve the accuracy of the interpretation of the micro-expressions.

[0045] Reference Figure 1 which shows a flowchart of a method of processing an image according to some embodiments of the present application, the present application provides a method of processing an image. As shown in Figure 1 , the method includes: in step 105, determining a target object enhanced image of an input image using a segmentation algorithm, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced and displayed. The method includes: in step 110, applying an integral image algorithm to the target object enhanced image to determine a target object location image.

[0046] In some examples, the target object enhanced image includes an image in which every pixel belonging to the target object is enhanced and displayed, while every pixel not belonging to the target object is not enhanced. In some examples, the target object enhanced image may include an image in which pixels belonging to the target object are displayed with enhanced brightness. In some examples, the target object enhanced image may be converted into a mask image. In some examples, applying an integral map algorithm to the target object enhanced image to determine the target object positioning image includes calculating and obtaining an integral map for the target object enhanced image converted into a mask image. An integral map is a method for quickly calculating the sum of a rectangular area in an image. The value of each pixel in the integral map represents the sum of all pixels in the image at the upper left corner of the pixel. Therefore, once the integral map of an image is calculated, the sum of any rectangular area of ​​the image can be quickly calculated. In some examples, the target object positioning image may take the form of a mask map and may be determined based on the integral map. For example, the value of each pixel in the target object positioning image may depend on whether the pixel value in the integral image is 0. If it is 0, the value of the pixel in the target object positioning image is 0. If it is not 0, the value of the pixel in the target object positioning image is 1, where 1 indicates that the pixel belongs to the target object, and 0 indicates that the pixel belongs to the image background or a non-target object.

[0047] In the technical solution of the embodiment of the present application, target object positioning is performed on the input image based on the classification of each pixel in the image as either a target object or a non-target object using a segmentation algorithm. The segmentation and positioning of the target object are combined, and the segmentation algorithm is combined with the integral image algorithm, which can improve the accuracy of target object positioning.

[0048] According to some embodiments of the present application, optionally, further reference is made to Figure 2 and Figure 3 , Figure 2 is a flowchart of a method for determining a target object of an input image using a segmentation algorithm according to some embodiments of the present application, and Figure 3 is a diagram showing the effect of the steps of segmenting a target object according to some embodiments of the present application, Figure 1 Step 102 in the method may further include: step 205, performing feature extraction on the input image to determine a pixel feature map; step 210, performing feature extraction on the input image to determine a context feature map; step 215, determining context association information of each pixel based on the pixel feature map and the context feature map; and step 220 determining the target object enhanced image based on the context association information and the input image, wherein the target object enhanced image is generated by changing the weight applied to each pixel based on the classification of each pixel as belonging to a target object or a non-target object.

[0049] In some examples, step 205 may include inputting the input image into a deep convolutional neural network to perform pixel-level feature extraction on the input image. In some examples, step 205 may include inputting the input image into HRNet18 to generate a feature map for each pixel in the input image. In some examples, the eigenvalue of each pixel in the pixel feature map may represent an initial classification of the pixel as belonging to a target object or a non-target object. In some examples, where the eigenvalue range of a pixel is 0-255, each pixel whose eigenvalue is higher than 128 may be considered to belong to a target object and each pixel whose eigenvalue is lower than 128 may be considered to belong to a non-target object. In some examples, the pixel feature map may be a matrix (pixel representation) representing pixel-level features of the input image after calculation by a deep convolutional neural network, and its image representation may be, for example, Figure 3 As shown in a in . In some examples, step 210 may include inputting the input image into a deep convolutional neural network to perform image block-level feature extraction on the input image. In some examples, step 210 may include inputting the input image into HRNet18 to generate a feature map of a pixel block including a center pixel in the input image. In some examples, the pixel block can be determined by selecting an appropriate convolution kernel n×n, where n is an odd number. As Figure 3As shown in b in the figure, the box in the figure represents the center pixel, and the pixels around the box plus the center pixel represent the pixel block. In some examples, the pixel block feature map can be a matrix (object region representation) representing the pixel block level features after the input image is calculated by the selected convolution kernel through the deep convolutional neural network. In some examples, the pixel block feature map represents the feature value extracted in units of the pixel block including the center pixel. Similarly, the feature value of the pixel block can represent the classification of the pixel block as a target object or a non-target object. In some examples, in the case where the feature value range of the pixel block is 0-255, each pixel block with a feature value higher than 128 can be considered to belong to the target object and each pixel block with a feature value lower than 128 can be considered to belong to a non-target object. In some examples, the pixel block feature value can represent the classification or possibility that the pixels around the center pixel in the pixel block belong to the target object or a non-target object. In this article, the pixel block feature map and the context feature map can be used interchangeably to represent the information of the surrounding pixels and / or context of the center pixel in the pixel block. In some examples, step 215 may include determining contextual association information for each pixel based on the pixel feature map determined in step 205 and the context feature map determined in step 210, wherein the contextual association information indicates the strength of the association between each pixel and the context of the pixel. In some examples, the contextual association information can be obtained by performing matrix multiplication on the pixel feature map determined in step 205 and the context feature map determined in step 210, and applying a softmax function thereto to obtain the contextual association information (pixel region relation) for each pixel. In some examples, when the pixel feature map of the center pixel indicates that the pixel belongs to the target object (non-target object) and the context feature map indicates that the context of the pixel also belongs to the target object (non-target object), the obtained contextual association information for the pixel is strong. In the case where the pixel feature map and the context feature map indicate opposite results (such as when the pixel feature map indicates that the center pixel belongs to the target object and the context feature map indicates that the context pixel of the center pixel belongs to the non-target object), the obtained contextual association information for the pixel is weak. In some examples, step 220 may include determining a final classification of each pixel as belonging to a target object or a non-target object based on the contextual information in step 215 , and generating a target object enhanced image by enhancing each pixel belonging to the target object based on the final classification.In some examples, the context association information (pixel region relation) obtained in step 215 is performed matrix multiplication with the context feature map (object region representation) determined in step 210 to obtain a weighted pixel-level feature map, which is concatenated with the pixel representation determined in step 205 to obtain a final pixel feature map. In some examples, the target object enhanced image is generated by changing the weight applied to each pixel based on the feature value of each pixel in the final pixel feature map, which in turn reflects the classification of the pixel as belonging to the target object or non-target object, for example as shown in c in Figure 3 In some examples, the target object enhanced image can be generated by increasing the weight applied to each pixel whose feature value is higher than 128.

[0050] The segmentation algorithm in this application not only considers the pixel-level classification information, but also takes into account the classification information of the context around the target pixel, determines the final classification result of the target pixel based on the association between the target pixel and its context, and further improves the accuracy of the classification of the target pixel by incorporating the context information into the classification algorithm, thereby providing a more accurate segmentation of the target object. The target object enhanced image is generated by changing the weight applied to each pixel that is finally classified as a target object, so that the target object is enhanced and displayed, thereby providing a more accurate basis for further positioning processing, which can further improve the accuracy of the positioning of the target object. The weight can be user-configurable. The change of the weight setting can affect the enhancement effect of the target object in the target object enhanced image, so that the desired target object enhancement effect can be achieved through user settings.

[0051] According to some embodiments of the present application, the application of the integral image algorithm to the target object enhanced image to determine the target object positioning image further comprises: determining an integral image according to the target object enhanced image; and determining the target object positioning image using the integral image.

[0052] In some examples, the integral image of the target object enhanced image is calculated and obtained, as shown in a and b in Figure 4 In some examples, normalization is performed on the integral image to find a set of parameters that can eliminate the influence of other transformation functions on image transformation using the invariant moments of the image:

[0053] img_normal=img_integral / max(img_integral(:)).

[0054] In some examples, an integral graph algorithm is applied to find the upper left and lower right points as follows:

[0055] x_left,y_left=img_normal>low_thr

[0056] x_right,y_right=img_normal>high_thr.

[0057] Applying the integral graph algorithm to the final classification result obtained by the segmentation algorithm of the present application can accurately locate the target object.

[0058] Applying the integral image algorithm to the target object enhanced image in which the target object has been enhanced and displayed can further improve the accuracy of locating the target object.

[0059] According to some embodiments of the present application, optionally, determining the integral image according to the target object enhanced image further includes applying a scaling factor to the target object enhanced image.

[0060] In some examples, a scaling factor (img_scale) is applied to the target object enhanced image converted into the mask map. In some examples, during the calculation of the integral image, the redundancy length can be expanded in the following ways to ensure positioning accuracy:

[0061] y_extend=(int)((y_right-y_left)*extend_scale_y / 2)

[0062] x_extend=(int)((x_right–x_left)*extend_scale_x / 2).

[0063] In the example of applying the scaling factor, the original image is mapped back based on the scaling factor img_scale according to the following formula to generate the target object positioning image, such as Figure 3 As shown in c:

[0064] x_top=(int)(max((x_left-x_extend),0) / img_scale)

[0065] y_top=(int)(max((y-left-y_extend),0) / img_scale)

[0066] x_bottom=(int)(max((x_left-x_extend),0) / img_scale)

[0067] y_bottom=(int)(max((y-left-y_extend),0) / img_scale).

[0068] By applying the scaling factor, the amount of data to be processed can be adjusted, thereby speeding up the calculation process and / or improving the accuracy of the integral map according to actual needs.

[0069] According to some embodiments of the present application, optionally, the method further includes: using a loss function to calculate a loss rate between the target object enhanced image and the input image; and feeding the calculated loss rate back to the segmentation algorithm.

[0070] In some examples, a cross entropy loss function may be used to calculate a loss ratio between the target object enhanced image and the input image generated in step 220. In some examples, the calculated loss ratio represents the similarity between the target object enhanced image and the original input image.

[0071] The loss ratio between the enhanced image of the target object output by the segmentation algorithm and the labeled production line image reflects the similarity between the enhanced image and the original input image. This loss ratio is fed back to the segmentation algorithm to perform supervised learning training. While achieving training fit regression, the accuracy of the segmentation algorithm can be improved through continuous training and learning.

[0072] According to some embodiments of the present application, optionally, the method further includes: updating the segmentation algorithm based on the loss rate or the labeled production line image or a combination of the two.

[0073] The segmentation algorithm in this application uses the calculated loss rate or labeled production line images, or a combination of these, as training data. This allows for supervised learning to continuously improve the accuracy of the segmentation algorithm in segmenting target objects. Furthermore, because the training data comes from real production lines, it can meet actual needs and be truly implemented and promoted on production lines.

[0074] According to some embodiments of the present application, optionally, further reference is made to Figure 5 , Figure 5 This is a network model architecture diagram of a segmentation algorithm for implementing the image processing method of some embodiments of the present application, wherein the segmentation algorithm is implemented by a deep convolutional neural network HRNet18.

[0075] In some examples, HRNet is a high-resolution network that maintains high-resolution representations throughout the entire process. Starting with a high-resolution subnet as the first stage, it gradually adds high-resolution to low-resolution subnets to form more stages, and connects multi-resolution subnets in parallel. Throughout the process, multi-scale repeated fusion is performed by repeatedly exchanging information on parallel multi-resolution subnets. The key points are estimated through the high-resolution representation output by the network. The network architecture is as follows Figure 4 In some examples, considering whether the segmentation of the target object depends on very high-level semantic information and a limited amount of real training data, a smaller model HRNet18 in the HRNet series is selected to implement the segmentation algorithm of this application.

[0076] HRNet18 maintains high-resolution features throughout the segmentation algorithm, facilitating accurate segmentation of the target object. Furthermore, different branches of the HRNet18 network generate features of varying resolutions, which interact with each other to obtain high-resolution features containing multi-channel information. Furthermore, for limited training data, choosing the HRNet18 model avoids the risk of overfitting, while its smaller size speeds up the entire segmentation algorithm.

[0077] According to some embodiments of the present application, reference Figure 1-Figure 5 The present application provides an image processing method, comprising: performing feature extraction on the input image to determine a pixel feature map; performing feature extraction on the input image to determine a context feature map; determining context-related information of each pixel based on the pixel feature map and the context feature map; determining a tab-enhanced image according to the context-related information and the input image, wherein the tab-enhanced image is generated by changing the weight applied to each pixel based on the classification of each pixel as belonging to the tab or not belonging to the tab; determining an integral map according to the tab-enhanced image, wherein a scaling factor is applied to the tab-enhanced image; and determining a tab positioning image using the integral map, wherein the segmentation algorithm is implemented by HRNet18.

[0078] Reference Figure 6 , which is a functional block diagram of an image processing system according to some embodiments of the present application, and the present application provides an image processing system. Figure 6 As shown, the system includes: a segmentation module 605, which is configured to use a segmentation algorithm to determine a target object enhanced image of an input image, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced; and a positioning image generation module 610, which is configured to apply an integral image algorithm to the target object enhanced image to determine a target object positioning image.

[0079] In the technical solution of the embodiment of the present application, target object positioning is performed on the input image based on the classification of each pixel in the image as either a target object or a non-target object using a segmentation algorithm. The segmentation and positioning of the target object are combined, and the segmentation algorithm is combined with the integral image algorithm, which can improve the accuracy of target object positioning.

[0080] According to some embodiments of the present application, optionally, further reference is made to Figure 7 , Figure 7 is a functional block diagram of a segmentation module according to some embodiments of the present application. The segmentation module 605 further includes: a feature extraction component 705 configured to perform feature extraction on the input image to determine a pixel feature map and perform feature extraction on the input image to determine a context feature map; a context component 710 configured to determine contextual association information of each pixel based on the pixel feature map and the context feature map; and an enhanced image generation component 715 configured to determine the target object enhanced image based on the contextual association information and the input image, wherein the target object enhanced image is generated by changing the weight applied to each pixel based on the classification of each pixel as belonging to a target object or a non-target object.

[0081] The segmentation algorithm in this application not only considers pixel-level classification information, but also takes into account the classification information of the context around the target pixel, and determines the final classification result of the target pixel based on the correlation between the target pixel and its context. By incorporating context information into the classification algorithm to further improve the accuracy of the classification of the target pixel, a more accurate segmentation of the target object is provided. The target object enhanced image is generated by changing the weight applied to each pixel that is finally classified as the target object, so that the target object is enhanced and displayed, thereby providing a more accurate basis for subsequent further positioning processing, which can further improve the accuracy of positioning the target object. The weight can be user-configurable. Changes in the weight setting can affect the enhancement effect of the target object in the target object enhanced image, so that the desired target object enhancement effect can be achieved through user settings.

[0082] According to some embodiments of the present application, optionally, continue to refer to Figure 6 The positioning image generation module 610 is further configured to: determine an integral image according to the target object enhanced image; and determine the target object positioning image using the integral image.

[0083] Applying the integral image algorithm to the target object enhanced image in which the target object has been enhanced and displayed can further improve the accuracy of locating the target object.

[0084] According to some embodiments of the present application, optionally, continue to refer to Figure 6, the positioning image generation module 610 is further configured to apply a scaling factor to the target object enhanced image.

[0085] By applying the scaling factor, the amount of data to be processed can be adjusted, thereby speeding up the calculation process and / or improving the accuracy of the integral map according to actual needs.

[0086] According to some embodiments of the present application, optionally, continue to refer to Figure 6 The system also includes a loss rate module 615, which is configured to: use a loss function to calculate the loss rate between the target object enhanced image and the input image; and feed the calculated loss rate back to the segmentation algorithm to update the segmentation module.

[0087] The loss ratio between the enhanced image of the target object output by the segmentation algorithm and the labeled production line image reflects the similarity between the enhanced image and the original input image. This loss ratio is fed back to the segmentation algorithm to perform supervised learning training. While achieving training fit regression, the accuracy of the segmentation algorithm can be improved through continuous training and learning.

[0088] According to some embodiments of the present application, optionally, continue to refer to Figure 6 , the segmentation module 605 is further configured to update the segmentation module based on the loss rate or the labeled production line images or a combination of the two.

[0089] The segmentation algorithm in this application uses the calculated loss rate or labeled production line images, or a combination of these, as training data. This allows for supervised learning to continuously improve the accuracy of the segmentation algorithm in segmenting target objects. Furthermore, because the training data comes from real production lines, it can meet actual needs and be truly implemented and promoted on production lines.

[0090] According to some embodiments of the present application, reference Figure 6 and Figure 7 , the present application provides an image processing system, comprising:

[0091] The segmentation module 605 includes:

[0092] a feature extraction component 705 configured to perform feature extraction on the input image to determine a pixel feature map and perform feature extraction on the input image to determine a context feature map;

[0093] a context component 710 configured to determine contextual association information for each pixel based on the pixel feature map and the context feature map;

[0094] an enhanced image generating component 715 configured to determine a tab-enhanced image based on the contextual information and the input image, wherein the tab-enhanced image is generated by varying a weight applied to each pixel based on a classification of the pixel as belonging to a tab or not belonging to a tab;

[0095] The positioning image generating module 610 is configured to: determine an integral image according to the tab enhanced image; and determine a tab positioning image using the integral image, wherein a scaling factor is applied to the tab enhanced image.

[0096] Reference Figure 8 , which is a structural block diagram of a computer system suitable for implementing an image processing system according to some embodiments of the present application. Figure 8 As shown, the system includes: a memory 028 having computer executable instructions stored thereon; and a processor 016 coupled to the memory 028, wherein the computer executable instructions, when executed by the processor, cause the system to perform the following operations: determine a target object enhanced image of an input image using a segmentation algorithm, wherein the target object enhanced image includes an image in which each pixel classified as a target object is enhanced; and apply an integral map algorithm to the target object enhanced image to determine a target object positioning image.

[0097] In some examples, Figure 8 A structural block diagram of a computer system 012 suitable for implementing a system for image processing according to some embodiments of the present application is shown. Figure 8 The computer system 012 shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0098] like Figure 8 As shown, computer system 012 is implemented as a general-purpose computing device. Components of computer system 012 may include, but are not limited to, one or more processors or processing units 016, a system memory 028, and a bus 018 that connects various system components (including system memory 028 and processing units 016).

[0099] Bus 018 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0100] The computer system 012 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer system 012, including volatile and non-volatile media, removable and non-removable media.

[0101] System memory 028 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 030 and / or cache memory 032. Computer system 012 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 034 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 Not shown, often called a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 018 via one or more data medium interfaces. The memory 028 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0102] A program / utility 040 having a set (at least one) of program modules 042 may be stored, for example, in memory 028. Such program modules 042 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 042 generally perform the functions and / or methods of the embodiments described herein.

[0103] The computer system 012 can also communicate with one or more external devices 014 (such as a keyboard, a pointing device, a display 024, etc.). In the present invention, the computer system 012 communicates with an external radar device, and can also communicate with one or more devices that enable a user to interact with the computer system 012, and / or communicate with any device that enables the computer system 012 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be performed through an input / output (I / O) interface 022. In addition, the computer system 012 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 020. As shown in the figure, the network adapter 020 communicates with other modules of the computer system 012 via the bus 018. It should be understood that although Figure 7Not shown, other hardware and / or software modules may be used in conjunction with computer system 012, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0104] The processing unit 016 executes various functional applications and data processing by running programs stored in the system memory 028, such as implementing the method flow provided by the embodiment of the present invention.

[0105] The aforementioned computer program may be provided in a computer storage medium, i.e., the computer storage medium is encoded with the computer program, which, when executed by one or more computers, causes the one or more computers to perform the method flow and / or device operations shown in the aforementioned embodiments of the present invention. For example, the method flow provided in the embodiments of the present invention may be performed by the aforementioned one or more processors.

[0106] As time goes by and technology develops, the meaning of "medium" becomes more and more extensive, and the dissemination of computer programs is no longer limited to tangible media, but can also be directly downloaded from the Internet, etc. Any combination of one or more computer-readable media can be used.

[0107] A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0108] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0109] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0110] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A method for processing an image, comprising: Determining whether each pixel in an input image belongs to a target object of the input image using a segmentation algorithm; determining a target object enhanced image of the input image based on the classification of the pixels, wherein the target object enhanced image includes an image in which each pixel classified as the target object is enhanced; as well as applying an integral image algorithm to the target object enhanced image to determine a target object localization image; The step of using a segmentation algorithm to determine whether each pixel in the input image belongs to a target object in the input image includes: Inputting the input image into a deep convolutional neural network to perform pixel-level feature extraction on the input image to obtain a feature value for each pixel in the input image; Based on the feature value of the pixel, it is determined whether the pixel belongs to the target object.

2. The method according to claim 1, wherein Determining the target object enhanced image of the input image based on the classification of the pixels further includes: performing feature extraction on the input image to determine a pixel feature map; performing feature extraction on the input image to determine a context feature map; Determining contextual association information for each pixel based on the pixel feature map and the context feature map; The target object enhanced image is determined according to the context association information and the input image, wherein pixels of the target object enhanced image include weight information, and the weight information is related to whether the pixel belongs to the target object.

3. The method according to claim 2, wherein The determining the target object enhanced image according to the context association information and the input image further comprises: Performing matrix multiplication on the contextual feature map based on the contextual association information to obtain a weighted pixel-level feature map; Obtaining a final pixel feature map according to the weighted pixel-level feature map and the pixel feature map; Based on the feature value of each pixel in the final pixel feature map, the weight applied to each pixel is changed to generate the target object enhanced image.

4. The method according to any one of claims 1 to 3, wherein Applying an integral image algorithm to the target object enhanced image to determine a target object positioning image further comprises: determining an integral image according to the target object enhanced image; and The target object positioning image is determined using the integral image.

5. The method according to claim 4, wherein Determining an integral image based on the target object enhanced image further includes applying a scaling factor to the target object enhanced image.

6. The method according to any one of claims 1 to 5, wherein The method further comprises: Calculating a loss ratio between the target object enhanced image and the input image using a loss function; and The calculated loss rate is fed back to the segmentation algorithm to update the segmentation algorithm.

7. The method according to any one of claims 1 to 6, wherein The segmentation algorithm is implemented by the deep convolutional neural network HRNet18.

8. An image processing system, comprising: a segmentation module configured to determine whether each pixel in an input image belongs to a target object of the input image using a segmentation algorithm; determining a target object enhanced image of the input image based on the classification of the pixels, wherein the target object enhanced image includes an image in which each pixel classified as the target object is enhanced; a positioning image generation module configured to apply an integral image algorithm to the target object enhanced image to determine a target object positioning image; The segmentation module is further configured to input the input image into a deep convolutional neural network to perform pixel-level feature extraction on the input image to obtain a feature value of each pixel in the input image; Based on the feature value of the pixel, it is determined whether the pixel belongs to the target object.

9. The system according to claim 8, wherein The segmentation module also includes: a feature extraction component configured to perform feature extraction on the input image to determine a pixel feature map and to perform feature extraction on the input image to determine a context feature map; a context component configured to determine contextual association information of each pixel based on the pixel feature map and the context feature map; An enhanced image generation component is configured to determine the target object enhanced image based on the context association information and the input image, wherein pixels of the target object enhanced image include weight information, and the weight information is related to whether the pixel belongs to the target object.

10. The system according to claim 9, wherein The enhanced image generation component is further configured to: Performing matrix multiplication on the contextual feature map based on the contextual association information to obtain a weighted pixel-level feature map; Obtaining a final pixel feature map according to the weighted pixel-level feature map and the pixel feature map; Based on the feature value of each pixel in the final pixel feature map, the weight applied to each pixel is changed to generate the target object enhanced image.

11. The system according to any one of claims 8 to 10, wherein: The positioning image generation module is further configured to: determining an integral image according to the target object enhanced image; and The target object positioning image is determined using the integral image.

12. The system according to claim 11, wherein The positioning image generation module is further configured to apply a scaling factor to the target object enhanced image.

13. The system according to any one of claims 8 to 12, wherein: The system also includes a loss rate module configured to: Calculating a loss ratio between the target object enhanced image and the input image using a loss function; and The calculated loss rate is fed back to the segmentation algorithm to update the segmentation algorithm.

14. An image processing system, comprising: a computer executable instruction memory stored thereon; as well as a processor coupled to the memory, wherein the computer-executable instructions, when executed by the processor, cause the system to: Determining whether each pixel in an input image belongs to a target object of the input image using a segmentation algorithm; determining a target object enhanced image of the input image based on the classification of the pixels, wherein the target object enhanced image includes an image in which each pixel classified as the target object is enhanced; as well as applying an integral image algorithm to the target object enhanced image to determine a target object localization image; The step of using a segmentation algorithm to determine whether each pixel in the input image belongs to a target object in the input image includes: Inputting the input image into a deep convolutional neural network to perform pixel-level feature extraction on the input image to obtain a feature value for each pixel in the input image; Based on the feature value of the pixel, it is determined whether the pixel belongs to the target object.

Images