Image processing method, device and storage medium

Abstract

The present application discloses an image processing method, device and storage medium. The method obtains a line scan image from line scan data; then downsamples each image block in the line scan image based on data transformation distribution; and inputs the sampled image into a preset backbone network to obtain corresponding image features; then performs self-supervised training based on the image features to obtain a target backbone network; and then uses the target backbone network to process the detection task corresponding to the image to be processed, thereby realizing the processing process of the line scan image based on self-supervised training. Since the image distribution characteristics of the line scan image are used for targeted sampling, and self-supervised learning is performed based on the sampled image, there is no need to use labeled data, which avoids the occurrence of labeling errors and improves the accuracy of line scan image detection.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] In the intelligent industrial quality inspection system, the design of visual algorithms is a crucial link. Since some industrial products are large in size, such as textiles and large consumer electronic products, the first thing to do for automated defect detection of such products is to efficiently obtain imaging information on the product surface. Therefore, in practical applications, line scan cameras are mostly used for efficient imaging scanning. Further, defects on the product surface can be detected based on the images obtained by imaging scanning. However, since defects are generally uncommon types or small in size, how to perform defect detection based on line scan images has become a problem.

[0003] Generally, the process of defect detection on line scan images can adopt the target detection method, that is, the training images annotated with defect level detection boxes are used to train the model to perform target detection.

[0004] However, the training process of the target detection model using labeled data requires a lot of training to find all possible defect types. Since the defect types may include a few uncommon types or defects with very small areas, and the defects require labelers to find the location of the defects from large-area line scan images or a large number of image blocks and provide detection box annotations, the collection of defects will be more difficult and prone to problems such as missed annotations, which will affect the accuracy of line scan image detection in the detection task. Summary of the invention

[0005] In view of this, the present application provides an image processing method that can effectively improve the accuracy of line scan image detection in detection tasks.

[0006] The first aspect of the present application provides an image processing method, which can be applied to a system or program including an image processing function in a terminal device, and specifically includes:

[0007] Acquire a line scan image from the line scan data, wherein the line scan image includes a plurality of image blocks;

[0008] Downsampling each image block in the line scan image based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block;

[0009] Inputting the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, the preset backbone network comprising a first feature extraction network and a second feature extraction network, the first feature extraction network being used to obtain the first image feature according to the first sampling image, the second feature extraction network being used to obtain the second image feature according to the second sampling image, and the weight value of the network parameter corresponding to the first feature extraction network being shared with the weight value of the network parameter corresponding to the second feature extraction network;

[0010] Predicting the first image feature to obtain a prediction result, and performing self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network;

[0011] In response to the input of the image to be processed in the line scan data, the target backbone network is used to process the detection task corresponding to the image to be processed.

[0012] Optionally, in some possible implementations of the present application, downsampling each image block in the line scan image based on the data transformation distribution to obtain a first sampled image and a second sampled image corresponding to each image block includes:

[0013] Dividing each of the image blocks in the line scan image to obtain a plurality of image grids corresponding to each of the image blocks;

[0014] Determining a downsampling method based on the data transformation distribution;

[0015] Adjacent image grids are sampled into two different images according to the downsampling method to obtain the first sampled image and the second sampled image.

[0016] Optionally, in some possible implementations of the present application, the method further includes:

[0017] Determine a sampling direction corresponding to the downsampling method;

[0018] Based on the sampling direction, a plurality of image grids of the line scan image in a corresponding direction are sampled to obtain a sampling unit;

[0019] Adjacent sampling units are sampled into two different images respectively to obtain the first sampling image and the second sampling image.

[0020] Optionally, in some possible implementations of the present application, determining a sampling direction corresponding to the downsampling mode includes:

[0021] Determine the line scan object corresponding to the line scan data;

[0022] Acquire texture information configured for the line scan object;

[0023] Texture repeatability is detected based on the texture information to determine a sampling direction corresponding to the downsampling method.

[0024] Optionally, in some possible implementations of the present application, the detecting of texture repeatability based on the texture information to determine a sampling direction corresponding to the downsampling method includes:

[0025] Acquire functional area information corresponding to the line scan object;

[0026] Dividing the texture information based on the functional area information to obtain regional texture;

[0027] Texture repeatability is detected according to the regional textures respectively to determine the sampling direction corresponding to each regional texture.

[0028] Optionally, in some possible implementations of the present application, predicting the first image feature to obtain a prediction result, and performing self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network, includes:

[0029] Inputting the first image feature into a fully connected layer for prediction to obtain the prediction result;

[0030] Approximating the prediction result to the second image feature to generate supervision information;

[0031] Based on the supervision information, the preset backbone network is self-supervised trained to obtain a target backbone network, and the supervision information will not be gradient-transmitted back to the second feature extraction network.

[0032] Optionally, in some possible implementations of the present application, inputting the first sampled image and the second sampled image into a preset backbone network to obtain a first image feature corresponding to the first sampled image and a second image feature corresponding to the second sampled image includes:

[0033] Inputting the first sampled image into the first feature extraction network in the preset backbone network, wherein the first feature extraction network comprises a plurality of first feature extraction blocks, and the first feature extraction block comprises a first residual unit and a second residual unit connected in series;

[0034] Performing channel down-sampling on the first sampled image based on the first residual unit to obtain first feature information;

[0035] Performing a convolution operation on the first feature information to obtain a first convolution feature;

[0036] Performing channel upsampling on the first convolution feature based on the second residual unit to obtain second feature information;

[0037] Performing a residual connection between the second feature information and the first sampled image to obtain the first image feature;

[0038] Inputting the second sampling image into the second feature extraction network in the preset backbone network, the second feature extraction network comprising a plurality of second feature extraction blocks, the second feature extraction block comprising a third residual unit and a fourth residual unit connected to each other;

[0039] Performing channel down-sampling on the second sampled image based on the third residual unit to obtain third feature information;

[0040] Performing a convolution operation on the third feature information to obtain a second convolution feature;

[0041] Performing channel upsampling on the second convolutional feature based on the fourth residual unit to obtain fourth feature information;

[0042] Perform a residual connection on the fourth feature information and the second sampling image to obtain the second image feature.

[0043] A second aspect of the present application provides an image processing device, comprising:

[0044] An acquisition unit, used for acquiring a line scan image in the line scan data, wherein the line scan image includes a plurality of image blocks;

[0045] A sampling unit, configured to downsample each image block in the line scan image based on data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block;

[0046] a processing unit, configured to input the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, wherein the preset backbone network includes a first feature extraction network and a second feature extraction network, wherein the first feature extraction network is configured to obtain the first image feature according to the first sampling image, and the second feature extraction network is configured to obtain the second image feature according to the second sampling image, and the weight value of the network parameter corresponding to the first feature extraction network is shared with the weight value of the network parameter corresponding to the second feature extraction network;

[0047] The processing unit is further configured to predict the first image feature to obtain a prediction result, so as to perform self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network;

[0048] The processing unit is also used to respond to the input of the image to be processed in the line scan data and use the target backbone network to process the detection task corresponding to the image to be processed.

[0049] Optionally, in some possible implementations of the present application, the sampling unit is specifically used to divide each of the image blocks in the line scan image to obtain a plurality of image grids corresponding to each of the image blocks;

[0050] The sampling unit is specifically used to determine a downsampling method based on the data transformation distribution;

[0051] The sampling unit is specifically configured to sample adjacent image grids into two different images respectively according to the downsampling method, so as to obtain the first sampling image and the second sampling image.

[0052] Optionally, in some possible implementations of the present application, the sampling unit is specifically used to determine a sampling direction corresponding to the downsampling method;

[0053] The sampling unit is specifically used to sample a plurality of image grids of the line scan image in a corresponding direction based on the sampling direction to obtain a sampling unit;

[0054] The sampling unit is specifically used to sample adjacent sampling units into two different images respectively to obtain the first sampling image and the second sampling image.

[0055] Optionally, in some possible implementations of the present application, the sampling unit is specifically used to determine the line scan object corresponding to the line scan data;

[0056] The sampling unit is specifically used to obtain texture information configured for the line scan object;

[0057] The sampling unit is specifically used to detect texture repeatability based on the texture information to determine a sampling direction corresponding to the downsampling method.

[0058] Optionally, in some possible implementations of the present application, the sampling unit is specifically used to obtain functional area information corresponding to the line scan object;

[0059] The sampling unit is specifically used to divide the texture information based on the functional area information to obtain regional texture;

[0060] The sampling unit is specifically used to detect texture repeatability according to the regional textures respectively, so as to determine the sampling direction corresponding to each of the regional textures.

[0061] Optionally, in some possible implementations of the present application, the processing unit is specifically configured to input the first image feature into a fully connected layer for prediction to obtain the prediction result;

[0062] The processing unit is specifically configured to bring the prediction result and the second image feature close to each other to generate supervision information;

[0063] The processing unit is specifically used to perform self-supervisory training on the preset backbone network based on the supervision information to obtain a target backbone network, and the supervision information will not perform gradient feedback to the second feature extraction network.

[0064] Optionally, in some possible implementations of the present application, the processing unit is specifically configured to input the first sampled image into the first feature extraction network in the preset backbone network, the first feature extraction network includes a plurality of first feature extraction blocks, and the first feature extraction block includes a first residual unit and a second residual unit connected to each other;

[0065] The processing unit is specifically configured to perform channel down-sampling on the first sampled image based on the first residual unit to obtain first feature information;

[0066] The processing unit is specifically configured to perform a convolution operation on the first feature information to obtain a first convolution feature;

[0067] The processing unit is specifically configured to perform channel upsampling on the first convolution feature based on the second residual unit to obtain second feature information;

[0068] The processing unit is specifically configured to perform a residual connection between the second feature information and the first sampled image to obtain the first image feature;

[0069] The processing unit is specifically configured to input the second sampling image into the second feature extraction network in the preset backbone network, wherein the second feature extraction network comprises a plurality of second feature extraction blocks, and the second feature extraction block comprises a third residual unit and a fourth residual unit connected to each other.

[0070] The processing unit is specifically configured to perform channel down-sampling on the second sampled image based on the third residual unit to obtain third feature information;

[0071] The processing unit is specifically configured to perform a convolution operation on the third feature information to obtain a second convolution feature;

[0072] The processing unit is specifically configured to perform channel upsampling on the second convolution feature based on the fourth residual unit to obtain fourth feature information;

[0073] The processing unit is specifically configured to perform a residual connection between the fourth feature information and the second sampling image to obtain the second image feature.

[0074] The third aspect of the present application provides a computer device, comprising: a memory, a processor and a bus system; the memory is used to store program code; the processor is used to execute the image processing method described in the first aspect or any one of the first aspects according to the instructions in the program code.

[0075] The fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the image processing method described in the first aspect or any one of the first aspects.

[0076] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method provided in the first aspect or various optional implementations of the first aspect.

[0077] It can be seen from the above technical embodiments that the embodiments of the present application have the following advantages:

[0078] By acquiring a line scan image in line scan data, the line scan image includes a plurality of image blocks; then downsampling each image block in the line scan image based on data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; and inputting the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, the preset backbone network includes a first feature extraction network and a second feature extraction network, the first feature extraction network is used to obtain a first image feature according to the first sampling image, and the second feature extraction network is used to obtain a second image feature according to the second sampling image, The weights of the first feature extraction network and the second feature extraction network are shared; then the first image feature is predicted to obtain a prediction result, and the preset backbone network is self-supervised trained based on the prediction result and the second image feature to obtain a target backbone network; then in response to the input of the image to be processed in the line scan data, the target backbone network is used to process the detection task corresponding to the image to be processed, thereby realizing the processing process of the line scan image based on self-supervised training. Since targeted sampling is performed using the image distribution characteristics of the line scan image, and self-supervised learning is performed based on the sampled image, there is no need to use labeled data, which avoids the occurrence of labeling errors and improves the accuracy of line scan image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present application or the technical embodiments in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0080] Figure 1 A diagram of the network architecture that runs the image processing system;

[0081] Figure 2 A flow chart of an image processing process provided by an embodiment of the present application;

[0082] Figure 3 A flowchart of an image processing method provided in an embodiment of the present application;

[0083] Figure 4 A schematic diagram of a scene of an image processing method provided in an embodiment of the present application;

[0084] Figure 5 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0085] Figure 6 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0086] Figure 7 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0087] Figure 8 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0088] Fig. 9 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0089] Fig.10 A schematic diagram of another image processing method provided in an embodiment of the present application;

[0090] Fig.11 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0091] Fig.12 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0092] Fig.13 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The embodiments of the present application provide an image processing method and a related device, which can be applied to a system or program including an image processing function in a terminal device, by acquiring a line scan image in line scan data, wherein the line scan image includes a plurality of image blocks; then downsampling each image block in the line scan image based on a data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; and inputting the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, wherein the preset backbone network includes a first feature extraction network and a second feature extraction network, wherein the first feature extraction network is used to obtain a first image feature according to the first sampling image, and the second feature extraction network is used to obtain a first image feature according to the first sampling image, and a second image feature according to the second sampling image. The second feature extraction network is used to obtain the second image feature according to the second sampling image, and the weights of the first feature extraction network and the second feature extraction network are shared; then the first image feature is predicted to obtain a prediction result, and the preset backbone network is self-supervised trained based on the prediction result and the second image feature to obtain a target backbone network; then in response to the input of the image to be processed in the line scan data, the target backbone network is used to process the detection task corresponding to the image to be processed, thereby realizing the processing process of the line scan image based on self-supervised training. Since the image distribution characteristics of the line scan image are used for targeted sampling, and self-supervised learning is performed based on the sampled image, there is no need to use labeled data, which avoids the occurrence of labeling errors and improves the accuracy of line scan image detection.

[0094] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0095] It is understandable that in the specific implementation of the present application, when related image data such as line scan data is applied to specific products or technologies, it is necessary to obtain the permission or consent of relevant personnel, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0096] It should be understood that the image processing method provided in the present application can be applied to a system or program including an image processing function in a terminal device, such as a defect detection application. Specifically, the image processing system can be run on a system such as Figure 1 In the network architecture shown in Figure 1 As shown in FIG. 1 , it is a network architecture diagram of the image processing system. As can be seen from the figure, the image processing system can provide image processing processes with multiple information sources. The data source of the information source is a line scan camera. By acquiring the line scan image obtained by the line scan camera, the server performs model training based on the line scan image to perform the detection task. It can be understood that Figure 1 A variety of terminal devices are shown in FIG. 1 . The terminal devices may be computer devices. In actual scenarios, more or fewer types of terminal devices may be involved in the image processing process. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in an actual scenario, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0097] In this embodiment, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0098] It can be understood that the above-mentioned image processing system can run on a personal mobile terminal, for example: as an application such as a defect detection application, it can also run on a server, and it can also run on a third-party device to provide image processing to obtain the image processing results of the information source; the specific image processing system can be run in the above-mentioned device in the form of a program, and it can also be run as a system component in the above-mentioned device, and it can also be used as a cloud service program.

[0099] In the intelligent industrial quality inspection system, the design of visual algorithms is a crucial link. Since some industrial products are large in size, such as textiles and large consumer electronic products, the first thing to do for automated defect detection of such products is to efficiently obtain imaging information on the product surface. Therefore, in practical applications, line scan cameras are mostly used for efficient imaging scanning. Further, defects on the product surface can be detected based on the images obtained by imaging scanning. However, since defects are generally uncommon types or small in size, how to perform defect detection based on line scan images has become a problem.

[0100] Generally, the process of defect detection on line scan images can adopt the target detection method, that is, the training images annotated with defect level detection boxes are used to train the model to perform target detection.

[0101] However, the training process of the target detection model using labeled data requires a lot of training to find all possible defect types. Since the defect types may include a few uncommon types or defects with very small areas, and the defects require labelers to find the location of the defects from large-area line scan images or a large number of image blocks and provide detection box annotations, the collection of defects will be more difficult and prone to problems such as missed annotations, which will affect the accuracy of line scan image detection in the detection task.

[0102] In order to solve the above problems, this application proposes an image processing method, which is applied to Figure 2 The image processing flow framework shown in FIG. Figure 2 As shown, it is a flow chart of an image processing provided by an embodiment of the present application. Through the line scan image sent by the terminal, the server performs a sampling process for the characteristics of the line scan image, and performs self-supervised training based on the sampled image, so as to use the trained backbone network to perform the detection task. That is, some typical characteristics of line scan imaging are used to construct a self-supervised training framework, avoiding the high labor cost caused by the collection and annotation of a large number of defect images.

[0103] It can be understood that the method provided in the present application can be written as a program to serve as a processing logic in a hardware system, or as an image processing device to implement the above processing logic in an integrated or external manner. As an implementation method, the image processing device obtains a line scan image in the line scan data, and the line scan image contains multiple image blocks; then downsamples each image block in the line scan image based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; and inputs the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image. The preset backbone network includes a first feature extraction network and a second feature extraction network. The first feature extraction network is used to obtain the first image feature based on the first sampling image, and the second feature extraction network is used to obtain the first image feature based on the second sampling image. To the second image feature, the weights of the first feature extraction network and the second feature extraction network are shared; then the first image feature is predicted to obtain a prediction result, and the preset backbone network is self-supervised trained based on the prediction result and the second image feature to obtain a target backbone network; and then in response to the input of the image to be processed in the line scan data, the target backbone network is used to process the detection task corresponding to the image to be processed, thereby realizing the processing process of the line scan image based on self-supervised training. Since the image distribution characteristics of the line scan image are used for targeted sampling, and self-supervised learning is performed based on the sampled image, there is no need to use labeled data, which avoids the occurrence of labeling errors and improves the accuracy of line scan image detection.

[0104] The embodiments provided in the embodiments of the present application relate to computer vision technology of artificial intelligence, which is specifically described by the following embodiments:

[0105] Combined with the above process architecture, the following will introduce the image processing method in this application. Please refer to Figure 3 , Figure 3 A flowchart of an image processing method provided in an embodiment of the present application, the processing method can be executed by a server or a terminal, and the embodiment of the present application at least includes the following steps:

[0106] 301. Obtain a line scan image from the line scan data.

[0107] In this embodiment, the line scan data is image data obtained based on a line scan camera. The data can be collected in real time or stored in the background. The specific data source depends on the actual scenario.

[0108] It can be understood that since the line scan camera contains a single row of pixels and is a two-dimensional image constructed by pixel lines one by one, the line scan image contains multiple image blocks. Specifically, the process of acquiring line scan data is as follows: Figure 4 As shown, Figure 4 A scene diagram of an image processing method provided for an embodiment of the present application; when constructing a line scan image, the camera and the object need to maintain relative motion, usually along a conveyor belt or a rotating axis. When the object moves in front of the camera, the camera will capture a new line of pixels. The software on the visual processor or image acquisition card will store each pixel line, and then reorganize the pixel data into a final two-dimensional image. The image acquisition process is better at capturing images of fast-moving discrete components on a conveyor belt, or constructing images of very large objects. In industrial AI quality inspection, line scan cameras are mainly used to process defect imaging on parts with large surface areas.

[0109] This embodiment is applied to defect detection of industrial parts, such as Figure 5 As shown, Figure 5 A schematic diagram of a scenario of another image processing method provided in an embodiment of the present application; the figure shows that a line scan image of an industrial part is used as input and the defects existing in the image to be detected are output, namely A1 and A2 in the figure.

[0110] 302. Downsample each image block in the line scan image based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block.

[0111] In this embodiment, by performing targeted sampling based on the image characteristics of the line scan image, the data processing capacity can be increased while the line scan image characteristics are not lost.

[0112] Specifically, considering that a complete and huge image will be generated on the product surface after line scanning imaging, such as Figure 6 As shown, Figure 6 A scene diagram of another image processing method provided in an embodiment of the present application; therefore, the line scan image can be cut using a fixed step size and image size, thereby forming a series of slightly overlapping points, each point corresponding to an image, and in order to apply the self-supervised pre-training method proposed in this embodiment, the original image can usually be made into relatively small image blocks, thereby generating thousands of different points.

[0113] After generating different points as above, for the same product, the images at different points may be partially similar (such as overlap between adjacent image blocks, or corresponding to areas with high consistency on the product surface); but they can all be distinguished by the product structure characteristics of the context; and for the same point, the surface structure of different products is basically the same, but due to the influence of exposure, product surface polishing, etc., there will be large differences in imaging, so this embodiment uses downsampling to process the image.

[0114] It is understandable that since line scan imaging often has a very high resolution (mainly to capture defects clearly), this leads to a large redundancy between pixels, especially in areas where the surface texture does not change much. When the model receives such images for training, due to the limited receptive field size, there may be only some homogeneous areas within the corresponding receptive field, which is not conducive to the modeling of contextual information in a larger range. Therefore, the data transformation distribution in the downsampling process of this embodiment utilizes two characteristics of line scan imaging; first, there are a large number of points, and at the same time, there is a natural correspondence between the image and the point. The imaging of the same point on different products has natural differences, and the imaging of the same product at different points also has some similarities. By training the model to predict the point corresponding to the input image, it is possible to learn feature representations that are invariant to natural image differences and sensitive to local changes, thereby assisting the learning of downstream tasks; second, the image resolution is high, and there is a high redundancy between pixels. By learning to make the image features of different forms of downsampling as close as possible, the model can be encouraged to learn scale-invariant feature representations.

[0115] In combination with the above analysis, the downsampling process of this embodiment can adopt chessboard sampling, that is, first divide each image block in the line scan image to obtain multiple image grids corresponding to each image block; then determine the downsampling method based on the data transformation distribution; and according to the downsampling method, sample adjacent image grids into two different images respectively to obtain a first sampled image and a second sampled image.

[0116] Specifically, the above sampling process is as follows: Figure 7 As shown, Figure 7 A schematic diagram of another image processing method provided in an embodiment of the present application; for an input image, the image can be divided into a series of grids based on image blocks, and adjacent grids are placed in corresponding positions of two different images. Figure 7 , that is, the dark image blocks are recombined into one image, and the light image blocks are combined into another image. Due to the high imaging resolution, the two images have high structural similarity. At the same time, due to the downsampling operation, there are large local differences. Through self-supervised learning, the model is encouraged to make the features of the two images as similar as possible, so that the model can learn a wider range of contextual information.

[0117] In addition, during the downsampling transformation operation, the processing can use chessboard-type reorganization, but considering that some point images will have a large range of repetitive textures in a certain direction, strip reorganization can also be used. In the strip reorganization process, the sampling direction needs to be considered. Therefore, for the sampling process, first determine the sampling direction corresponding to the downsampling method; then based on the sampling direction, the sampling unit in the corresponding direction, the sampling unit contains multiple image grids; and sample the adjacent sampling units into two different images respectively to obtain the first sampling image and the second sampling image, thereby improving the sampling efficiency.

[0118] Furthermore, for line scan images with unclear sampling directions, we can first determine the line scan object corresponding to the line scan data; then obtain texture information configured for the line scan object; and then perform texture repeatability detection based on the texture information to determine the sampling direction corresponding to the downsampling method, thereby improving the accuracy of the sampling direction.

[0119] In another possible scenario, considering that different functional areas of a part may have different textures, sampling can be performed separately. That is, first, the functional area information corresponding to the line scan object is obtained; then the texture information is divided based on the functional area information to obtain the regional texture; and then the texture repeatability is detected according to the regional texture to determine the sampling direction corresponding to each regional texture, thereby improving the adaptability of the sampling process to different part products.

[0120] Specifically, for the above-mentioned strip reorganization sampling process, it is Figure 7 T2 and T3 shown encourage images to model contextual information in both horizontal and vertical directions by distributing adjacent strip regions into two images each.

[0121] It can be understood that the downsampling process adopted in this embodiment is used to encourage the model to learn a wider range of context information. The specific distribution adopted depends on the actual scenario and is therefore not limited to a regular downsampling scheme.

[0122] Through the downsampling strategy based on image patches, the model is encouraged to utilize the image context information in the subsequent steps. The backbone network trained by self-supervision can be directly used for downstream tasks. The pre-trained model will be used as the backbone network of the detection model, thereby reducing the need for subsequent labeled samples.

[0123] 303. Input the first sampled image and the second sampled image into a preset backbone network to obtain a first image feature corresponding to the first sampled image and a second image feature corresponding to the second sampled image.

[0124] In this embodiment, the preset backbone network is a feature extraction network for supporting self-supervised training, which includes a first feature extraction network and a second feature extraction network. The first feature extraction network is used to obtain first image features based on a first sampling image, and the second feature extraction network is used to obtain second image features based on a second sampling image, and the weights of the first feature extraction network and the second feature extraction network are shared.

[0125] In a possible scenario, both the first feature extraction network and the second feature extraction network (Encoder) use ResNet50 as the basic network. The specific feature extraction process is as follows: Figure 8 As shown, Figure 8 A scene diagram of another image processing method provided in an embodiment of the present application; the figure shows that the preset backbone network is composed of 5 feature extraction blocks, each block is composed of residual units connected to each other, and for each residual unit, channel downsampling is first performed, and then 3x3 convolution is used for feature transformation, and then the channel is restored to the original channel size through channel upsampling, and finally the output feature is obtained through residual connection with the input. Therefore, for the feature extraction process, the first sampled image is first input into the first feature extraction network in the preset backbone network, the first feature extraction network includes multiple first feature extraction blocks, and the first feature extraction block includes a first residual unit and a second residual unit connected to each other; then, based on the first residual unit, the first sampled image is channel downsampled to obtain the first feature information; and the first feature information is convolved to obtain the first convolution feature; and then, based on the second residual unit, the first convolution feature is channel upsampled to obtain the second feature information, and the second feature information is residually connected to the first sampled image to obtain the first image feature.

[0126] Correspondingly, for the second feature extraction network, the second sampling image is first input into the second feature extraction network in the preset backbone network, the second feature extraction network includes multiple second feature extraction blocks, and the second feature extraction block includes a third residual unit and a fourth residual unit connected front and back; then, based on the third residual unit, the second sampling image is channel-downsampled to obtain third feature information; and the third feature information is convolved to obtain second convolution features; then, based on the fourth residual unit, the second convolution features are channel-upsampled to obtain fourth feature information; and the fourth feature information is residually connected with the second sampling image to obtain second image features.

[0127] In addition, each feature extraction block uses Maxpool or convolution with a step size of 2 to achieve spatial downsampling, which increases the network receptive field and local translation invariance. At the same time, the number of reference channels of each block increases as the network level deepens, so that richer semantic information can be extracted.

[0128] It is understandable that the backbone network structure in this embodiment is not limited to the ResNet network structure, but may also be other network structures that can extract high-resolution feature maps with strong semantic information, which is not limited here.

[0129] Through the self-supervised pre-training method proposed in this embodiment, images at the same point will be encouraged to be as similar as possible, thereby encouraging the model to distinguish between normal texture changes and changes caused by environmental factors such as exposure. At the same time, images at different points will be distinguished, which will encourage the model to learn to understand the structure of the product surface and thus learn rich contextual information. The robustness to imaging conditions and normal textures, as well as the ability to model contextual information, will improve the performance of downstream inspection tasks, thereby avoiding the collection and labeling of a large number of defective samples.

[0130] 304. Predict the first image feature to obtain a prediction result, and perform self-supervisory training on a preset backbone network based on the prediction result and the second image feature to obtain a target backbone network.

[0131] In this embodiment, the process of self-supervised training is to design the data transformation distribution τ in order to make full use of the characteristics of line scan data, and encourage the model to learn to have non-deformation feature representations such as natural exposure of the image and natural texture changes on the surface of the product. At the same time, through the downsampling transformation based on the image block, the model is encouraged to learn to capture a wider range of contextual information. These are all beneficial features for downstream detection tasks.

[0132] Specifically, the self-supervised training process first inputs the first image feature into the fully connected layer for prediction to obtain a prediction result; then the prediction result is close to the second image feature to generate supervision information; and based on the supervision information, the preset backbone network is self-supervised trained to obtain the target backbone network, and the supervision information will not be gradient-transmitted back to the second feature extraction network.

[0133] In one possible scenario, the model architecture for self-supervised training is as follows Fig. 9 As described above, for an input line scan image, two different image transformations are sampled from a data transformation distribution τ, and two transformed sampled images, namely x1 and x2, are obtained accordingly, and are sent to two weight-sharing backbone networks respectively, so as to extract the first image feature z1 and the second image feature z2, and z1 will be further obtained through a fully connected layer Predictor to obtain p1, and finally it will force p1 and z2 to be as close as possible, and use this as a supervisory signal to train the entire network, as shown in the following formula:

[0134]

[0135] Among them, p1 is the prediction result, and z2 is the second image feature.

[0136] Furthermore, in order to avoid the above method from falling into a collapsed solution (i.e., the network directly predicts the constant) during the optimization process, the gradient backpropagation on one side of the twin network is turned off. It can play the role of the real label during the training process. Through this self-supervised training method, by encouraging the features of specific image pairs to be as close as possible, that is, forcing p1 and z2 to be as close as possible, since both p1 and z2 are obtained through τ transformation, the network is encouraged to learn features that are invariant to the transformation τ. When designing the transformation τ, the characteristics of the data and the characteristics required by the task should be taken into account.

[0137] It is understandable that the structure of self-supervised pre-training can also adopt other self-supervised learning frameworks, such as MoCO V2, etc. These self-supervised learning frameworks can be embedded in this embodiment.

[0138] Through self-supervised pre-training, the backbone network can adapt well to the characteristics of line scan imaging, thereby obtaining a more semantic feature representation, which is beneficial for fine-tuning training of downstream tasks and avoids the high cost of direct large-scale supervised learning.

[0139] 305. In response to input of an image to be processed in the line scan data, use the target backbone network to process the detection task corresponding to the image to be processed.

[0140] In this embodiment, the input of the image to be processed in the line scan data can be a line scan image collected in real time by a line scan camera, or it can be an image to be processed in the line scan data stored in the background. The specific data source depends on the actual scenario. By performing defect detection on the image to be processed based on the target backbone network, an image to be processed containing a detection frame can be obtained. The detection frame marks the defects of the product.

[0141] The above embodiment introduces how to use the characteristics of line scan images for self-supervised learning, thereby avoiding the collection of a large number of defect samples. After the self-supervised training is completed, the backbone network weights with stronger expression capabilities are obtained, and the detectors actually used for defect detection can use various embodiments widely used in the industry, such as the Cascade R-CNN shown above, or other detectors. The embodiment does not limit the form of the specific detection model.

[0142] In one possible scenario, the detection task is performed as follows Fig.10 As shown, Fig.10A scene diagram of another image processing method provided in an embodiment of the present application; the figure shows the detection process of Cascade R-CNN, the target backbone network configuration and convolution layer of this embodiment, the features of the input line scan image are extracted through the convolution layer, and the subsequent cascade detection process is performed. That is, Cascade R-CNN uses different IOU thresholds to train multiple cascade detectors. The cascade method is to find a positive sample with a higher IoU for training in the next stage by adjusting the bounding boxes. Specifically, H0 in the figure represents the RPN network, H1 represents the head of Faster R-CNN for detection and classification, C1 represents the final classification result, and B1 represents the final bounding box regression result. After Cascade R-CNN obtains the detection box after B1 regression, it inputs it into the H2 part and continues to regress, and so on to the H3 part, so that the accuracy of the bounding box is improved to a certain extent each time, so as to achieve the effect of improving the accuracy of the detection box.

[0143] This embodiment uses a backbone network pre-training method based on self-supervised learning, which makes full use of the imaging characteristics of line scan images, namely: first, the imaging of the same point of different products has large differences, and this type of difference is different from the artificial data enhancement method. It is caused by normal texture changes and imaging condition changes of actual products, which is more authentic and can encourage the model to learn features that are more robust to these natural differences. At the same time, by learning the differences between different points of the same product, the model can also be encouraged to use contextual information to learn structural differences; second, the pixel redundancy problem caused by the high resolution of line scan imaging can be encouraged by downsampling operations based on image blocks to use a wider range of contextual information. The above two points are required for downstream defect detection tasks, so this embodiment can alleviate the high cost of collecting a large number of defect samples and data annotation, and improve the application value.

[0144] In combination with the above embodiments, it can be known that by acquiring a line scan image in the line scan data, the line scan image includes a plurality of image blocks; then, each image block in the line scan image is downsampled based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; and the first sampling image and the second sampling image are input into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, the preset backbone network includes a first feature extraction network and a second feature extraction network, the first feature extraction network is used to obtain a first image feature according to the first sampling image, and the second feature extraction network is used to obtain a second image feature according to the second sampling image. Image features, the weights of the first feature extraction network and the second feature extraction network are shared; the first image feature is then predicted to obtain a prediction result, and a preset backbone network is self-supervised trained based on the prediction result and the second image feature to obtain a target backbone network; and then in response to the input of the image to be processed in the line scan data, the target backbone network is used to process the detection task corresponding to the image to be processed, thereby realizing the processing process of the line scan image based on self-supervised training. Since the image distribution characteristics of the line scan image are used for targeted sampling, and self-supervised learning is performed based on the sampled image, there is no need to use labeled data, which avoids the occurrence of labeling errors and improves the accuracy of line scan image detection.

[0145] In order to better implement the above embodiments of the present application, the following also provides related devices for implementing the above embodiments. Fig.11 , Fig.11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The processing device 1100 includes:

[0146] An acquisition unit 1101 is used to acquire a line scan image from the line scan data, where the line scan image includes a plurality of image blocks;

[0147] A sampling unit 1102 is used to downsample each image block in the line scan image based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block;

[0148] The processing unit 1103 is used to input the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, wherein the preset backbone network includes a first feature extraction network and a second feature extraction network, wherein the first feature extraction network is used to obtain the first image feature according to the first sampling image, and the second feature extraction network is used to obtain the second image feature according to the second sampling image, and the weight value of the network parameter corresponding to the first feature extraction network is shared with the weight value of the network parameter corresponding to the second feature extraction network;

[0149] The processing unit 1103 is further configured to predict the first image feature to obtain a prediction result, and to perform self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network;

[0150] The processing unit 1103 is further configured to respond to an input of an image to be processed in the line scan data and use the target backbone network to process a detection task corresponding to the image to be processed.

[0151] Optionally, in some possible implementations of the present application, the sampling unit 1102 is specifically configured to divide each of the image blocks in the line scan image to obtain a plurality of image grids corresponding to each of the image blocks;

[0152] The sampling unit 1102 is specifically configured to determine a downsampling method based on the data transformation distribution;

[0153] The sampling unit 1102 is specifically configured to sample adjacent image grids into two different images according to the downsampling method, so as to obtain the first sampling image and the second sampling image.

[0154] Optionally, in some possible implementations of the present application, the sampling unit 1102 is specifically used to determine a sampling direction corresponding to the downsampling method;

[0155] The sampling unit 1102 is specifically used to divide the sampling unit 1102 in the corresponding direction based on the sampling direction, and the sampling unit 1102 includes a plurality of image grids;

[0156] The sampling unit 1102 is specifically configured to sample adjacent sampling units 1102 into two different images respectively, so as to obtain the first sampling image and the second sampling image.

[0157] Optionally, in some possible implementations of the present application, the sampling unit 1102 is specifically used to determine the line scan object corresponding to the line scan data;

[0158] The sampling unit 1102 is specifically used to obtain texture information configured for the line scan object;

[0159] The sampling unit 1102 is specifically configured to detect texture repeatability based on the texture information to determine a sampling direction corresponding to the downsampling method.

[0160] Optionally, in some possible implementations of the present application, the sampling unit 1102 is specifically used to obtain functional area information corresponding to the line scan object;

[0161] The sampling unit 1102 is specifically configured to divide the texture information based on the functional area information to obtain regional texture;

[0162] The sampling unit 1102 is specifically configured to perform texture repeatability detection according to the regional textures respectively, so as to determine the sampling direction corresponding to each of the regional textures.

[0163] Optionally, in some possible implementations of the present application, the processing unit 1103 is specifically configured to input the first image feature into a fully connected layer for prediction to obtain the prediction result;

[0164] The processing unit 1103 is specifically configured to bring the prediction result and the second image feature close to each other to generate supervision information;

[0165] The processing unit 1103 is specifically used to perform self-supervisory training on the preset backbone network based on the supervision information to obtain a target backbone network, and the supervision information will not perform gradient feedback to the second feature extraction network.

[0166] Optionally, in some possible implementations of the present application, the processing unit 1103 is specifically configured to input the first sampled image into the first feature extraction network in the preset backbone network, the first feature extraction network includes a plurality of first feature extraction blocks, and the first feature extraction block includes a first residual unit and a second residual unit connected to each other;

[0167] The processing unit 1103 is specifically configured to perform channel down-sampling on the first sampled image based on the first residual unit to obtain first feature information;

[0168] The processing unit 1103 is specifically configured to perform a convolution operation on the first feature information to obtain a first convolution feature;

[0169] The processing unit 1103 is specifically configured to perform channel upsampling on the first convolution feature based on the second residual unit to obtain second feature information;

[0170] The processing unit 1103 is specifically configured to perform a residual connection on the second feature information and the first sampled image to obtain the first image feature;

[0171] The processing unit 1103 is specifically configured to input the second sampled image into the second feature extraction network in the preset backbone network, wherein the second feature extraction network comprises a plurality of second feature extraction blocks, and the second feature extraction block comprises a third residual unit and a fourth residual unit connected to each other.

[0172] The processing unit 1103 is specifically configured to perform channel down-sampling on the second sampled image based on the third residual unit to obtain third feature information;

[0173] The processing unit 1103 is specifically configured to perform a convolution operation on the third feature information to obtain a second convolution feature;

[0174] The processing unit 1103 is specifically configured to perform channel upsampling on the second convolution feature based on the fourth residual unit to obtain fourth feature information;

[0175] The processing unit 1103 is specifically configured to perform a residual connection on the fourth feature information and the second sampling image to obtain the second image feature.

[0176] By acquiring a line scan image in line scan data, the line scan image includes a plurality of image blocks; then downsampling each image block in the line scan image based on data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; and inputting the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, the preset backbone network includes a first feature extraction network and a second feature extraction network, the first feature extraction network is used to obtain a first image feature according to the first sampling image, and the second feature extraction network is used to obtain a second image feature according to the second sampling image, The weights of the first feature extraction network and the second feature extraction network are shared; then the first image feature is predicted to obtain a prediction result, and the preset backbone network is self-supervised trained based on the prediction result and the second image feature to obtain a target backbone network; then in response to the input of the image to be processed in the line scan data, the target backbone network is used to process the detection task corresponding to the image to be processed, thereby realizing the processing process of the line scan image based on self-supervised training. Since targeted sampling is performed using the image distribution characteristics of the line scan image, and self-supervised learning is performed based on the sampled image, there is no need to use labeled data, which avoids the occurrence of labeling errors and improves the accuracy of line scan image detection.

[0177] The present application also provides a terminal device, such as Fig.12 The figure is a schematic diagram of the structure of another terminal device provided by the embodiment of the present application. For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example:

[0178] Fig.12 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Fig.12 The mobile phone includes: a radio frequency (RF) circuit 1210, a memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a processor 1280, and a power supply 1290. Those skilled in the art will understand that Fig.12 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone:

[0180] The RF circuit 1210 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1280 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 1210 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0181] The memory 1220 can be used to store software programs and modules. The processor 1280 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1220. The memory 1220 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1220 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0182] The input unit 1230 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232. The touch panel 1231, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1231 or near the touch panel 1231 using any suitable object or accessory such as a finger, stylus, etc., and the air touch operation within a certain range on the touch panel 1231), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1231 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 1280, and can receive and execute commands sent by the processor 1280. In addition, the touch panel 1231 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1231, the input unit 1230 can also include other input devices 1232. Specifically, the other input devices 1232 can include but are not limited to one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, etc.

[0183] The display unit 1240 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1240 may include a display panel 1241. Optionally, the display panel 1241 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1231 may cover the display panel 1241. When the touch panel 1231 detects a touch operation on or near it, it is transmitted to the processor 1280 to determine the type of touch event. Subsequently, the processor 1280 provides a corresponding visual output on the display panel 1241 according to the type of touch event. Although in Fig.12 In the embodiment, the touch panel 1231 and the display panel 1241 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1231 and the display panel 1241 can be integrated to realize the input and output functions of the mobile phone.

[0184] The mobile phone may also include at least one sensor 1250, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1241 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1241 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0185] The audio circuit 1260, the speaker 1261, and the microphone 1262 can provide an audio interface between the user and the mobile phone. The audio circuit 1260 can transmit the received audio data to the speaker 1261 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1262 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1260 and converted into audio data, and then the audio data is output to the processor 1280 for processing, and then sent to another mobile phone through the RF circuit 1210, or the audio data is output to the memory 1220 for further processing.

[0186] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1270. It provides users with wireless broadband Internet access. Fig.12 A WiFi module 1270 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0187] The processor 1280 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1220, and calling data stored in the memory 1220, it executes various functions of the mobile phone and processes data, thereby performing overall detection of the mobile phone. Optionally, the processor 1280 may include one or more processing units; optionally, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1280.

[0188] The mobile phone also includes a power supply 1290 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 1280 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0189] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0190] In the embodiment of the present application, the processor 1280 included in the terminal also has the function of executing each step of the above-mentioned page processing method.

[0191] The present application also provides a server. Fig.13 , Fig.131 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1322 (for example, one or more processors) and memory 1332, and one or more storage media 1330 (for example, one or more mass storage devices) storing application programs 1342 or data 1344. Among them, the memory 1332 and the storage medium 1330 may be short-term storage or permanent storage. The program stored in the storage medium 1330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1322 may be configured to communicate with the storage medium 1330 to execute a series of instruction operations in the storage medium 1330 on the server 1300.

[0192] The server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input and output interfaces 1358, and / or one or more operating systems 1341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0193] The steps performed by the management device in the above embodiment can be based on the Fig.13 The server structure shown.

[0194] The present application also provides a computer-readable storage medium in which an image processing instruction is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned Figures 3 to 10 The illustrated embodiment describes the steps performed by the image processing device in the method.

[0195] The present application also provides a computer program product including an image processing instruction, which, when executed on a computer, enables the computer to execute the above-mentioned Figures 3 to 10 The illustrated embodiment describes the steps performed by the image processing device in the method.

[0196] The present application also provides an image processing system, which may include Fig.11 The image processing device in the described embodiment, or Fig.12 The terminal device in the described embodiment, or Fig.13 The server being described.

[0197] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0198] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0200] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0201] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical embodiment of the present application is essentially or part of the prior art that contributes to the prior art or all or part of the technical embodiment can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, an image processing device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0202] As described above, the above embodiments are only used to illustrate the technical embodiments of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical embodiments described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical embodiments from the spirit and scope of the technical embodiments of the various embodiments of the present application.

Claims

1. A method for processing an image, characterized in that: include: Acquire a line scan image from the line scan data, wherein the line scan image includes a plurality of image blocks; Downsampling each image block in the line scan image based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; Inputting the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, the preset backbone network comprising a first feature extraction network and a second feature extraction network, the first feature extraction network being used to obtain the first image feature according to the first sampling image, the second feature extraction network being used to obtain the second image feature according to the second sampling image, and the weight value of the network parameter corresponding to the first feature extraction network being shared with the weight value of the network parameter corresponding to the second feature extraction network; Predicting the first image feature to obtain a prediction result, and performing self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network; In response to input of an image to be processed in the line scan data, using the target backbone network to process a detection task corresponding to the image to be processed; The downsampling of each image block in the line scan image based on the data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block includes: Dividing each of the image blocks in the line scan image to obtain a plurality of image grids corresponding to each of the image blocks; Determining a downsampling method based on the data transformation distribution; Determine a sampling direction corresponding to the downsampling method; Based on the sampling direction, a plurality of image grids of the line scan image in a corresponding direction are sampled to obtain a sampling unit; Adjacent sampling units are sampled into two different images respectively to obtain the first sampling image and the second sampling image.

2. The method according to claim 1, characterized in that The determining of the sampling direction corresponding to the downsampling mode includes: Determine the line scan object corresponding to the line scan data; Acquire texture information configured for the line scan object; Texture repeatability is detected based on the texture information to determine a sampling direction corresponding to the downsampling method.

3. The method according to claim 2, characterized in that The detecting of texture repeatability based on the texture information to determine the sampling direction corresponding to the downsampling method includes: Acquire functional area information corresponding to the line scan object; Dividing the texture information based on the functional area information to obtain regional texture; Texture repeatability is detected according to the regional textures respectively to determine the sampling direction corresponding to each regional texture.

4. The method according to claim 1, characterized in that: The step of predicting the first image feature to obtain a prediction result, and performing self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network, includes: Inputting the first image feature into a fully connected layer for prediction to obtain the prediction result; Approximating the prediction result to the second image feature to generate supervision information; Based on the supervision information, the preset backbone network is self-supervised trained to obtain a target backbone network, and the supervision information will not be gradient-transmitted back to the second feature extraction network.

5. The method according to claim 1, characterized in that The step of inputting the first sampled image and the second sampled image into a preset backbone network to obtain a first image feature corresponding to the first sampled image and a second image feature corresponding to the second sampled image includes: Inputting the first sampled image into the first feature extraction network in the preset backbone network, wherein the first feature extraction network comprises a plurality of first feature extraction blocks, and the first feature extraction block comprises a first residual unit and a second residual unit connected in series; Performing channel down-sampling on the first sampled image based on the first residual unit to obtain first feature information; Performing a convolution operation on the first feature information to obtain a first convolution feature; Performing channel upsampling on the first convolution feature based on the second residual unit to obtain second feature information; Performing a residual connection between the second feature information and the first sampled image to obtain the first image feature; Inputting the second sampling image into the second feature extraction network in the preset backbone network, wherein the second feature extraction network comprises a plurality of second feature extraction blocks, and the second feature extraction block comprises a third residual unit and a fourth residual unit connected to each other; Performing channel down-sampling on the second sampled image based on the third residual unit to obtain third feature information; Performing a convolution operation on the third feature information to obtain a second convolution feature; Performing channel upsampling on the second convolutional feature based on the fourth residual unit to obtain fourth feature information; Perform a residual connection on the fourth feature information and the second sampling image to obtain the second image feature.

6. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a line scan image in the line scan data, wherein the line scan image includes a plurality of image blocks; A sampling unit, configured to downsample each image block in the line scan image based on data transformation distribution to obtain a first sampling image and a second sampling image corresponding to each image block; a processing unit, configured to input the first sampling image and the second sampling image into a preset backbone network to obtain a first image feature corresponding to the first sampling image and a second image feature corresponding to the second sampling image, wherein the preset backbone network includes a first feature extraction network and a second feature extraction network, wherein the first feature extraction network is configured to obtain the first image feature according to the first sampling image, and the second feature extraction network is configured to obtain the second image feature according to the second sampling image, and the weight value of the network parameter corresponding to the first feature extraction network is shared with the weight value of the network parameter corresponding to the second feature extraction network; The processing unit is further configured to predict the first image feature to obtain a prediction result, so as to perform self-supervisory training on the preset backbone network based on the prediction result and the second image feature to obtain a target backbone network; The processing unit is further used to respond to the input of the image to be processed in the line scan data and use the target backbone network to process the detection task corresponding to the image to be processed; The sampling unit is specifically used to divide each of the image blocks in the line scan image to obtain a plurality of image grids corresponding to each of the image blocks; The sampling unit is specifically used to determine a downsampling method based on the data transformation distribution; The sampling unit is specifically configured to sample adjacent image grids into two different images according to the downsampling method, so as to obtain the first sampling image and the second sampling image; The sampling unit is specifically used to determine the sampling direction corresponding to the downsampling method; The sampling unit is specifically used to divide the sampling units in the corresponding direction based on the sampling direction, and the sampling unit includes a plurality of image grids; The sampling unit is specifically used to sample adjacent sampling units into two different images respectively to obtain the first sampling image and the second sampling image.

7. The device according to claim 6, characterized in that The sampling unit is specifically used to determine the line scan object corresponding to the line scan data; The sampling unit is specifically used to obtain texture information configured for the line scan object; The sampling unit is specifically used to detect texture repeatability based on the texture information to determine a sampling direction corresponding to the downsampling method.

8. The device according to claim 7, characterized in that The sampling unit is specifically used to obtain the functional area information corresponding to the line scan object; The sampling unit is specifically used to divide the texture information based on the functional area information to obtain regional texture; The sampling unit is specifically used to detect texture repeatability according to the regional textures respectively, so as to determine the sampling direction corresponding to each of the regional textures.

9. The device according to claim 6, characterized in that The processing unit is specifically configured to input the first image feature into a fully connected layer for prediction to obtain the prediction result; The processing unit is specifically configured to bring the prediction result and the second image feature close to each other to generate supervision information; The processing unit is specifically used to perform self-supervisory training on the preset backbone network based on the supervision information to obtain a target backbone network, and the supervision information will not perform gradient feedback to the second feature extraction network.

10. The device according to claim 6, characterized in that The processing unit is specifically configured to input the first sampled image into the first feature extraction network in the preset backbone network, wherein the first feature extraction network comprises a plurality of first feature extraction blocks, and the first feature extraction block comprises a first residual unit and a second residual unit connected in series; The processing unit is specifically configured to perform channel down-sampling on the first sampled image based on the first residual unit to obtain first feature information; The processing unit is specifically configured to perform a convolution operation on the first feature information to obtain a first convolution feature; The processing unit is specifically configured to perform channel upsampling on the first convolution feature based on the second residual unit to obtain second feature information; The processing unit is specifically configured to perform a residual connection between the second feature information and the first sampled image to obtain the first image feature; The processing unit is specifically configured to input the second sampling image into the second feature extraction network in the preset backbone network, wherein the second feature extraction network comprises a plurality of second feature extraction blocks, and the second feature extraction block comprises a third residual unit and a fourth residual unit connected to each other. The processing unit is specifically configured to perform channel down-sampling on the second sampled image based on the third residual unit to obtain third feature information; The processing unit is specifically configured to perform a convolution operation on the third feature information to obtain a second convolution feature; The processing unit is specifically configured to perform channel upsampling on the second convolution feature based on the fourth residual unit to obtain fourth feature information; The processing unit is specifically configured to perform a residual connection between the fourth feature information and the second sampling image to obtain the second image feature.

11. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program codes; the processor is used to execute the image processing method according to any one of claims 1 to 6 according to instructions in the program codes.

12. A computer program product, comprising a computer program / instructions, wherein the computer program / instructions are stored in a computer-readable storage medium, characterized in that: When the computer program / instructions in the computer-readable storage medium are executed by a processor, the steps of the image processing method described in any one of claims 1 to 6 are implemented.

Images